feat: update dist-git dir rendering - #348
Conversation
Add a global, persistent '--without-lockfile' boolean flag that selects a preview mode in which azldev tracks resolved upstream commits in generated component configuration instead of per-component lock files. The flag defaults to false, which keeps azldev's existing lock-file behavior unchanged. The registered command set and configuration loading both depend on the mode, and both happen before cobra parses the command line, so the flag is hand-parsed up front by App.PreParseGlobalFlags. Only exact '--without-lockfile' and '--without-lockfile=<value>' tokens are recognized, and scanning stops at the '--' terminator so positional text is never mistaken for the flag. The CLI entry point pre-parses the process arguments before registering commands; Execute pre-parses again so an App executed directly behaves identically. The mode is carried on Env via EnvOptions.WithoutLockfile, and the lock store is not created when the mode is active, so every lock-file consumer degrades to a no-op without needing mode-specific code. The flag is part of the CLI surface, so the help, MCP tool-schema, and usage-error snapshots pick up the new global option. Refs: microsoft#323
Teach the project configuration loader the lock-file-free mode selected by '--without-lockfile'. In that mode component definitions merge with override semantics rather than additively, so a generated upstream-commit TOML can replace a component's configured pin, and the config file that supplied the pin is recorded separately from the component's primary TOML so synthetic history can follow it. Because a single file may now hold a partial component definition, component validation is deferred until the whole project is assembled. The project's lock directory is left unset in this mode; a project may still declare 'lock-dir' for compatibility with the default mode, where the value keeps its existing meaning. The default mode is unchanged: components merge additively, each config file is validated on its own, and the lock directory keeps its default. The load-time modes are carried in an internal loadOptions value instead of adding more positional booleans to every loader helper. Refs: microsoft#323
Add a store for the generated component configuration that pins resolved upstream commits in lock-file-free mode. Each component gets a normal component TOML holding only 'spec.upstream-commit', written with a header marking it as generated, so the project's ordinary include and merge rules supply the pin to every other command. The package is self-contained and unused until the component commands adopt it; nothing changes in azldev's default mode. Refs: microsoft#323
Add the lock-file-free source of synthetic dist-git history. Instead of walking a component's lock file for input-fingerprint changes, the new discovery path walks the generated upstream-commit TOML that pinned the component and records the pin held at each commit that touched it, including the pin carried by a commit that deletes the file. Discovery is the only part that differs: the recorded change data is identical, so interleaving and replay are shared with the default mode. No fork point is persisted in this mode, so the upstream walk is bounded by the repository's first-parent root rather than by an import commit. Preparers select the source with the new WithoutLockfileHistory option, which is rejected together with dirty detection because there is no fingerprint to compare a working tree against. Component build, render, and prepare-sources now assemble their git-repo preparer options through a single mode-aware helper. Refs: microsoft#323
Register the component commands that maintain resolved state according to the selected mode. The default mode keeps history, query, and update; the lock-file-free mode instead gets refresh-upstream-commit, which resolves each selected upstream component at the distro snapshot and records the result as generated component TOML, plus hidden no-op stand-ins for the lock-file commands so existing invocations report that they do nothing. Because a stale generated pin can make strict configuration loading fail, the refresh command is annotated to load configuration permissively, and commands can now be excluded from generated Markdown docs so the no-op stand-ins stay out of the reference documentation. Component commands take construction options so that lock-file-only flags, such as '--skip-lock-validation', are simply not registered in lock-file-free mode instead of being accepted and ignored. Both modes are exercised by unit tests, including a lock-file-free test environment. Refs: microsoft#323
Add the lock-file-free implementation of 'component changed'. Without lock files there are no stored fingerprints to compare, so the project configuration is loaded independently at both refs, using normal TOML parsing including recursive includes, merging, component defaults, and the generated upstream-commit TOMLs, and the resolved component build inputs are compared directly. The comparison covers normalized component configuration, upstream commit or local spec-directory contents, overlay source filenames and contents, and the effective distro release version, while documentation, publishing, test-selection, scheduling-hint, snapshot-time, and checkout-path-only fields are excluded. All-component scans use the union of components in both refs, so additions and deletions are reported without consulting the current checkout. The command dispatches to the implementation matching the active mode and documents the comparison it performs in its long help; the git plumbing helpers are shared between the two. Refs: microsoft#323
Emit agent skills and instruction files that match the mode azldev is running in. A catalog selects the content: the default mode keeps the lock-file workflow and the azldev-update-component skill, while lock-file-free mode documents the generated upstream-commit workflow and the azldev-refresh-upstream-commit skill. Skills, instruction wrappers, and the emitted layout are shared. Only the documents whose subject differs get a mode-specific template, layered over the default template of the same name, and a small registry replaces the skill entries and instruction trigger text that name the command. The bindings injected into the content now carry both the lock directory and the generated upstream-commit directory. 'docs agent install' and 'docs agent show' select the catalog from the environment, so the emitted files, the served skill documents, and the read-only MCP tool all describe the commands the user actually has. Refs: microsoft#323
Add a how-to that describes the opt-in lock-file-free preview: what the flag changes, how to order the generated upstream-commit includes, how to refresh a component, how change detection differs, and how to emit agent files for the mode. Cross-reference it from the config-system explanation and the config-file reference, where the merge and include rules differ. Regenerate the CLI reference and JSON schema. The generated reference documents azldev's default mode, so the only change is the new global flag in each command's inherited options; the preview mode's command surface is covered by the how-to and by '--without-lockfile <command> --help'. Refs: microsoft#323
Add scenario coverage for both modes of the preview flag: the global flag appears in help, the component command set and lock-file-only flags follow the selected mode, a lock-file command invoked in lock-file-free mode reports that it does nothing, the emitted agent skills describe the mode's workflow, and the MCP tool surface matches the registered commands. Also compare components across two commits in lock-file-free mode, where a changed local spec is detected from the project configuration resolved at each ref and no lock directory is created. Refs: microsoft#323
Record the new opt-in preview mode under Unreleased, making clear that the global flag defaults to off, that azldev's behavior is unchanged without it, and that the preview surface is not stable. Each entry states which behavior applies only when the flag is passed. Refs: microsoft#323
…amples The refresh-upstream-commit command is only registered when the preview --without-lockfile global flag is set, so its examples and CI-gate error message were missing that required flag. Refs: microsoft#323
Validate custom source declarations before converting their script names to absolute paths in lock-file-free mode. Final assembled-config validation now checks the effective script filename while preserving rejection of unsafe paths and input conflicts. Refs: microsoft#323
Render complete post-overlay dist-git directories without invoking mock, rpmautospec, or spectool. Preserve all prepared files and retain the old skip-file-filter setting only as an ignored compatibility option. Lock-file-free rendering no longer inspects generated upstream-commit TOML history or creates synthetic commits. Update tests, generated references, configuration docs, and scenario expectations for the simplified pipeline. Refs: microsoft#348
5978654 to
69dde50
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The lock-file-free agent skill template still documents render.skip-file-filter as functional even though the PR makes it deprecated/ignored, which would mislead users.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces a preview --without-lockfile mode that tracks resolved upstream commits via generated per-component TOML (instead of lock files), updates config loading/merging/validation semantics to support that mode, and adjusts rendering/documentation to reflect dist-git preservation behavior.
Changes:
- Add global
--without-lockfileflag, plumb mode into app/env, and register mode-specific component command surfaces (including legacy no-op shims). - Add generated upstream-commit TOML storage + refresh workflow, plus config loader override-merge semantics and validation adjustments needed for partial component definitions.
- Update synthetic history discovery paths and refresh related docs/skills/schema/help snapshots.
File summaries
| File | Description |
|---|---|
| schemas/azldev.schema.json | Updates schema description for deprecated skip-file-filter. |
| scenario/snapshots/TestSnapshotsContainer_help_stdout_1.snap | Help snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshotsContainer_--help_stdout_1.snap | Help snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshotsContainer_--bogus-flag_stderr_1.snap | Help/error snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshots_help_stdout_1.snap | Help snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshots_--help_with_color_stdout_1.snap | Color help snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshots_--help_stdout_1.snap | Help snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestSnapshots_--bogus-flag_stderr_1.snap | Help/error snapshot updated to include --without-lockfile. |
| scenario/snapshots/TestMCPServerMode_1.snap.json | MCP schema snapshot updated to include without-lockfile flag. |
| pkg/app/azldev_cli/azldev.go | CLI entrypoint updated to pre-parse args before command registration. |
| internal/upstreamcommit/store.go | New store for generated upstream-commit TOML files. |
| internal/upstreamcommit/store_test.go | Unit tests for upstream-commit store behavior. |
| internal/projectgen/projectgen_test.go | Updates for new LoadProjectConfig signature/behavior. |
| internal/projectconfig/render.go | Deprecates/ignores skip-file-filter and excludes it from fingerprinting. |
| internal/projectconfig/project.go | Adds mode-sensitive project validation path for lock-file-free merge semantics. |
| internal/projectconfig/loader.go | Introduces load options + override-merge path in lock-file-free mode. |
| internal/projectconfig/fingerprint_test.go | Registers fingerprint exclusion for deprecated SkipFileFilter. |
| internal/projectconfig/configfile.go | Refactors validation to support deferred component validation + custom script path handling. |
| internal/projectconfig/config.go | Adds withoutLockfile mode to config loading and clears lock dir in that mode. |
| internal/projectconfig/config_test.go | Updates tests and adds coverage for lock dir behavior by mode. |
| internal/projectconfig/component.go | Adds upstream-commit provenance tracking and override-merge support. |
| internal/app/azldev/env.go | Plumbs lock-file-free mode into env and disables lock store when active. |
| internal/app/azldev/core/testutils/testenv.go | Adds test env constructor for lock-file-free mode. |
| internal/app/azldev/core/sources/synthistory.go | Refactors repo opening to be config-file-path based. |
| internal/app/azldev/core/sources/synthistory_upstreamcommit.go | New synthetic-history discovery from upstream-commit TOML changes. |
| internal/app/azldev/core/sources/synthistory_upstreamcommit_internal_test.go | Unit tests for upstream-commit synthetic history helpers. |
| internal/app/azldev/core/sources/sourceprep.go | Adds preparer option + logic to choose synthetic-history source by mode. |
| internal/app/azldev/core/sources/render_process.py | Removes old render batch-processing script. |
| internal/app/azldev/core/sources/query_process.py | Updates query script docs now that render script is removed. |
| internal/app/azldev/core/sources/mockprocessor.go | Removes render batch-processing plumbing; keeps shared batched script runner for query. |
| internal/app/azldev/core/sources/mockprocessor_test.go | Removes tests for deleted render batch-processing internals. |
| internal/app/azldev/core/components/filter.go | Splits lock-validation flag registration from base component filter flags. |
| internal/app/azldev/command.go | Adds annotations/utilities for permissive-config and markdown-doc exclusion. |
| internal/app/azldev/cmds/docs/markdown.go | Adds markdown-doc exclusion hiding while preserving original hidden states. |
| internal/app/azldev/cmds/docs/agent.go | Makes agent skill emission/serving mode-aware and adds upstream-commit dir binding. |
| internal/app/azldev/cmds/docs/agent_internal_test.go | Updates agent cmd tests for mode-aware catalog behavior. |
| internal/app/azldev/cmds/component/update.go | Refactors orphan handling via shared reconciliation helper. |
| internal/app/azldev/cmds/component/update_test.go | Minor test fix (uses test constant for component name). |
| internal/app/azldev/cmds/component/render_internal_test.go | Removes tests for deleted/obsolete render file-filter behavior. |
| internal/app/azldev/cmds/component/refreshupstreamcommit_resolution_test.go | Ensures refresh resolves upstream state each run in lock-file-free mode. |
| internal/app/azldev/cmds/component/refreshupstreamcommit_internal_test.go | Tests saving behavior for generated upstream-commit TOMLs. |
| internal/app/azldev/cmds/component/query.go | Uses mode-aware component flag registration helper. |
| internal/app/azldev/cmds/component/preparesources.go | Uses shared preparer option selection (mode-aware synthetic history). |
| internal/app/azldev/cmds/component/preparerctx.go | Centralizes preparer options by mode for build/render/prepare-sources. |
| internal/app/azldev/cmds/component/preparerctx_internal_test.go | Tests render synthetic-history option selection by mode. |
| internal/app/azldev/cmds/component/orphans.go | New shared orphan reconciliation helper for per-component state stores. |
| internal/app/azldev/cmds/component/mockprocessor.go | Updates mock package set naming for provenance vs query paths. |
| internal/app/azldev/cmds/component/mockprocessor_internal_test.go | Removes tests for deleted render mock processor helper. |
| internal/app/azldev/cmds/component/mockproc.go | Renames render mock package preset to provenance preset. |
| internal/app/azldev/cmds/component/list.go | Mode-aware lock-validation flag hiding behavior. |
| internal/app/azldev/cmds/component/legacy.go | Adds hidden no-op shims for lock-file-only commands in preview mode. |
| internal/app/azldev/cmds/component/legacy_internal_test.go | Tests legacy no-op shim behavior. |
| internal/app/azldev/cmds/component/history.go | Uses mode-aware component flag registration and removes render skip-file-filter kind mention. |
| internal/app/azldev/cmds/component/history_internal_test.go | Updates customization collector expectations for deprecated render config. |
| internal/app/azldev/cmds/component/history_customizations.go | Removes render customization emission for deprecated render config. |
| internal/app/azldev/cmds/component/diffsources.go | Uses mode-aware component flag registration helper. |
| internal/app/azldev/cmds/component/component.go | Registers different component subcommand sets depending on mode. |
| internal/app/azldev/cmds/component/component_test.go | Tests mode-specific command surface and flag registration differences. |
| internal/app/azldev/cmds/component/cmdoptions.go | New shared “construct flags by mode” helpers for component commands. |
| internal/app/azldev/cmds/component/changed.go | Adds mode-dependent component changed behavior and help text. |
| internal/app/azldev/cmds/component/changed_test.go | Tests changed command long help differs by mode. |
| internal/app/azldev/cmds/component/build.go | Uses centralized preparer option selection (mode-aware synthetic history). |
| internal/app/azldev/app.go | Adds --without-lockfile global flag and pre-parse global flags before registration/loading. |
| internal/app/azldev/app_test.go | Adds tests for without-lockfile parsing/idempotence and env plumbing. |
| internal/app/azldev/agentskill/doc.go | Documents mode-specific skill catalogs and template layering. |
| internal/app/azldev/agentskill/content/withoutlockfile/remove-component.md.tmpl | New lock-file-free skill content variant for removing a component. |
| internal/app/azldev/agentskill/content/withoutlockfile/refresh-upstream-commit.md.tmpl | New lock-file-free skill content variant for refreshing upstream commits. |
| internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl | New lock-file-free skill content variant for component TOML guidance. |
| internal/app/azldev/agentskill/content/withoutlockfile/build-component.md.tmpl | New lock-file-free skill content variant for build workflow guidance. |
| internal/app/azldev/agentskill/content/withoutlockfile/azldev.md.tmpl | New lock-file-free azldev overview skill content variant. |
| internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl | New lock-file-free skill content variant for adding a component. |
| internal/app/azldev/agentskill/agentskill_test.go | Tests for catalog content/skill selection correctness by mode. |
| internal/app/azldev/agentskill/agentskill_internal_test.go | Updates internal instruction rendering tests for catalog-based APIs. |
| docs/user/reference/config/project.md | Clarifies rendered output is dist-git directories. |
| docs/user/reference/config/config-file.md | Adds preview note about include ordering for generated upstream commits. |
| docs/user/reference/config/components.md | Deprecates skip-file-filter in docs and clarifies rpmautospec/provenance behavior. |
| docs/user/reference/cli/azldev.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_version.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_repo.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_repo_query.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_project.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_project_new.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_project_init.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_package.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_package_list.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_test.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_list.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_inject-files.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_customize.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_build.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_image_boot.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_docs.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_docs_markdown.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_docs_agent.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_docs_agent_show.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_docs_agent_install.md | CLI docs updated for new global flag and updated synopsis text. |
| docs/user/reference/cli/azldev_config.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_config_generate-schema.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_config_dump.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component.md | CLI docs updated for new global flag and render wording. |
| docs/user/reference/cli/azldev_component_update.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_test.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_render.md | CLI docs updated to describe dist-git preservation and new global flag. |
| docs/user/reference/cli/azldev_component_query.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_prepare-sources.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_list.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_history.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_diff-sources.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_changed.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_build.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_component_add.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_completion.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_completion_zsh.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_completion_powershell.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_completion_fish.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_completion_bash.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_wget.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_mock.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_mock_shell.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_mock_build-rpms.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_mcp.md | CLI docs updated for new global flag. |
| docs/user/reference/cli/azldev_advanced_download-sources.md | CLI docs updated for new global flag. |
| docs/user/README.md | Adds how-to link for preview lock-file-free mode. |
| docs/user/how-to/preview-without-lockfile.md | New how-to guide describing preview mode behavior and workflow. |
| docs/user/explanation/config-system.md | Adds preview note about override-merge semantics and deferred validation. |
| CHANGELOG.md | Adds Unreleased changelog entries for preview mode + behavioral changes. |
| .github/instructions/agent-skills.instructions.md | Documents mode-specific skill/catalog behavior and new binding. |
Review details
Suppressed comments (1)
internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl:91
- This section describes
render.skip-file-filteras if it changes render behavior, but in this PR the setting is explicitly deprecated/ignored (render always preserves every file in the prepared dist-git dir). The skill should match the actual behavior so users don’t try to rely on a no-op setting.
- Files reviewed: 140/140 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | Field | Purpose | | ||
| | --- | --- | | ||
| | `spec` | where the spec comes from (see below) | | ||
| | `overlays` / `overlay-files` | targeted spec/source edits (see the `azldev-overlays` skill) | | ||
| | `build.defines` / `build.with` / `build.without` | RPM macro and bcond build tweaks | | ||
| | `release.calculation` | how the `Release` tag is managed | | ||
| | `render.skip-file-filter` | rendering edge-case escape hatch | | ||
|
|
| assert.Equal(t, `# This file was generated by 'azldev component refresh-upstream-commit' | ||
| # Do not edit this file, changes will be lost | ||
| # For more details see 'azldev component refresh-upstream-commit --help' | ||
| [components] | ||
| [components.bash] | ||
| [components.bash.spec] | ||
| upstream-commit = 'abcdef1234567' | ||
| `, string(data)) |
There was a problem hiding this comment.
🟡 Changes recommended
Pre-parsing global flags is not actually idempotent as documented (state can stick across calls), and a newly added merge error message uses incorrect context, both of which should be corrected before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 140/140 changed files
- Comments generated: 2
- Review effort level: Lite
| func (a *App) PreParseGlobalFlags(args []string) { | ||
| // Reset accumulating state so repeated calls are idempotent. | ||
| a.configFiles = nil | ||
|
|
||
| a.withoutLockfile = parseWithoutLockfileFlag(args) | ||
| a.handParseConfigFlags(args) | ||
| } |
| err := mergo.Merge(c, &otherTopLevel, mergo.WithOverride, mergo.WithAppendSlice) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to merge project info:\n%w", err) | ||
| } |
Update the rendering process to preserve the upstream dist-git content as well as fixing the changelog and release calculation/generation.
On top of #323