Skip to content

feat: update dist-git dir rendering - #348

Draft
Dan Streetman (ddstreet) wants to merge 13 commits into
microsoft:mainfrom
ddstreet:dist-git-rendering
Draft

feat: update dist-git dir rendering#348
Dan Streetman (ddstreet) wants to merge 13 commits into
microsoft:mainfrom
ddstreet:dist-git-rendering

Conversation

@ddstreet

Copy link
Copy Markdown
Contributor

Update the rendering process to preserve the upstream dist-git content as well as fixing the changelog and release calculation/generation.

On top of #323

Copilot AI added 12 commits September 8, 2026 19:18
Add a global, persistent '--without-lockfile' boolean flag that selects a
preview mode in which azldev tracks resolved upstream commits in generated
component configuration instead of per-component lock files. The flag
defaults to false, which keeps azldev's existing lock-file behavior
unchanged.

The registered command set and configuration loading both depend on the
mode, and both happen before cobra parses the command line, so the flag is
hand-parsed up front by App.PreParseGlobalFlags. Only exact
'--without-lockfile' and '--without-lockfile=<value>' tokens are
recognized, and scanning stops at the '--' terminator so positional text is
never mistaken for the flag. The CLI entry point pre-parses the process
arguments before registering commands; Execute pre-parses again so an App
executed directly behaves identically.

The mode is carried on Env via EnvOptions.WithoutLockfile, and the lock
store is not created when the mode is active, so every lock-file consumer
degrades to a no-op without needing mode-specific code.

The flag is part of the CLI surface, so the help, MCP tool-schema, and
usage-error snapshots pick up the new global option.

Refs: microsoft#323
Teach the project configuration loader the lock-file-free mode selected by
'--without-lockfile'. In that mode component definitions merge with
override semantics rather than additively, so a generated upstream-commit
TOML can replace a component's configured pin, and the config file that
supplied the pin is recorded separately from the component's primary TOML
so synthetic history can follow it.

Because a single file may now hold a partial component definition,
component validation is deferred until the whole project is assembled.
The project's lock directory is left unset in this mode; a project may
still declare 'lock-dir' for compatibility with the default mode, where
the value keeps its existing meaning.

The default mode is unchanged: components merge additively, each config
file is validated on its own, and the lock directory keeps its default.
The load-time modes are carried in an internal loadOptions value instead of
adding more positional booleans to every loader helper.

Refs: microsoft#323
Add a store for the generated component configuration that pins resolved
upstream commits in lock-file-free mode. Each component gets a normal
component TOML holding only 'spec.upstream-commit', written with a header
marking it as generated, so the project's ordinary include and merge rules
supply the pin to every other command.

The package is self-contained and unused until the component commands adopt
it; nothing changes in azldev's default mode.

Refs: microsoft#323
Add the lock-file-free source of synthetic dist-git history. Instead of
walking a component's lock file for input-fingerprint changes, the new
discovery path walks the generated upstream-commit TOML that pinned the
component and records the pin held at each commit that touched it,
including the pin carried by a commit that deletes the file.

Discovery is the only part that differs: the recorded change data is
identical, so interleaving and replay are shared with the default mode. No
fork point is persisted in this mode, so the upstream walk is bounded by
the repository's first-parent root rather than by an import commit.

Preparers select the source with the new WithoutLockfileHistory option,
which is rejected together with dirty detection because there is no
fingerprint to compare a working tree against. Component build, render, and
prepare-sources now assemble their git-repo preparer options through a
single mode-aware helper.

Refs: microsoft#323
Register the component commands that maintain resolved state according to
the selected mode. The default mode keeps history, query, and update; the
lock-file-free mode instead gets refresh-upstream-commit, which resolves
each selected upstream component at the distro snapshot and records the
result as generated component TOML, plus hidden no-op stand-ins for the
lock-file commands so existing invocations report that they do nothing.

Because a stale generated pin can make strict configuration loading fail,
the refresh command is annotated to load configuration permissively, and
commands can now be excluded from generated Markdown docs so the no-op
stand-ins stay out of the reference documentation.

Component commands take construction options so that lock-file-only flags,
such as '--skip-lock-validation', are simply not registered in
lock-file-free mode instead of being accepted and ignored. Both modes are
exercised by unit tests, including a lock-file-free test environment.

Refs: microsoft#323
Add the lock-file-free implementation of 'component changed'. Without lock
files there are no stored fingerprints to compare, so the project
configuration is loaded independently at both refs, using normal TOML
parsing including recursive includes, merging, component defaults, and the
generated upstream-commit TOMLs, and the resolved component build inputs
are compared directly.

The comparison covers normalized component configuration, upstream commit
or local spec-directory contents, overlay source filenames and contents,
and the effective distro release version, while documentation, publishing,
test-selection, scheduling-hint, snapshot-time, and checkout-path-only
fields are excluded. All-component scans use the union of components in
both refs, so additions and deletions are reported without consulting the
current checkout.

The command dispatches to the implementation matching the active mode and
documents the comparison it performs in its long help; the git plumbing
helpers are shared between the two.

Refs: microsoft#323
Emit agent skills and instruction files that match the mode azldev is
running in. A catalog selects the content: the default mode keeps the
lock-file workflow and the azldev-update-component skill, while
lock-file-free mode documents the generated upstream-commit workflow and
the azldev-refresh-upstream-commit skill.

Skills, instruction wrappers, and the emitted layout are shared. Only the
documents whose subject differs get a mode-specific template, layered over
the default template of the same name, and a small registry replaces the
skill entries and instruction trigger text that name the command. The
bindings injected into the content now carry both the lock directory and
the generated upstream-commit directory.

'docs agent install' and 'docs agent show' select the catalog from the
environment, so the emitted files, the served skill documents, and the
read-only MCP tool all describe the commands the user actually has.

Refs: microsoft#323
Add a how-to that describes the opt-in lock-file-free preview: what the
flag changes, how to order the generated upstream-commit includes, how to
refresh a component, how change detection differs, and how to emit agent
files for the mode. Cross-reference it from the config-system explanation
and the config-file reference, where the merge and include rules differ.

Regenerate the CLI reference and JSON schema. The generated reference
documents azldev's default mode, so the only change is the new global flag
in each command's inherited options; the preview mode's command surface is
covered by the how-to and by '--without-lockfile <command> --help'.

Refs: microsoft#323
Add scenario coverage for both modes of the preview flag: the global flag
appears in help, the component command set and lock-file-only flags follow
the selected mode, a lock-file command invoked in lock-file-free mode
reports that it does nothing, the emitted agent skills describe the mode's
workflow, and the MCP tool surface matches the registered commands.

Also compare components across two commits in lock-file-free mode, where a
changed local spec is detected from the project configuration resolved at
each ref and no lock directory is created.

Refs: microsoft#323
Record the new opt-in preview mode under Unreleased, making clear that the
global flag defaults to off, that azldev's behavior is unchanged without
it, and that the preview surface is not stable. Each entry states which
behavior applies only when the flag is passed.

Refs: microsoft#323
…amples

The refresh-upstream-commit command is only registered when the
preview --without-lockfile global flag is set, so its examples and
CI-gate error message were missing that required flag.

Refs: microsoft#323
Validate custom source declarations before converting their script names to
absolute paths in lock-file-free mode. Final assembled-config validation now
checks the effective script filename while preserving rejection of unsafe
paths and input conflicts.

Refs: microsoft#323
Copilot AI lite review requested due to automatic review settings September 9, 2026 13:43
Render complete post-overlay dist-git directories without invoking mock,
rpmautospec, or spectool. Preserve all prepared files and retain the old
skip-file-filter setting only as an ignored compatibility option.

Lock-file-free rendering no longer inspects generated upstream-commit TOML
history or creates synthetic commits. Update tests, generated references,
configuration docs, and scenario expectations for the simplified pipeline.

Refs: microsoft#348

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The lock-file-free agent skill template still documents render.skip-file-filter as functional even though the PR makes it deprecated/ignored, which would mislead users.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces a preview --without-lockfile mode that tracks resolved upstream commits via generated per-component TOML (instead of lock files), updates config loading/merging/validation semantics to support that mode, and adjusts rendering/documentation to reflect dist-git preservation behavior.

Changes:

  • Add global --without-lockfile flag, plumb mode into app/env, and register mode-specific component command surfaces (including legacy no-op shims).
  • Add generated upstream-commit TOML storage + refresh workflow, plus config loader override-merge semantics and validation adjustments needed for partial component definitions.
  • Update synthetic history discovery paths and refresh related docs/skills/schema/help snapshots.
File summaries
File Description
schemas/azldev.schema.json Updates schema description for deprecated skip-file-filter.
scenario/snapshots/TestSnapshotsContainer_help_stdout_1.snap Help snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshotsContainer_--help_stdout_1.snap Help snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshotsContainer_--bogus-flag_stderr_1.snap Help/error snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshots_help_stdout_1.snap Help snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshots_--help_with_color_stdout_1.snap Color help snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshots_--help_stdout_1.snap Help snapshot updated to include --without-lockfile.
scenario/snapshots/TestSnapshots_--bogus-flag_stderr_1.snap Help/error snapshot updated to include --without-lockfile.
scenario/snapshots/TestMCPServerMode_1.snap.json MCP schema snapshot updated to include without-lockfile flag.
pkg/app/azldev_cli/azldev.go CLI entrypoint updated to pre-parse args before command registration.
internal/upstreamcommit/store.go New store for generated upstream-commit TOML files.
internal/upstreamcommit/store_test.go Unit tests for upstream-commit store behavior.
internal/projectgen/projectgen_test.go Updates for new LoadProjectConfig signature/behavior.
internal/projectconfig/render.go Deprecates/ignores skip-file-filter and excludes it from fingerprinting.
internal/projectconfig/project.go Adds mode-sensitive project validation path for lock-file-free merge semantics.
internal/projectconfig/loader.go Introduces load options + override-merge path in lock-file-free mode.
internal/projectconfig/fingerprint_test.go Registers fingerprint exclusion for deprecated SkipFileFilter.
internal/projectconfig/configfile.go Refactors validation to support deferred component validation + custom script path handling.
internal/projectconfig/config.go Adds withoutLockfile mode to config loading and clears lock dir in that mode.
internal/projectconfig/config_test.go Updates tests and adds coverage for lock dir behavior by mode.
internal/projectconfig/component.go Adds upstream-commit provenance tracking and override-merge support.
internal/app/azldev/env.go Plumbs lock-file-free mode into env and disables lock store when active.
internal/app/azldev/core/testutils/testenv.go Adds test env constructor for lock-file-free mode.
internal/app/azldev/core/sources/synthistory.go Refactors repo opening to be config-file-path based.
internal/app/azldev/core/sources/synthistory_upstreamcommit.go New synthetic-history discovery from upstream-commit TOML changes.
internal/app/azldev/core/sources/synthistory_upstreamcommit_internal_test.go Unit tests for upstream-commit synthetic history helpers.
internal/app/azldev/core/sources/sourceprep.go Adds preparer option + logic to choose synthetic-history source by mode.
internal/app/azldev/core/sources/render_process.py Removes old render batch-processing script.
internal/app/azldev/core/sources/query_process.py Updates query script docs now that render script is removed.
internal/app/azldev/core/sources/mockprocessor.go Removes render batch-processing plumbing; keeps shared batched script runner for query.
internal/app/azldev/core/sources/mockprocessor_test.go Removes tests for deleted render batch-processing internals.
internal/app/azldev/core/components/filter.go Splits lock-validation flag registration from base component filter flags.
internal/app/azldev/command.go Adds annotations/utilities for permissive-config and markdown-doc exclusion.
internal/app/azldev/cmds/docs/markdown.go Adds markdown-doc exclusion hiding while preserving original hidden states.
internal/app/azldev/cmds/docs/agent.go Makes agent skill emission/serving mode-aware and adds upstream-commit dir binding.
internal/app/azldev/cmds/docs/agent_internal_test.go Updates agent cmd tests for mode-aware catalog behavior.
internal/app/azldev/cmds/component/update.go Refactors orphan handling via shared reconciliation helper.
internal/app/azldev/cmds/component/update_test.go Minor test fix (uses test constant for component name).
internal/app/azldev/cmds/component/render_internal_test.go Removes tests for deleted/obsolete render file-filter behavior.
internal/app/azldev/cmds/component/refreshupstreamcommit_resolution_test.go Ensures refresh resolves upstream state each run in lock-file-free mode.
internal/app/azldev/cmds/component/refreshupstreamcommit_internal_test.go Tests saving behavior for generated upstream-commit TOMLs.
internal/app/azldev/cmds/component/query.go Uses mode-aware component flag registration helper.
internal/app/azldev/cmds/component/preparesources.go Uses shared preparer option selection (mode-aware synthetic history).
internal/app/azldev/cmds/component/preparerctx.go Centralizes preparer options by mode for build/render/prepare-sources.
internal/app/azldev/cmds/component/preparerctx_internal_test.go Tests render synthetic-history option selection by mode.
internal/app/azldev/cmds/component/orphans.go New shared orphan reconciliation helper for per-component state stores.
internal/app/azldev/cmds/component/mockprocessor.go Updates mock package set naming for provenance vs query paths.
internal/app/azldev/cmds/component/mockprocessor_internal_test.go Removes tests for deleted render mock processor helper.
internal/app/azldev/cmds/component/mockproc.go Renames render mock package preset to provenance preset.
internal/app/azldev/cmds/component/list.go Mode-aware lock-validation flag hiding behavior.
internal/app/azldev/cmds/component/legacy.go Adds hidden no-op shims for lock-file-only commands in preview mode.
internal/app/azldev/cmds/component/legacy_internal_test.go Tests legacy no-op shim behavior.
internal/app/azldev/cmds/component/history.go Uses mode-aware component flag registration and removes render skip-file-filter kind mention.
internal/app/azldev/cmds/component/history_internal_test.go Updates customization collector expectations for deprecated render config.
internal/app/azldev/cmds/component/history_customizations.go Removes render customization emission for deprecated render config.
internal/app/azldev/cmds/component/diffsources.go Uses mode-aware component flag registration helper.
internal/app/azldev/cmds/component/component.go Registers different component subcommand sets depending on mode.
internal/app/azldev/cmds/component/component_test.go Tests mode-specific command surface and flag registration differences.
internal/app/azldev/cmds/component/cmdoptions.go New shared “construct flags by mode” helpers for component commands.
internal/app/azldev/cmds/component/changed.go Adds mode-dependent component changed behavior and help text.
internal/app/azldev/cmds/component/changed_test.go Tests changed command long help differs by mode.
internal/app/azldev/cmds/component/build.go Uses centralized preparer option selection (mode-aware synthetic history).
internal/app/azldev/app.go Adds --without-lockfile global flag and pre-parse global flags before registration/loading.
internal/app/azldev/app_test.go Adds tests for without-lockfile parsing/idempotence and env plumbing.
internal/app/azldev/agentskill/doc.go Documents mode-specific skill catalogs and template layering.
internal/app/azldev/agentskill/content/withoutlockfile/remove-component.md.tmpl New lock-file-free skill content variant for removing a component.
internal/app/azldev/agentskill/content/withoutlockfile/refresh-upstream-commit.md.tmpl New lock-file-free skill content variant for refreshing upstream commits.
internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl New lock-file-free skill content variant for component TOML guidance.
internal/app/azldev/agentskill/content/withoutlockfile/build-component.md.tmpl New lock-file-free skill content variant for build workflow guidance.
internal/app/azldev/agentskill/content/withoutlockfile/azldev.md.tmpl New lock-file-free azldev overview skill content variant.
internal/app/azldev/agentskill/content/withoutlockfile/add-component.md.tmpl New lock-file-free skill content variant for adding a component.
internal/app/azldev/agentskill/agentskill_test.go Tests for catalog content/skill selection correctness by mode.
internal/app/azldev/agentskill/agentskill_internal_test.go Updates internal instruction rendering tests for catalog-based APIs.
docs/user/reference/config/project.md Clarifies rendered output is dist-git directories.
docs/user/reference/config/config-file.md Adds preview note about include ordering for generated upstream commits.
docs/user/reference/config/components.md Deprecates skip-file-filter in docs and clarifies rpmautospec/provenance behavior.
docs/user/reference/cli/azldev.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_version.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_repo.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_repo_query.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_project.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_project_new.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_project_init.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_package.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_package_list.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_test.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_list.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_inject-files.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_customize.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_build.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_image_boot.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_docs.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_docs_markdown.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_docs_agent.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_docs_agent_show.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_docs_agent_install.md CLI docs updated for new global flag and updated synopsis text.
docs/user/reference/cli/azldev_config.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_config_generate-schema.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_config_dump.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component.md CLI docs updated for new global flag and render wording.
docs/user/reference/cli/azldev_component_update.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_test.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_render.md CLI docs updated to describe dist-git preservation and new global flag.
docs/user/reference/cli/azldev_component_query.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_prepare-sources.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_list.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_history.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_diff-sources.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_changed.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_build.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_component_add.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_completion.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_completion_zsh.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_completion_powershell.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_completion_fish.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_completion_bash.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_wget.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_mock.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_mock_shell.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_mock_build-rpms.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_mcp.md CLI docs updated for new global flag.
docs/user/reference/cli/azldev_advanced_download-sources.md CLI docs updated for new global flag.
docs/user/README.md Adds how-to link for preview lock-file-free mode.
docs/user/how-to/preview-without-lockfile.md New how-to guide describing preview mode behavior and workflow.
docs/user/explanation/config-system.md Adds preview note about override-merge semantics and deferred validation.
CHANGELOG.md Adds Unreleased changelog entries for preview mode + behavioral changes.
.github/instructions/agent-skills.instructions.md Documents mode-specific skill/catalog behavior and new binding.
Review details

Suppressed comments (1)

internal/app/azldev/agentskill/content/withoutlockfile/comp-toml.md.tmpl:91

  • This section describes render.skip-file-filter as if it changes render behavior, but in this PR the setting is explicitly deprecated/ignored (render always preserves every file in the prepared dist-git dir). The skill should match the actual behavior so users don’t try to rely on a no-op setting.
  • Files reviewed: 140/140 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +28 to +35
| Field | Purpose |
| --- | --- |
| `spec` | where the spec comes from (see below) |
| `overlays` / `overlay-files` | targeted spec/source edits (see the `azldev-overlays` skill) |
| `build.defines` / `build.with` / `build.without` | RPM macro and bcond build tweaks |
| `release.calculation` | how the `Release` tag is managed |
| `render.skip-file-filter` | rendering edge-case escape hatch |

Comment on lines +29 to +36
assert.Equal(t, `# This file was generated by 'azldev component refresh-upstream-commit'
# Do not edit this file, changes will be lost
# For more details see 'azldev component refresh-upstream-commit --help'
[components]
[components.bash]
[components.bash.spec]
upstream-commit = 'abcdef1234567'
`, string(data))
Copilot AI review requested due to automatic review settings September 9, 2026 13:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pre-parsing global flags is not actually idempotent as documented (state can stick across calls), and a newly added merge error message uses incorrect context, both of which should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 140/140 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment on lines +494 to +500
func (a *App) PreParseGlobalFlags(args []string) {
// Reset accumulating state so repeated calls are idempotent.
a.configFiles = nil

a.withoutLockfile = parseWithoutLockfileFlag(args)
a.handParseConfigFlags(args)
}
Comment on lines +512 to +515
err := mergo.Merge(c, &otherTopLevel, mergo.WithOverride, mergo.WithAppendSlice)
if err != nil {
return fmt.Errorf("failed to merge project info:\n%w", err)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants