feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply

[Mikola Lysenko (mikolalysenko)]

Single combined branch (supersedes #112, #113, #114 — all closed). Branch feat/vendor-overhaul is checked out in the main repo for local testing; main is fully merged in. 14 commits, reviewable commit-by-commit; each landed with its own tests green.

Part 1 — Vendor UX fixes (was #112)

Driven by the Flowise scan --vendor failures (vendored 7/10, --prune never cleaned vendored state):

• Percent-decode purl components: the patches API serves scoped purls encoded (pkg:npm/%40scope/…); the crawler and vendor coordinate parsers now decode them — fixes "package not installed" for scoped packages in both apply and vendor, and stops --prune from seeing encoded manifest entries as always-prunable. Decoding happens after the /-and-@ splits and before the path-safety guards (RED tests prove %2e%2e-style encoded traversal still rejects).
• Vendor auto-force on content mismatch: the vendor stage is a private copy and every apply write path is hash-gated to exactly afterHash, so a beforeHash mismatch no longer fails the vendor — the stage is overwritten with verified patched content, annotated before the confirm prompt, and surfaced as vendor_content_mismatch_overwritten. Missing files still fail closed without --force. First vendor of an already-apply-patched package now reports applied (was a miscounted skip).
• Exact-version override pin takeover (pnpm + yarn berry): a user pin equal to the patched version (Flowise: "tar-fs": "3.1.0") is taken over in place — same key on both pnpm surfaces (map-parity), pinned value recorded as the wiring original, every revert path restores it verbatim. Ranges/selector-chains still refuse.
• Prune lifecycle for vendored packages: scan --prune now reverts manifest-dropped vendored entries, reverts entries whose dependency left the lockfile graph (per-flavor in-use probes; detached exempt; undeterminable → keep) and drops their manifest entries, and sweeps orphan uuid dirs. JSON gc gains revertedVendoredEntries/removedVendorOrphanDirs (+ preview names).

Part 2 — Lockfile-driven discovery + verified auto-fetch (was #113)

Makes scan / scan --vendor / vendor work on a completely fresh clone, across ecosystems:

• Vendor auto-fetch (no flag): a manifest patch with no installed copy stages from its own committed artifact (sha256-verified against the vendor ledger; offline-safe re-vendor) or fetches the lockfile-resolved pristine artifact, verified against the lock's recorded integrity fail-closed before any write:

ecosystem	source	verifier
npm / pnpm / yarn classic / bun	lock resolved URL else conventional registry URL	SRI (strongest of multi-hash); yarn sha1 fragment fallback
yarn berry	registry URL	lock 10c0/ cache-zip checksum, rebuilt from the fetched tarball via the deterministic berry_zip recipe
cargo	static.crates.io	Cargo.lock sha256 of the .crate
golang	module proxy (SOCKET_GOPROXY → GOPROXY → proxy.golang.org), !case escaping	go.sum h1: dirhash in memory — algorithm validated against a live sum.golang.org lookup
composer	dist.url (zipball top dir stripped)	dist.shasum (empty → never fetched)
gem	{remote}/downloads/{gem}	Gemfile.lock CHECKSUMS sha256 (bundler ≥ 2.6)
pypi	uv.lock wheel URL (pure py3-none-any only)	uv.lock wheel sha256

Unverifiable entries are never fetched (vendor_fetch_unverifiable → calm package_not_installed); attempted-and-failed is a distinct vendor_fetch_failed; successes warn vendor_fetched_missing. --offline refuses calmly. Hardened download/extraction: http(s)-only, size/entry caps, traversal-guarded, exec bits preserved.

• Scan lockfile supplement (default-on): discovery reads npm-family (all five flavors), Cargo.lock, go.sum, composer.lock, Gemfile.lock, uv.lock/poetry.lock/pinned requirements.txt and adds not-installed dependencies — [NOT INSTALLED] table marker + stderr note, JSON lockfileOnlyPackages + packages[].notInstalled, counted as scanned for --prune (wiped node_modules no longer prunes lockfile-listed entries), partitioned out of --apply before download (calm skip, exit 0, no manifest writes), passed through to --vendor's auto-fetch. Vendored-ledger entries also stay discoverable pre-install.

Part 3 — Mismatch-tolerant apply (was #114)

A file matching neither beforeHash nor afterHash previously hard-failed the in-place apply (the flatted@3.3.1 bad-baseline case). Now by default:

• stderr warning per file (content_mismatch_overwritten) + Skipped warning event beside the package's Applied event,
• the file is overwritten with the full verified patched content (the partial bsdiff strategy self-disables on a wrong base; archive/blob writes are hash-gated to exactly afterHash),
• missing afterHash blobs are redownloaded by hash on demand (the default --download-mode diff may not have staged them),
• a missing pre-existing file is still a hard error (only --force skips it),
• the new global --strict (env SOCKET_STRICT) restores fail-closed; --force overrides it. Core: apply_package_patch's force: bool became MismatchPolicy { Warn | Strict | Force }.

Test plan

• ~120 new tests across the three parts (unit, in-process, wiremock e2e), RED-first where the bug was reproducible; full inventory in the per-part commit messages.
• cargo test --workspace --all-features: 101/102 suites green — the one red is the pre-existing non-blocking setup_matrix_composer aspirational suite ("BASELINE GAP", untouched here). cargo clippy --workspace --all-features clean.
• Live verification against the public proxy:
  • Flowise: scan --vendor went from "Vendored 0; 2 failed; 1 refused" to 10/10 vendored; pnpm install --frozen-lockfile accepts the takeover surgery; vendored flatted installs at afterHash 6/6.
  • Completely fresh clone (lockfile only, no node_modules): scan --vendor --yes discovers, fetches from registry.npmjs.org integrity-verified, vendors; npm ci installs the patched bytes; second run is already_vendored.
  • pnpm workspace repro of the reported failures: scan --apply --yes now applies 2/2 (flatted warns + applies via blob; the %40-encoded SDK resolves); apply --strict exits 1 with the old error, files untouched.
  • go.sum h1: dirhash verified against sum.golang.org for golang.org/x/text@v0.14.0.

Known upstream issue (not fixed here)

The flatted@3.3.1 patch record (uuid 5cac955f-eab1-4d29-8f4f-c408a6cc9647) has beforeHashes matching no published flatted artifact — to report to the patch team; this PR makes the CLI handle it gracefully either way.

repair owns the vendored-artifact lifecycle (added 2026-06-12)

• In-memory patch blobs: vendoring writes no .socket/blobs|diffs or temp files — content is staged in memory (PatchSources.mem_blobs), and re-runs harvest it from the committed artifact itself. A vendored project's .socket/ holds only manifest.json + vendor/.
• repair rebuilds missing/corrupt vendored artifacts (health = per-file afterHashes + the ledger's whole-file sha256) through the normal vendor backends — wired hot paths rebuild the artifact only, lockfiles stay byte-identical, the ledger entry is not re-recorded. Pristine sources: installed copy (offline-capable) → lockfile-verified registry fetch → the pre-vendor registry fragment recovered from the ledger's wiring originals.
• Full no-ledger reconstruction: when .socket is gone entirely, the lockfile's .socket/vendor/<eco>/<uuid>/… references drive recovery — record from the manifest or the patch API (detached entry with the record embedded), pristine fetched unverified from the conventional registry and the deterministically REBUILT tarball verified against the integrity the rewired lockfile records (tamper ⇒ removed, fail).
• New envelope vocabulary: rebuilt action + summary.rebuilt; error codes vendor_artifact_{missing,corrupt,rebuilt,rebuild_failed,unrepairable}, vendor_uuid_mismatch. repair no longer errors manifest_not_found when vendor traces exist.
• pnpm multi-version fix: vendor edits previously matched "ours" by package NAME only — projects vendoring several versions of one package got sibling overrides clobbered and duplicated lock keys (pnpm hard-rejects). All probes now bind to name@version; half-edited locks fail closed.

Live-verified on Flowise: scan --vendor (16/16) → pnpm install --frozen-lockfile → rm -rf .socket → repair (16/16 reconstructed from the lockfile alone) → frozen install again, all exit 0.

Out of scope (documented)

.npmrc/auth registry config (env overrides + lock-recorded URLs are the escape hatches); pipenv/pdm lock parsing; platform wheels / platform gems; npm v1 legacy locks; range-pin takeover.

🤖 Generated with Claude Code