compression: whole-record items[] crushing (ctxzip v0.3.0) + keep_patterns learning loop + relaxed tool caps#279
Open
initializ-mk wants to merge 6 commits into
Open
compression: whole-record items[] crushing (ctxzip v0.3.0) + keep_patterns learning loop + relaxed tool caps#279initializ-mk wants to merge 6 commits into
initializ-mk wants to merge 6 commits into
Conversation
kubectl -o json ({"items": [...]}) previously compressed via line-mode
text dedup, scattering records. ctxzip v0.2.0's JSON object walker
routes array fields through whole-record row selection: anomalous
records survive intact, the rest offload as a retrievable array of
complete records. Measured on the run-004 shape at 251KB: 62,754 ->
1,498 tokens (97.6%), crash record whole. No forge-side code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rievals Item 3 of the post-#241 roadmap: every context_expand hit means compression dropped something a model needed. Captured in-process at expansion time (audit events go to stderr/sinks the runtime cannot assume it can read back): - extractCandidates mines the retrieved original for domain-state token shapes (CamelCase with 2+ humps, ALLCAPS identifiers), excluding tokens the error floor or configured keep_patterns already protect — those were KEPT, so they cannot be why the model expanded (uses ctxzip's newly exported crush.IsErrorLike; pseudo-version bump, no release per current cadence). - suggestionStore persists counts across restarts in .forge/ctxzip-suggestions.json (atomic writes, 512-entry bound with lowest-count eviction, corruption starts fresh — advisory data never fails anything). - At 3 distinct expansions a pattern is surfaced exactly once: a context_pattern_suggested audit event (pattern, expansions, tools) plus a log line. - New `forge compression suggestions` renders the accumulated candidates as a table plus a paste-ready compression.keep_patterns block for entries past the threshold. Suggestions are advisory by design — the operator reviews before adopting; a token retrieved often is evidence, not proof. Tests: extraction shapes + floor/config exclusions (including the subtlety that ImagePullBackOff is floor-kept via "backoff" and must NOT be suggested), threshold/once semantics, persistence round-trip, bounded store, and end-to-end through hook -> marker -> context_expand -> suggestion event. Docs synced (context-compression, audit-logging, cli-reference, forge.md skill); link check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
initializ-mk
changed the title
…tream Containerized agents lose .forge/ on pod restart, and multi-replica Deployments count expansions independently — the local suggestions file alone is restart-fragile and dilution-prone (review question on item 3). Every context_expanded audit event now carries the producing tool and the top mined keep-pattern candidates (<=5, highest-frequency first, mined once and shared with the local flywheel). A platform consuming the audit export can rebuild counting fleet-wide from the stream alone; the local file becomes a single-agent convenience rather than the source of truth. context_pattern_suggested's tools field is now a proper array (pre-merge, no consumers yet). Docs: new "Persistence and containers" section in the learning-loop doc (volume guidance: store_path on a per-replica mount, emptyDir vs PVC; audit-stream aggregation recommended for fleets; the CCR store itself needs no care — TTL + re-run-the-tool covers restarts). Event field inventories updated in audit-logging and the forge.md skill. End-to-end test asserts all three expansion events carry the producing tool and the domain token within the candidate cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s 12-20%) kubectl describe / get -o yaml — the weakest live shape — now folds boring subtrees (managedFields, env lists, long scalar values) behind per-subtree markers via indentation-tree analysis; crashing container sections are protected by the shared error floor with zero k8s-specific knowledge. Measured through the envelope path: crashing describe-pod 75% (was 12-20% live), get -o yaml with managedFields 49%. Doc note in context-compression.md. No forge-side code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 11, 2026
Open
Tool-side caps cut data before the compression hook ever sees it — the live k8s run showed grep_search's 50-match default silently dropping the one CrashLoopBackOff row at match #78, producing a confidently wrong triage. Compression makes those caps the wrong place to economize: the cut destroys data that would have been offloaded losslessly. The executor now stamps the tool context with tools.WithRelaxedLimits when DeferToolResultTruncation is set (i.e. compression on), and the destructive cap sites consult it: - builtins TruncateOutputCtx: 2000 lines / 50KB → 16x (32000 / 800KB) - grep_search default max_results: 50 → 500 (explicit args untouched) - file_read default limit: 2000 → 32000 lines - MCP adapter maxResultChars: 16x, bounded 4MB absolute cli_execute's 1MB cap is deliberately unchanged — it reports truncated:true honestly. The loop's post-hook cap and 16x/4MB pre-hook safety ceiling still bound everything downstream, so relaxed never means unbounded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the interim pseudo-version with the tagged release (YAML crusher, categorical drop summaries, IsErrorLike export). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
initializ-mk
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two roadmap items from the post-#241 backlog, on one branch.
1. ctxzip v0.2.0 — whole-record compression for kubectl-style
items[]JSONkubectl get -o json(and every list-API shape like it) is an object wrapping an array; it previously fell through to line-mode text dedup, scattering records. ctxzip v0.2.0's depth-limited JSON object walker routes array fields through whole-record row selection: anomalous records survive intact (the error floor sees full record text), the rest offload as a retrievable array of complete records. Also: categorical drop summaries — markers/sentinels now say what was offloaded ([offloaded: 139 running, 24 batch completed],status.phase: 118 Running, 2 Succeeded), so models can answer count questions without retrieving.Measured on the live run-004 shape at 251KB: 62,754 → 1,498 tokens (97.6%), crashing pod record kept whole. No forge-side code change for this part.
2. keep_patterns learning loop (
forge-core/compress/feedback.go)Every
context_expandhit means compression dropped something a model needed. Captured in-process at expansion time (audit events go to stderr/sinks the runtime can't assume it can read back):keep_patternsalready protect — those were kept, so they can't be why the model expanded (uses ctxzip's newly exportedcrush.IsErrorLike; pinned as a pseudo-version pending the next batched release).forge/ctxzip-suggestions.json(atomic writes, 512-entry bound, corruption starts fresh — advisory data never fails anything)context_pattern_suggestedaudit event + log lineforge compression suggestionsrenders the candidates as a table plus a paste-readycompression.keep_patternsblockSuggestions are advisory by design — evidence, not proof; the operator reviews before adopting.
Tests
Extraction shapes + floor/config exclusions (including the subtlety that
ImagePullBackOffis floor-kept viabackoffand must NOT be suggested), threshold/once semantics, persistence round-trip, bounded store, end-to-end hook → marker →context_expand→ suggestion event. Fullforge-cli+forge-coresuites and lint green. Docs synced per the/sync-docsmapping (context-compression, audit-logging, cli-reference, forge.md knowledge skill); link check clean.🤖 Generated with Claude Code