Skip to content

Fix daily AIC snapshot completeness and quota handling - #60136

Merged
pelikhan merged 8 commits into
mainfrom
copilot/fix-aic-persist-scan-observations
Sep 11, 2026
Merged

Fix daily AIC snapshot completeness and quota handling#60136
pelikhan merged 8 commits into
mainfrom
copilot/fix-aic-persist-scan-observations

Conversation

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Daily AIC accounting could reuse incomplete observations, repeatedly fetch historical artifacts, and report partial totals as under budget after quota failures. Conclusion jobs also appended to restored caches rather than preserving the activation scan.

  • Complete observations

    • Publish versioned activation scan snapshots containing every resolved run.
    • Revalidate repository, workflow, attempt, and completion metadata before reuse.
    • Keep authoritative totals derived from fresh workflow history.
  • Strict accounting

    • Validate agent, detection, and evals usage independently.
    • Preserve valid usage across reruns and reject malformed numeric values.
    • Avoid double-counting raw usage and overlapping summaries.
  • Quota handling

    • Observe rate-limit headers on REST and native artifact requests.
    • Stop after the first failed historical read.
    • Prevent incomplete scans from producing under_budget.
  • Compiler and runtime

    • Move snapshot publication into activation and remove conclusion cache writes.
    • Collect evals usage from topology-aware default and ARC/DinD paths.
    • Regenerate affected workflow lock files and document recompilation requirements.


✨ PR Review Safe Output Test - Run 34562419664

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • clients2.google.com
  • mtalk.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"
    - "mtalk.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · claude · sonnet46 · 69.7 AIC · ⌖ 18.3 AIC · ⊞ 7.8K ·
Comment /smoke-claude to run again

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix persistence of complete scan observations and stop quota fan-out Fix daily AIC snapshot completeness and quota handling Sep 11, 2026
Copilot AI requested a review from pelikhan September 11, 2026 02:01
@pelikhan
pelikhan marked this pull request as ready for review September 11, 2026 02:19
Copilot AI balanced review requested due to automatic review settings September 11, 2026 02:19
@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "proxy.golang.org"

See Network Configuration for more information.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer failed during the skills-based review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Warning

Firewall blocked 4 domains

The following domains were blocked by the firewall during workflow execution:

  • codeload.github.com
  • github.com
  • patchdiff.githubusercontent.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "codeload.github.com"
    - "github.com"
    - "patchdiff.githubusercontent.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch PR file list

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer completed successfully!

Cannot run Ponytail review: /tmp/gh-aw/agent/pr-diff.patch is empty.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Ponytail Reviewer for #60136

@github-actions

Copy link
Copy Markdown
Contributor
🏗️ ADR Required — draft added for PR #60136

I required ADR enforcement because this PR adds more than 100 new lines in default business-logic directories (pkg/) according to the prefetch summary (default_business_additions: 122).

Evidence used

  • PR title: Fix daily AIC snapshot completeness and quota handling
  • PR body describes an architectural shift from conclusion-time cache reuse to complete, versioned activation scan snapshots, stricter quota failure handling, and independent agent/detection/evals accounting.
  • Changed files include broad runtime/compiler updates and regenerated workflow lock files, with pr.diff unavailable because the PR touches 320 files, so the gate relied on pr.json and pr-files.json.
  • No existing ADR was present in the PR body or under docs/adr/ for this decision.

Action taken

  • Added draft ADR: docs/adr/60136-fix-daily-aic-snapshot-completeness-and-quota-handling.md

Next action for the author

  • Review and refine the draft ADR so it accurately captures the intended long-term decision and trade-offs for daily AIC accounting persistence and quota handling.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "proxy.golang.org"

See Network Configuration for more information.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · gpt54 · 28 AIC · ⊞ 10.1K ·
Comment /review to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Outstanding quota, cache validation, workflow filtering, and evals accounting findings block approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes daily AIC accounting with verified activation snapshots, strict quota handling, and complete evals usage collection.

Changes:

  • Moves snapshot restoration and publication to activation.
  • Adds strict component validation and quota-aware scans.
  • Updates evals collection, documentation, tests, and workflow lockfiles.
File summaries
File Summary
pkg/workflow/notify_comment.go Updates usage uploads and removes conclusion cache writes.
pkg/workflow/notify_comment_test.go Tests notification and upload changes.
pkg/workflow/notify_comment_conclusion_helpers.go Updates conclusion helpers.
pkg/workflow/evals_steps.go Collects topology-aware evals token usage.
pkg/workflow/evals_steps_test.go Tests evals usage collection.
pkg/workflow/daily_aic_workflow_guardrail_test.go Adds daily AIC guardrail coverage.
pkg/workflow/compiler_activation_daily_aic.go Generates activation restore and publication steps.
docs/src/content/docs/reference/cost-management.md Documents snapshot and quota behavior.
actions/setup/sh/collect_usage_artifact_files.sh Repacks usage artifacts.
actions/setup/js/restore_aic_scan_cache.cjs Restores trusted activation snapshots.
actions/setup/js/daily_aic_scan.cjs Scans and validates historical AIC usage.
actions/setup/js/daily_aic_component_coverage.cjs Validates usage component coverage.
actions/setup/js/daily_aic_cache_helpers.cjs Supports cache validation and reuse.
actions/setup/js/daily_aic_api_budget.cjs Handles API budgets and quota state.
actions/setup/js/artifact_client.cjs Observes artifact API quota responses.
.github/workflows/workflow-normalizer.lock.yml Regenerated workflow lockfile.
.github/workflows/windows-grower.lock.yml Regenerated workflow lockfile.
.github/workflows/weekly-network-domains-audit.lock.yml Regenerated workflow lockfile.
.github/workflows/weekly-issue-summary.lock.yml Regenerated workflow lockfile.
.github/workflows/super-linter.lock.yml Regenerated workflow lockfile.
.github/workflows/step-name-alignment.lock.yml Regenerated workflow lockfile.
.github/workflows/static-analysis-report.lock.yml Regenerated workflow lockfile.
.github/workflows/spec-extractor.lock.yml Regenerated workflow lockfile.
.github/workflows/smoke-update-cross-repo-pr.lock.yml Regenerated workflow lockfile.
.github/workflows/smoke-pi.lock.yml Regenerated workflow lockfile.
.github/workflows/smoke-copilot-arm.lock.yml Regenerated workflow lockfile.
.github/workflows/smoke-codex.lock.yml Regenerated workflow lockfile.
.github/workflows/smoke-claude.lock.yml Regenerated workflow lockfile.
.github/workflows/slide-deck-maintainer.lock.yml Regenerated workflow lockfile.
.github/workflows/scout.lock.yml Regenerated workflow lockfile.
.github/workflows/schema-consistency-checker.lock.yml Regenerated workflow lockfile.
.github/workflows/safe-output-health.lock.yml Regenerated workflow lockfile.
.github/workflows/repository-quality-improver.lock.yml Regenerated workflow lockfile.
.github/workflows/refactoring-cadence.lock.yml Regenerated workflow lockfile.
.github/workflows/python-data-charts.lock.yml Regenerated workflow lockfile.
.github/workflows/prompt-clustering-analysis.lock.yml Regenerated workflow lockfile.
.github/workflows/portfolio-analyst.lock.yml Regenerated workflow lockfile.
.github/workflows/pdf-summary.lock.yml Regenerated workflow lockfile.
.github/workflows/outcome-collector.lock.yml Regenerated workflow lockfile.
.github/workflows/org-health-report.lock.yml Regenerated workflow lockfile.
.github/workflows/mcp-inspector.lock.yml Regenerated workflow lockfile.
.github/workflows/lockfile-stats.lock.yml Regenerated workflow lockfile.
.github/workflows/linter-miner.lock.yml Regenerated workflow lockfile.
.github/workflows/jsweep.lock.yml Regenerated workflow lockfile.
.github/workflows/instructions-janitor.lock.yml Regenerated workflow lockfile.
.github/workflows/go-fan.lock.yml Regenerated workflow lockfile.
.github/workflows/github-mcp-tools-report.lock.yml Regenerated workflow lockfile.
.github/workflows/github-mcp-structural-analysis.lock.yml Regenerated workflow lockfile.
.github/workflows/firewall-escape.lock.yml Regenerated workflow lockfile.
.github/workflows/feature-grower.lock.yml Regenerated workflow lockfile.
.github/workflows/dead-code-remover.lock.yml Regenerated workflow lockfile.
.github/workflows/daily-max-ai-credits-test.lock.yml Regenerated workflow lockfile.
.github/workflows/daily-formal-spec-verifier.lock.yml Regenerated workflow lockfile.
.github/workflows/daily-arxiv-researcher.lock.yml Regenerated workflow lockfile.
.github/workflows/chaos-pr-bundle-fuzzer.lock.yml Regenerated workflow lockfile.
.changeset/patch-complete-daily-aic-observations.md Adds the release note for complete observations.
Review details
  • Files reviewed: 101/321 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread actions/setup/js/check_daily_aic_workflow_guardrail.cjs Outdated
@pelikhan

Copy link
Copy Markdown
Collaborator

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix https://github.com/github/gh-aw/actions/runs/34554629146/job/103128887174#step:9:1

Fixed in 5bef55b. Regenerated the stale WASM golden fixtures for the new activation-phase AIC snapshot flow; the targeted failure, full WASM golden suite, and final pre-push validation now pass.

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Smoke OTEL completed successfully!

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Smoke OTEL

…st-scan-observations

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot The latest review still shows unresolved blocking findings on quota handling and accounting here. Please address the outstanding review feedback, verify the failed automation signals (`activa...

Addressed in f408b5a and refreshed with main in 1055b0f. The exact-threshold accounting blocker now reports exceeded and has regression coverage. I inspected all three jobs: activation failed closed on incomplete historical component evidence as designed; Pi failed on an unsupported provider model; Gemini failed because cloud-hypervisor does not support the requested host access. Full local lint, unit/full tests, recompilation, and the final pre-push gate pass.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill before handing this back to maintainers.

Run: https://github.com/github/gh-aw/actions/runs/34576660531

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 23.4 AIC · ⊞ 8.9K ·
Comment /souschef to run again

…st-scan-observations

# Conflicts:
#	.github/workflows/release.lock.yml

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill before ...

Completed the PR-finisher pass. Run 34576660531 and all 10 of its jobs succeeded. I merged current main in 0703f266, regenerated the conflicted workflow lock, and the final repository gate passed. The substantive review finding is fixed and replied to; the only other open threads are non-actionable smoke-test praise. The branch is clean and current, with maintainer review/CI now the remaining merge gates.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill before handing this back to maintainers.

Run: https://github.com/github/gh-aw/actions/runs/34579367023

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 17.4 AIC · ⊞ 9.3K ·
Comment /souschef to run again

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill before ...

Completed the PR-finisher pass. Run 34579367023 succeeded across all 10 jobs, the branch already contains current main, and the full local validation sequence plus final repository gate passed. The only open review threads are the already-fixed/outdated threshold finding and two non-actionable smoke-test praise comments; maintainer review and CI on the current head remain the final merge gates.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot These items still need follow-up on this PR:

Please resolve the remaining feedback, refresh the branch if needed, and run the pr-finisher skill before handing this back to maintainers.

Run: https://github.com/github/gh-aw/actions/runs/34581583531

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 45.9 AIC · ⊞ 9.3K ·
Comment /souschef to run again

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot These items still need follow-up on this PR:

Completed the PR-finisher pass. The threshold thread is resolved, and I replied directly to both GitHub Actions smoke-test threads confirming that no code changes are required. Run 34581583531 passed all 10 jobs; the branch is current with main, and formatting, lint, impacted/full tests, recompilation, and the final repository gate all pass. The available GitHub tooling cannot mutate review-thread resolution state, so a maintainer must click Resolve on the two acknowledged smoke-test threads before merge.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill before handing this back to maintainers.

Run: https://github.com/github/gh-aw/actions/runs/34589658410

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 42.4 AIC · ⊞ 8.9K ·
Comment /souschef to run again

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot The compact report still shows this PR needs follow-up. Please inspect any remaining blocking review or automation signals, refresh the branch if needed, and run the pr-finisher skill befor...

Completed the PR-finisher pass. Run 34589658410 passed all 10 jobs, all review threads are now resolved, and the branch remains current with main. Formatting, lint, impacted/full tests, recompilation, and the final repository gate all pass; no additional code changes were needed. Maintainer approval remains the only visible merge gate.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

…st-scan-observations

# Conflicts:
#	.github/workflows/release.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Resolved in eb9a2d4. Merged current main as a true two-parent merge commit and regenerated all conflicted workflow lock files from their Markdown sources. Formatting, lint, impacted/full tests, recompilation, and the final repository gate pass; the branch is clean and contains current main.

@pelikhan
pelikhan merged commit 7fc8f64 into main Sep 11, 2026
@pelikhan
pelikhan deleted the copilot/fix-aic-persist-scan-observations branch September 11, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(aic): Persist complete scan observations and stop quota fan-out

4 participants