Skip to content

Correlate upstream 400s with request tool surface and negotiate MCP protocol version - #8861

Merged
lpcox merged 3 commits into
mainfrom
copilot/awf-fix-gpt-5-6-models
Sep 22, 2026
Merged

lpcox merged 3 commits into
mainfrom
copilot/awf-fix-gpt-5-6-models

Conversation

Copilot AI commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Copilot CLI 1.0.83 requests produce immediate zero-token 400s. Two gaps block root-causing it: the upstream_error_response diagnostic records the failing response but nothing about the tool definitions that were sent, and enclave-mcp-server answers initialize with a hardcoded protocol revision regardless of what the client negotiated (1.0.83 negotiates 2025-11-25).

api-proxy: request tool-surface diagnostics

  • summarizeRequestTools() (upstream-response.js) parses the request body's tools array, handling the chat-completions ({type:'function', function:{name}}) and Responses/Anthropic ({name}) shapes.
  • Summary flows into upstream_error_response via both error paths — the buffered-400 retry path (upstream-retry.js) and the streaming/piped path — as:
{
  "event": "upstream_error_response", "status": 400, "model": "gpt-5.6-sol",
  "request_tool_count": 3,
  "request_tool_names": ["github-list_issues", "safeoutputs-create_issue"],
  "request_unnamed_tool_count": 1
}
  • Names and counts only, capped at 32 names — no descriptions, schemas, or message content, so the existing redaction posture is unchanged. request_unnamed_tool_count is the signal for malformed tool entries, the suspected 400 trigger.

enclave-mcp-server: protocol version negotiation

  • initialize now echoes the client's requested revision when it is one of 2025-11-25, 2025-06-18, 2025-03-26, else returns the newest supported (2025-11-25, previously the fixed 2025-06-18).
  • The server implements only initialize, tools/list, tools/call, which are identical across those revisions, so accepting all three is a wire-compatible widening rather than a semantics change.

Docs

  • containers/api-proxy/README.md documents the new fields and the existing AWF_MAX_ERROR_RESPONSE_CAPTURE_BYTES body capture.
  • docs/enclaves-architecture.md documents the negotiated revision set.

The -32022 rejections in the original report come from the github and safeoutputs MCP servers, which live in gh-aw; this change fixes AWF's own server and supplies the payload correlation needed to confirm or rule out the MCP causal chain for the rest.

Copilot AI changed the title [WIP] Fix Copilot CLI 1.0.83 model request errors Correlate upstream 400s with request tool surface and negotiate MCP protocol version Sep 22, 2026
Copilot AI requested a review from lpcox September 22, 2026 02:28
@lpcox
lpcox marked this pull request as ready for review September 22, 2026 02:32
Copilot AI balanced review requested due to automatic review settings September 22, 2026 02:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Advertised 2025-03-26 support lacks required batch handling, and malformed empty tool names evade the new diagnostic.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity · 2 Low severity

Open (3)
What changed in this PR

Adds diagnostics for upstream 400 responses and broadens enclave MCP protocol negotiation.

Changes:

  • Logs bounded tool names/counts with upstream errors.
  • Negotiates three MCP protocol revisions.
  • Adds tests and documentation.
File Description
containers/​api-proxy/​README.md Documents upstream diagnostics.
containers/​api-proxy/​upstream-log.js Adds tool-summary log fields.
containers/​api-proxy/​upstream-response.js Extracts request tool metadata.
containers/​api-proxy/​upstream-response.test.js Tests tool diagnostics.
containers/​api-proxy/​upstream-retry.js Propagates summaries through 400 handling.
containers/​enclave/​mcp-server/​mcp-protocol.js Adds protocol negotiation.
docs/​enclaves-architecture.md Documents supported revisions.
src/​enclave/​mcp-server.test.ts Tests protocol negotiation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

const SUPPORTED_MCP_PROTOCOL_VERSIONS = Object.freeze([
'2025-11-25',
'2025-06-18',
'2025-03-26',
Comment thread containers/api-proxy/README.md Outdated

## Upstream error diagnostics

Every non-2xx upstream response is logged as an `upstream_error_response`
Comment on lines +76 to +85
function extractToolName(tool) {
if (!tool || typeof tool !== 'object') return null;
// OpenAI/Copilot chat-completions shape: { type: 'function', function: { name } }
if (tool.function && typeof tool.function === 'object' && typeof tool.function.name === 'string') {
return tool.function.name;
}
// Responses API and Anthropic shape: { type: 'function'|..., name }
if (typeof tool.name === 'string') return tool.name;
return null;
}
@lpcox

lpcox commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed the review feedback in commit 8d19e38: empty/whitespace tool names now count as unnamed, the README now scopes diagnostics to terminal non-2xx responses, and 2025-03-26 is no longer advertised until batch handling is implemented.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

  1. Download the docs-preview-pr-8861 artifact from the workflow run
  2. Unzip and open index.html in your browser

Built from commit 5ffda84

@lpcox
lpcox enabled auto-merge (squash) September 22, 2026 02:44
@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8861

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #8861

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Security Guard completed successfully!

Security review of PR #8861 complete. No security-weakening changes found. The PR adds tool surface diagnostics for error correlation and MCP protocol version negotiation — both changes maintain security boundaries with proper input validation and data sanitization.

Generated by Security Guard for #8861

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • msfeed25.pkgs.visualstudio.com
  • update.googleapis.com
  • www.google.com
  • www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

@github-actions

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 93.10% 93.13% 📈 +0.03%
Statements 91.68% 91.72% 📈 +0.04%
Functions 91.79% 91.83% 📈 +0.04%
Branches 85.25% 85.27% 📈 +0.02%
📁 Per-file Coverage Changes (2 files)
File Lines (Before → After) Statements (Before → After)
src/log-directory-setup.ts 96.2% → 100.0% (+3.78%) 96.3% → 100.0% (+3.71%)
src/commands/predownload.ts 93.3% → 100.0% (+6.67%) 91.3% → 100.0% (+8.70%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Cloud Hypervisor + Copilot

  • ✅ PASS: list_pull_requests (github/gh-aw-firewall) succeeded via MCP
  • ✅ PASS: https://github.com → 200
  • ✅ PASS: wrote/read /tmp/gh-aw/agent/smoke-cloud-hypervisor-35680543213.txt
  • ✅ PASS: (example.com/redacted) → 000 (blocked)

All 4 checks passed.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
gh CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8861 · claude · haiku45 · 56 AIC · ⊞ 4.7K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox Network isolation smoke test results:

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (github.com) reachable: allowed=200
  • ✅ Blocked domain (example.com) denied: 403 via proxy, blocked

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode

PASS — All tests successful

  • MCP connectivity: ✅ (2 merged PRs fetched)
  • GitHub.com HTTP: ✅ (200)
  • File I/O: ✅ (smoke test file created and verified)
  • BYOK inference: ✅ (running in direct BYOK mode via api-proxy → api.githubcopilot.com)

Status: Direct BYOK mode (COPILOT_PROVIDER_API_KEY) operational

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

📡 Smoke Test: API Proxy OpenTelemetry Tracing — Results

Scenario Result Details
1. Module Loading otel.js loaded; isEnabled()true; exports 15 functions incl. startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown
2. Test Suite 3 suites, 68/68 tests passed (otel.test.js, otel-fanout.test.js, otel-workload-identity.test.js)
3. Env Var Forwarding env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID + GITHUB_AW_OTEL_PARENT_SPAN_ID to agent; api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, trace ID, and parent span ID to api-proxy
4. Token Tracker Integration token-tracker-http.js exposes onUsage callback as the OTEL hook point
5. OTEL Diagnostics ⚠️ (expected) No otel.jsonl span file under the api-proxy sandbox log dir — expected since this smoke run made no proxied LLM API calls through api-proxy; workflow-level trace (/tmp/gh-aw/otel.jsonl) is present (1 span)

Overall: All tracing scenarios validated. No regressions detected in OTEL span creation, GenAI token attributes, parent-context propagation, or env-var wiring.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis (PING): ✅ PONG
  • PostgreSQL (pg_isready): ✅ accepting connections
  • PostgreSQL (SELECT 1): ✅ 1

Overall: PASS

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.21.0 v22.23.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: FAILED — Node.js version mismatch between host and chroot environments. smoke-chroot label not applied.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Maven's default local repository (~/.m2/repository) was not writable for the runner user in this environment (owned by root), causing an initial LocalRepositoryNotAccessibleException on Java compile. Worked around by pointing <localRepository> in the Maven settings to a writable path (/tmp/gh-aw/agent/m2-repo); this is an environment/permissions quirk unrelated to the firewall itself, all Java builds/tests then passed cleanly through the Squid proxy.

All 18 projects across all 8 ecosystems built/installed and passed their test suites successfully through the AWF firewall proxy.

Generated by Build Test Suite for #8861 · copilot · auto · 46.7 AIC · ⊞ 11.8K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

#8854 Sync D14 runner-doctor catalog with #8801 Cloud Hypervisor retry-fix follow-up
#8853 feat(routing): add the routing configuration surface and contracts
GitHub MCP review/detail: ✅
Playwright title check: ✅
File write/read: ✅
Discussion lookup/comment: ✅
Build AWF: ✅
Overall: PASS

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • msfeed25.pkgs.visualstudio.com
  • update.googleapis.com
  • www.google.com
  • www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@lpcox
lpcox merged commit 9cde10a into main Sep 22, 2026
152 of 156 checks passed
@lpcox
lpcox deleted the copilot/awf-fix-gpt-5-6-models branch September 22, 2026 02:57

This branch was successfully deployed

1 active deployment
aoai-model 8d19e389 Deployed Sep 22, 2026 by lpcox via conclusion #1728
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[awf] api-proxy: Copilot CLI 1.0.83 gpt-5.6-* models fail with zero-token http_400, response body not logged

3 participants