Activate task-level model routing in the API proxy server - #8966
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
|
@copilot make sure that all decisions by the model router are logged in a structured log that will be easy for an agent to parse and easy to export to a dashboard |
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Added structured decision logging for the enforcement layer in 8e3f8c2 (routing-enforcement.js). Every screenRequest call now records |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unpublishable bootstrap failures can leave the proxy running without a result, and rejected bodies are incorrectly logged as admitted.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Activates task-level model routing in the API proxy lifecycle and enforces selected models on live requests.
Changes:
- Adds secure routing-session setup, result publication, and shutdown handling.
- Integrates routing enforcement into HTTP and WebSocket handling.
- Adds lifecycle, enforcement, and packaging tests.
| File | Description |
|---|---|
containers/api-proxy/startup.test.js |
Tests routing lifecycle ordering and exit handling. |
containers/api-proxy/startup.js |
Starts, drains, and completes routing sessions. |
containers/api-proxy/server.js |
Constructs and injects the production routing session. |
containers/api-proxy/server-factory.test.js |
Tests request and upgrade enforcement wiring. |
containers/api-proxy/server-factory.js |
Screens routed traffic and applies routing transforms. |
containers/api-proxy/routing-runtime.test.js |
Tests secure file handling and session publication. |
containers/api-proxy/routing-runtime.js |
Implements production routing orchestration. |
containers/api-proxy/routing-enforcement.test.js |
Tests routing decision telemetry. |
containers/api-proxy/routing-enforcement.js |
Adds structured admit/reject observations. |
containers/api-proxy/Dockerfile |
Packages routing runtime modules. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if (!privateFs.existsSync(path.join(outputDir, 'selection.json')) && | ||
| !privateFs.existsSync(path.join(outputDir, 'failure.json'))) { | ||
| publishRoutingResult(outputDir, 'failure.json', result.failure); | ||
| } |
| safeRecordDecision(observer, { | ||
| decision: 'admit', | ||
| reason: 'selected_model_pinned', | ||
| method: req.method, | ||
| pathname, | ||
| selected_model: selection.choice.model, | ||
| selected_effort: selection.choice.effort ?? null, | ||
| }); |
|
@copilot address the review feedback |
…ody validation passes Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Addressed both review threads in 3dc... (see below commits): the bootstrap-failure publish is now guarded with
Fixed in 1d980a1: bootstrap-failure publish is now guarded with |
|
✅ Copilot review passed with no inline comments. @copilot Add the |
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
|
🔑 Smoke Copilot BYOK is testing direct BYOK mode on this pull request... |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "o205451.ingest.us.sentry.io"See Network Configuration for more information.
|
|
🔌 Smoke Services — All services reachable! ✅
|
github-actions
Bot
added
the
smoke-copilot-network-isolation
Smoke Test: Copilot BYOK (Direct) Mode✅ GitHub MCP: Connected (PR #8975, #8970 verified) Status: PASS 🟢
|
Smoke Test: Cloud Hypervisor + Copilot
All checks passed. Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"
- "github.com"See Network Configuration for more information.
|
|
Smoke Test: Services Connectivity
Overall: PASS
|
Smoke Test: API Proxy OpenTelemetry Tracing
Overall: ✅ All scenarios passed. OTEL module initialization, span/attribute logic, env var propagation, and token-tracker hook points are all working as designed; graceful degradation confirmed (no errors, clean exports). Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "o205451.ingest.us.sentry.io"See Network Configuration for more information.
|
|
Smoke Test: Copilot Engine —
Overall: PASS
|
Smoke Test: Claude Engine Validation
Overall result: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "api.anthropic.com"See Network Configuration for more information.
|
Chroot Runtime Version Comparison
Result: ❌ Not all versions match — Node.js version differs between host and chroot environment.
|
Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "ab.chatgpt.com"
- "msfeed25.pkgs.visualstudio.com"See Network Configuration for more information.
|
Gemini Smoke Test Results
Overall Status: PASS Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "github.com"
- "play.googleapis.com"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS All 18 project builds/installs and test suites completed successfully. No errors encountered. Java required a local Maven repository path ( Warning Firewall blocked 8 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "api.nuget.org"
- "bun.sh"
- "dc.services.visualstudio.com"
- "deno.land"
- "dl.deno.land"
- "github.com"
- "releaseassets.githubusercontent.com"
- "repo.maven.apache.org"See Network Configuration for more information.
|
Section 13a said the routing candidate-pool foundation was present but not called by the running proxy. PR #8966 (merged) wired the routing controller and request enforcement into the live API proxy server: a routing session now starts after key validation/model discovery and screens every inference request against the selected model/effort. Clarify that host-side staging and validation of apiProxy.routing input (PR #8985) is still open, so the feature is not yet end-to-end functional despite the proxy-side wiring being shipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Section 13a said the routing candidate-pool foundation was present but not called by the running proxy. PR #8966 (merged) wired the routing controller and request enforcement into the live API proxy server: a routing session now starts after key validation/model discovery and screens every inference request against the selected model/effort. Clarify that host-side staging and validation of apiProxy.routing input (PR #8985) is still open, so the feature is not yet end-to-end functional despite the proxy-side wiring being shipped. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The routing controller and enforcement from #8942 exist but nothing constructs them, nothing reads the private files the host watches, and the running proxy never calls them. This wires both ends: a routing session the server owns for the run, started after startup validation and drained on shutdown.
routing-runtime.js(new — the twelfth routing module)AWF_ROUTING_CONFIG, the allow/deny policy env vars, the live Copilot adapter, the discovered model cache, andproxyRequest/checkRateLimit. Returnsnullwhen routing is unconfigured, so an unset deployment constructs nothing.O_NOFOLLOWdescriptor that is size- and link-checked before parsing. The low-level reader preserves filesystem errors; only the public loader translates them into routing codes.selection.json,failure.json,runtime-failure.json,complete.json.runtime-failure.jsonand never throws back into that stream. If that publication fails, the proxy exits 78 rather than leaving the host waiting on a claim nobody verified.Server wiring
server-factory.js— screens inference after health and reflect but before the enabled check, so a rejection never reveals provider configuration; admitted routed requests use the enforcement transform, not the adapter's (which would reintroduce aliases and fallback); upgrades are rejected while routing is active.server.js— constructs the session and passes it to the server factory andbootPrimary.startup.js— routing starts only after key validation and model discovery resolve;shutdown()drains before servers close, andcompleteShutdown()runs after log and telemetry shutdown, exiting 78 if it throws.Dockerfile— copies the twelve routing runtime modules (enforced by the existingdockerfile-copy-coverageguard).Behavior
Without
AWF_ROUTING_CONFIGthe proxy is unchanged. With the block configured, it fails closed — routing cannot load the conversation and every inference request is rejected — until issue 06 supplies the private mounts and the container conversation path. That failure is intentional; no missing value is defaulted.Two review suggestions were deliberately not taken: the 90s deadline is anchored at session construction, and upgrades are rejected for every adapter whenever a session exists. Both are the specified fail-closed contract ("do not enable routed WebSocket requests"); relaxing them is a separate decision.
Tests cover the new runtime module plus the factory and startup wiring, including drain ordering and both exit-78 paths.