integration/v0.13.0-rc4: x402 settle-guard, /v1 discovery fix, hermes context config, flow de-flakes, dead-code sweep

.agents/skills/obol-stack-dev/SKILL.md

@@ -2,7 +2,7 @@
 name: obol-stack-dev
 description: CLI-first Obol Stack development and QA runbook. Use when working on obol-stack lifecycle, obol CLI surfaces, x402 seller/buyer tests, live Base Sepolia OBOL smoke, Anvil fork regressions, ERC-8004 registration, LiteLLM paid routing, release-smoke, cloudflared, Renovate image bumps, or remote QA worktrees.
 metadata:
-  version: "3.1.0"
+  version: "3.2.0"
   domain: infrastructure
   role: specialist
   scope: development-and-testing
@@ -67,7 +67,7 @@ OBOL_TOKEN_BASE_SEPOLIA=0x0a09371a8b011d5110656ceBCc70603e53FD2c78
 
 **Release notes**: start from `.github/release-template.md`. Keep generated `What's Changed` / `New Contributors` / `Full Changelog` at the bottom. v0.9.0 is the style reference. No private keys, seed phrases, hostnames, personal paths, or raw bearer tokens.
 
-## Hard-Won Lessons (from release-smoke 2026-05-13)
+## Hard-Won Lessons (release-smoke 2026-05-13 + 2026-07-14)
 
 When the smoke gate goes red, check these first — each was a multi-hour debug:
 
@@ -84,7 +84,9 @@ When the smoke gate goes red, check these first — each was a multi-hour debug:
 | facilitator arm64 image runs amd64 binary | Was an `ObolNetwork/x402-rs` prom-overlay arm64 manifest packaging bug. | **Fixed upstream**: `ObolNetwork/x402-rs#3` (merged 2026-05-13, `668b7bb`) dropped the redundant `--platform=$BUILDPLATFORM` pin from the prom-overlay builder stage. Registry image republished; arm64 manifest now ships an aarch64 ELF (digest `sha256:b209345c…`). The `X402_FACILITATOR_SKIP_PULL` knob has been removed from `flows/lib.sh`. |
 | flow-13 catalog assertion crashes with `'str' object has no attribute 'get'` | The smoke helper parsed `/api/services.json` as the old bare array and iterated object keys from the current catalog envelope. | `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` accepts both the current `{"services":[...]}` envelope and the legacy list form. |
 | Hermes install times out fetching `bedag/raw-2.0.2.tgz` | Hermes deploys generated Kubernetes resources through the third-party `bedag/raw` Helm chart. A chart repo/network miss blocks agent install even when the product logic is fine. | Classify as dependency/bootstrap drift. Longer-term debt reduction: replace the external raw chart with a small first-party/local raw-manifest chart or render/apply an embedded first-party Hermes chart. |
-| flow-02 stalls on `monitoring-kube-state-metrics` `ContainerCreating` | The kubelet is stuck pulling `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. | Treat as registry/bootstrap drift. Preflight or pre-cache the exact image before full release-smoke; do not change flow assertions to hide it. |
+| flow-02 stalls on `monitoring-kube-state-metrics` `ContainerCreating` | The kubelet is stuck pulling `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. | Treat as registry/bootstrap drift. Preflight or pre-cache the exact image before full release-smoke; do not change flow assertions to hide it. Cleared on plain rerun 2026-07-14 — a one-off is not a regression. |
+| flow-11 step 43 / flow-03 step 2: intermittent 404 `{'detail':'Not Found'}` on paid or first inference | Poisoned LiteLLM model group: auto-discovery registered the host vLLM with a bare `api_base` (no `/v1`) alongside the correct `setup custom` entry; LiteLLM shuffles ~50/50 and never retries a 404. Latent since v0.10.0, NOT flow- or rc-specific. | Fixed in #745 (`internal/model/discover.go` appends `/v1`). Decisive evidence: vLLM access log alternating `POST /chat/completions 404` / `POST /v1/chat/completions 200`. Drift checker can't see it (names only) — #746. |
+| flow-03 step 3 / flow-04: tool-call not returned, agent HTTP 000 | The QA vLLM endpoint was relaunched without its tool-call/reasoning parser flags (`--enable-auto-tool-choice --tool-call-parser ... --reasoning-parser ...`). flow-03 asserts tool-call passthrough; a parserless endpoint fails it every run. | Always relaunch the QA endpoint with its publisher-exact launcher script, never a hand-rolled unit. Preflight before a smoke run: one direct `curl` chat-completion with `tools[]` against the endpoint must return `tool_calls`. |
 
 **Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`obol kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic.
 

CLAUDE.md

@@ -439,7 +439,7 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit
 3. **ConfigMap propagation** — ~60-120s for k3d file watcher; force restart for immediate effect
 4. **ExternalName services** — do not work with Traefik Gateway API; use ClusterIP + Endpoints
 5. **eRPC `eth_call` cache** — default TTL is 10s for unfinalized reads; `buy.py balance` can lag behind an already-settled paid request for a few seconds
-6. **`/v1` required in `api_base` for `paid/*` route** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1`, not `...:8402`. Without `/v1`, LiteLLM calls `/chat/completions` on the buyer; the buyer's mux returns `404 page not found` (Go default), which LiteLLM surfaces as `OpenAIException - 404 page not found`.
+6. **`/v1` required in EVERY `openai/` `api_base`** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`; it POSTs `<api_base>/chat/completions` verbatim. Applies to the `paid/*` buyer route (`...:8402/v1`), `obol model setup custom` endpoints (validation enforces it by probing `<endpoint>/chat/completions`), AND local-server auto-discovery (fixed in #745 — discovery previously registered bare bases). The invariant lives on `buildCustomEndpointEntryWithOptions` (internal/model/model.go); buyer-side tooling uses the OPPOSITE convention (base without `/v1`, appends the full path — `buyprompts.ChatCompletionsURL`). Full producer/consumer audit: #745. Type-level enforcement tracked in #746.
 7. **LiteLLM restart is fallback, not the default buy path** — the validated happy path is `buy.py buy`/`process --all`/same-name top-up without a manual LiteLLM restart. Controller hot-add/hot-delete + buyer reload is expected to make `paid/<model>` appear and disappear in place. If a paid alias still fails after controller reconciliation and buyer sidecar reports the upstream, restart LiteLLM as a fallback investigation step. Reloader watches ONLY `litellm-secrets` (key rotation needs a pod replacement; it rolls gaplessly via RollingUpdate maxUnavailable:0). It must NOT watch `litellm-config` — model_list changes are hot-applied via /model/new//model/delete and a CM-triggered rollout would gap inference (issue #321); `obol model status` surfaces CM-vs-router drift if a hot call silently failed. Buyer CM writes (top-ups/refills) hot-reload via `/admin/reload` with no restart — do not add the buyer CMs to any Reloader annotation.
 8. **x402-verifier CA bundle missing → TLS failure** — The `x402-verifier` image is distroless (no CA store). The `ca-certificates` ConfigMap in `x402` namespace must be populated from the host CA bundle or the verifier cannot TLS-verify calls to the facilitator (`https://x402.gcp.obol.tech`), causing `x509: certificate signed by unknown authority` on every payment. **Fixed**: `obol stack up` calls `x402verifier.PopulateCABundle` after infrastructure deployment; `obol sell http` calls it before creating the ServiceOffer. If `Payment verification failed` errors still occur, check verifier logs for the x509 error and repopulate manually: `kubectl create configmap ca-certificates -n x402 --from-file=ca-certificates.crt=/etc/ssl/cert.pem --dry-run=client -o yaml | kubectl replace -f -`
 9. **`EnsureVerifier` overwrites helmfile's image pin under `OBOL_DEVELOPMENT=true`** — `internal/x402/setup.go` reads embedded `x402.yaml` (hard-coded image pin) and `kubectl apply`s it. Without an in-memory rewrite this overwrites the helmfile-managed `:latest` deployment with the embedded pin → every source change to the verifier silently bypassed. Fix shipped in `5a10fb8` (rewrites pins in-memory before apply); structural regression test: `internal/x402/setup_structure_test.go` (`TestEnsureVerifier_NoInlineRegex`). **If you add a new component installed via `kubectl apply` of an embedded manifest**, give it the same dev-rewrite treatment.
@@ -456,6 +456,8 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit
 19. **`obol sell info set` times out but profile CM updated** — the dev CLI wrote `x402/obol-storefront-profile` but the in-cluster `serviceoffer-controller` is still on an image that publishes `/api/services.json` as a bare `services[]` array (no `displayName` envelope). Symptom: `configmap/obol-storefront-profile configured` then `timed out waiting for controller to publish /api/services.json`; `kubectl get cm -n x402 obol-skill-md -o jsonpath='{.data.services\.json}'` starts with `[` not `{`. Fix: rebuild + import + restart `serviceoffer-controller` (see `sell info` bullet above). `obol stack up` with a warm cache (`built == 0`) does not pick up controller source changes unless `OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller`.
 20. **402 page silently falls back to JSON when the template errors** — `sendPaymentRequiredHTML` swallows template-exec errors and re-sends the JSON body, so referencing a field in `payment_required.html` that isn't in the render's data struct doesn't crash anything: browsers just start getting JSON. Symptom: `Content-Type: application/json` on an `Accept: text/html` request. Any template-field addition needs the struct field added in `paymentrequired.go` AND a test asserting the HTML branch still renders (the existing branding tests check Content-Type/markup).
 21. **`html/template` rejects `data:` URIs in URL contexts (`#ZgotmplZ`)** — inline logos/favicons from `sell info set --logo-file` are `data:image/...;base64` URIs; interpolating them into `src=`/`href=` via a plain string yields the literal `#ZgotmplZ` (broken image). Branding asset URLs must go through `storefront.SafeAssetURL` (validates http(s)/`data:image` then returns `template.URL`); regression test `TestPaymentRequiredHTML_InlineDataURILogo`.
+22. **Poisoned LiteLLM model group — intermittent 404 `{'detail': 'Not Found'}` on ~50% of requests** — two `model_list` deployments share a `model_name` but disagree on `api_base` (one with `/v1`, one without); LiteLLM shuffles between them and does not retry a 404. Historic cause: auto-discovery registering the same host endpoint that `obol model setup custom` later added correctly (#745). Diagnosis: compare the router's live view (`GET /model/info`) against the `litellm-config` CM — the drift checker compares model NAMES only and cannot see divergent `api_base` (#746). The vLLM access log is decisive: alternating `POST /chat/completions 404` / `POST /v1/chat/completions 200`.
+23. **Buyer disconnect ≠ no charge (zombie settlement)** — a client abort routinely does NOT propagate past cloudflared; the upstream agent finishes the turn, the handler returns 2xx, and settlement fires → buyer debited for a response nobody received (proven on-chain 2026-07-14, the Bankr incident). The verifier skips settlement when the request context is already canceled (#743), but that only covers propagated cancels — the real protection is keeping paid agent runs SHORT (`Agent.spec.maxTurns`; runs must finish inside the ~100s tunnel window and typical client timeouts). Concurrency: Hermes caps simultaneous runs in-process (`gateway.api_server.max_concurrent_runs`, internal default 10, 429 "Too many concurrent runs"; exposed as `Agent.spec.maxConcurrentRuns`, 0 = disabled) — keep it coherent with the edge `spec.limits.maxInFlight` Traefik gate; 4xx responses are never settled, so both gates are financially safe.
 
 For a fuller debug catalog with symptom->fix mapping, see `.agents/skills/obol-stack-dev/references/release-smoke-debugging.md`.
 

cmd/obol/buy.go

@@ -806,18 +806,6 @@ func authMultiplier(priceUnit string) int {
 	return 1
 }
 
-// perRequestEstimate divides a per-MTok atomic price by the temporary
-// tokens-per-request constant the controller uses today. Until the
-// facilitator implements usage-based settlement, this is the actual
-// flat per-call charge a buyer pays — surface it so users can compare
-// against perRequest offers without converting in their head.
-func perRequestEstimate(perMTokAtomic *big.Int) *big.Int {
-	if perMTokAtomic == nil {
-		return nil
-	}
-	return new(big.Int).Quo(perMTokAtomic, big.NewInt(int64(schemas.ApproxTokensPerRequest)))
-}
-
 func pricePerAuth(price *big.Int, multiplier int) (*big.Int, error) {
 	if price == nil || price.Sign() <= 0 {
 		return nil, errors.New("internal: price is non-positive")

cmd/obol/sell.go

@@ -3932,30 +3932,6 @@ func envOrDefault(key, fallback string) string {
 	return fallback
 }
 
-func formatPriceTableSummary(priceTable schemas.PriceTable, symbol string) string {
-	if symbol == "" {
-		symbol = "USDC"
-	}
-	switch {
-	case priceTable.PerRequest != "":
-		return fmt.Sprintf("%s %s/request", priceTable.PerRequest, symbol)
-	case priceTable.PerMTok != "":
-		return fmt.Sprintf("%s %s/request (approx from %s %s/MTok @ %d tok/request)",
-			priceTable.EffectiveRequestPrice(), symbol,
-			priceTable.PerMTok, symbol,
-			schemas.ApproxTokensPerRequest,
-		)
-	case priceTable.PerHour != "":
-		return fmt.Sprintf("%s %s/request (approx from %s %s/hour @ %d min/request)",
-			priceTable.EffectiveRequestPrice(), symbol,
-			priceTable.PerHour, symbol,
-			schemas.ApproxMinutesPerRequest,
-		)
-	default:
-		return fmt.Sprintf("0 %s/request", symbol)
-	}
-}
-
 func formatRoutePriceSummary(route x402verifier.RouteRule) string {
 	symbol := route.AssetSymbol
 	if symbol == "" {

flows/flow-03-inference.sh

@@ -48,15 +48,28 @@ for i in $(seq 1 15); do
     sleep 2
 done
 
-out=$(curl -sf --max-time 120 -X POST http://localhost:8001/v1/chat/completions \
-    -H "Content-Type: application/json" \
-    -H "Authorization: Bearer $LITELLM_KEY" \
-    -d "{\"model\":\"$LITELLM_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 2+2? Reply with the number only.\"}],\"max_tokens\":10,\"stream\":false}" 2>&1) || true
+# First completion after stack-up is flaky (port-forward/router warm-up race:
+# observed failing with empty output under curl -f while the very next request
+# succeeds). Retry a few times and keep the HTTP status so a real failure is
+# diagnosable instead of an empty string.
+out="" code=""
+for attempt in 1 2 3; do
+    out=$(curl -s --max-time 120 -w "\n%{http_code}" -X POST http://localhost:8001/v1/chat/completions \
+        -H "Content-Type: application/json" \
+        -H "Authorization: Bearer $LITELLM_KEY" \
+        -d "{\"model\":\"$LITELLM_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 2+2? Reply with the number only.\"}],\"max_tokens\":10,\"stream\":false}" 2>&1) || true
+    code="${out##*$'\n'}"
+    out="${out%$'\n'*}"
+    if echo "$out" | grep -q "choices"; then
+        break
+    fi
+    sleep 10
+done
 
 if echo "$out" | grep -q "choices"; then
-    pass "LiteLLM inference returned choices"
+    pass "LiteLLM inference returned choices (attempt $attempt)"
 else
-    fail "LiteLLM inference failed — ${out:0:300}"
+    fail "LiteLLM inference failed — HTTP ${code:-000}: ${out:0:300}"
 fi
 
 # §3d: Tool-call passthrough
@@ -94,7 +107,7 @@ tool_out=$(curl -sf --max-time 120 -X POST http://localhost:8001/v1/chat/complet
         "messages":[{"role":"user","content":"Call the get_weather tool for London. Do not answer in text."}],
         "tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],
         "tool_choice":{"type":"function","function":{"name":"get_weather"}},
-        "temperature":0,"max_tokens":100,"stream":false
+        "temperature":0,"max_tokens":2048,"stream":false
     }' 2>&1) || true
 
 if echo "$tool_out" | tool_call_name >/dev/null 2>&1; then
@@ -108,7 +121,7 @@ else
             "model":"'"$LITELLM_MODEL"'",
             "messages":[{"role":"user","content":"Call the get_weather tool with location London. Do not answer in text."}],
             "tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],
-            "temperature":0,"max_tokens":100,"stream":false
+            "temperature":0,"max_tokens":2048,"stream":false
         }' 2>&1) || true
 
     if echo "$tool_out" | tool_call_name >/dev/null 2>&1; then

flows/flow-07-sell-verify.sh

@@ -61,27 +61,6 @@ if [ -n "$TUNNEL_URL" ]; then
     TUNNEL_IP=$(resolve_public_ipv4 "$TUNNEL_HOST" || true)
 fi
 
-tunnel_get_code() {
-    local url="$1"
-    local code
-    if [ -n "$TUNNEL_HOST" ] && [ -n "$TUNNEL_IP" ]; then
-        if code=$(curl -sS --max-time 15 -o /dev/null -w '%{http_code}' \
-            --resolve "$TUNNEL_HOST:443:$TUNNEL_IP" \
-            "$url" 2>/dev/null); then
-            printf '%s\n' "$code"
-        else
-            printf '000\n'
-        fi
-    else
-        if code=$(curl -sS --max-time 15 -o /dev/null -w '%{http_code}' \
-            "$url" 2>/dev/null); then
-            printf '%s\n' "$code"
-        else
-            printf '000\n'
-        fi
-    fi
-}
-
 tunnel_get_file_code() {
     local url="$1"
     local outfile="$2"

flows/flow-08-buy.sh

@@ -59,10 +59,6 @@ purchase_request_ready() {
         -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>&1 || true
 }
 
-purchase_request_absent() {
-    ! "$OBOL" kubectl get purchaserequests.obol.org "$PURCHASE_NAME" -n "$AGENT_NS" >/dev/null 2>&1
-}
-
 buyer_sidecar_status() {
     "$OBOL" kubectl exec -n llm deployment/litellm -c litellm -- \
         python3 -c "
@@ -86,12 +82,6 @@ print('ready')
 " 2>&1 || true
 }
 
-agent_buy_skill_balance() {
-    "$OBOL" kubectl exec \
-        -n "$AGENT_NS" "deploy/$AGENT_DEPLOY" -c "$AGENT_CONTAINER" -- \
-        python3 "$AGENT_BUY_PY" balance --chain base-sepolia 2>&1 || true
-}
-
 agent_wallet_anvil_balance() {
     env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \
         --rpc-url "$ANVIL_RPC" 2>&1 || true

flows/flow-11-dual-stack.sh

@@ -411,30 +411,6 @@ PY
     exit 1
 }
 
-run_tail_or_fail() {
-    local desc="$1"
-    local success="$2"
-    local success_lines="${3:-3}"
-    shift 3
-
-    step "$desc"
-    local out rc
-    set +e
-    out=$("$@" 2>&1)
-    rc=$?
-    set -e
-
-    if [ "$rc" -ne 0 ]; then
-        printf '%s\n' "$out" | tail -120
-        fail "$desc failed (exit $rc)"
-        emit_metrics
-        exit "$rc"
-    fi
-
-    printf '%s\n' "$out" | tail -"$success_lines"
-    pass "$success"
-}
-
 refresh_alice_ports() {
     ALICE_HTTP_PORT="${FLOW11_ALICE_HTTP_PORT:-$(pick_free_port)}"
     ALICE_HTTP_ALT_PORT="${FLOW11_ALICE_HTTP_ALT_PORT:-$(pick_free_port)}"
@@ -1456,9 +1432,9 @@ else
     mkdir -p "$diag_dir"
     echo "  [diag] capturing cluster state to $diag_dir" >&2
     alice kubectl logs -n x402 deploy/x402-verifier --tail=200 > "$diag_dir/alice-verifier.log" 2>&1 || true
-    alice kubectl logs -n llm deploy/litellm -c x402-buyer --tail=200 > "$diag_dir/alice-buyer.log" 2>&1 || true
+    alice kubectl logs -n llm deploy/x402-buyer --tail=200 > "$diag_dir/alice-buyer.log" 2>&1 || true
     bob   kubectl logs -n x402 deploy/x402-verifier --tail=200 > "$diag_dir/bob-verifier.log"   2>&1 || true
-    bob   kubectl logs -n llm deploy/litellm -c x402-buyer --tail=200 > "$diag_dir/bob-buyer.log"     2>&1 || true
+    bob   kubectl logs -n llm deploy/x402-buyer --tail=200 > "$diag_dir/bob-buyer.log" 2>&1 || true
     alice kubectl get serviceoffer -A -o yaml > "$diag_dir/alice-serviceoffers.yaml" 2>&1 || true
     bob   kubectl get purchaserequest -A -o yaml > "$diag_dir/bob-purchaserequests.yaml" 2>&1 || true
     cleanup_pid "$PF_AGENT"