discovery.md: what should a client do when the card fetch returns a 429 from the recommended rate limit?

[flober81]

One question on the discovery mechanics, from an operator's side.

docs/discovery.md reserves <streamable-http-url>/server-card and notes the resulting path-namespacing as a benefit (a server at https://host/mcp yields https://host/mcp/server-card). Under Denial of Service — the guidance restored in #21 — it recommends that servers rate-limit the card endpoint, and asks clients to respect Cache-Control and use ETags. The client flow has a single negative outcome, "Discovery unavailable for this domain", and it hangs off the catalog fetch (/.well-known/ai-catalog.json); the later step "Fetch Server Card from url" has no failure branch at all.

Why this is structural rather than hypothetical: a limit protecting an MCP endpoint is normally scoped by path prefix and placed ahead of application code. In my deployment, anonymous requests under the API prefix hit a per-client bucket in middleware before the application initializes and before any database query; authenticated requests are exempt elsewhere, but not under the MCP path itself, because resolving a bearer key there costs an HMAC and an indexed query per attempt. A card served under that prefix shares that bucket with all other anonymous traffic from the same client address by default — unless the operator carves it out, which the document does not tell them to do. A single first GET will not exhaust it, but a crawler fleet, a shared corporate egress, or a client that already made requests under the prefix can arrive with the bucket spent — and first contact is the one moment it holds neither a cache entry nor an ETag, so the document's mitigation is not yet available. A 403 from a bot filter, or a challenge page returned with 200, collapses the same way.

For comparison, RFC 9309 §2.3.1.3/§2.3.1.4 separates "unavailable" (4xx) from "unreachable" (5xx/network) for robots.txt, and Google's crawler documentation carves 429 out of its 4xx rule (https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#http-status-codes). In the SEP text at the current PR head and in docs/discovery.md including its client-flow section, I find no mention of 429 or Retry-After; the SEP text repeats the recommendation ("Servers SHOULD rate-limit discovery endpoints") without naming a status code either.

So, one question: what should a client do when the card fetch — not the catalog — returns a 429 produced by the rate limiting the same document recommends: treat the entry as unavailable, retry after Retry-After, or fall back to inline data in the catalog?

Florian Berger — Bolzano/Bozen (IT)

[jarrettdustinqq]

The missing branch should be local to the card entry, rather than turning one failed card fetch into domain-wide discovery failure. At current revision 526201b, an entry contains exactly one of url or data, so inline fallback is not available for that same URL entry.

A narrow proposed policy:

Card-fetch outcome	Proposed client behavior
Successful response containing a valid Server Card	Continue with that entry. HTTP success alone is insufficient; a 200 HTML challenge page is not a card.
304 following conditional retrieval	Reuse the corresponding stored representation under HTTP cache rules; without one, there is no card to use.
429	Record the entry as temporarily unavailable. A client may stop or retry within its cancellation, attempt, and elapsed-time budgets. If retrying with a valid Retry-After, wait at least that long; if it exceeds the budget, stop rather than cap the delay and retry early.
503, transient network failure, or another retryable failure	Use a bounded backoff policy; honor an applicable valid Retry-After. Do not infer that the server or catalog entry does not exist.
Other unrecoverable failure or invalid card	Report failure for that entry and continue with independent entries, without an immediate retry loop.

RFC 6585 §4 permits Retry-After on 429 and prohibits caching the 429 response. RFC 9110 §10.2.3 allows both a delay in seconds and an HTTP-date. Missing, malformed, or already elapsed retry times still need a client backoff floor and bounded attempts. These HTTP rules do not themselves choose MCP's retry budget or require a client to keep retrying.

There is also a browser-specific dependency: the current CORS example exposes only ETag. Retry-After is not a Fetch safelisted response header. To make server-directed retries observable by cross-origin browser clients, expose ETag, Retry-After and send the relevant CORS headers on rate-limit responses too, including when middleware emits them. Otherwise a client may see no readable delay—or only a network error if CORS itself fails.

Suggested conformance cases: first contact receiving 429 without a cache; valid seconds/date delays; a delay longer than the remaining budget; malformed/missing delay; cancellation during waiting; 200 HTML; 304 with/without a stored card; two catalog entries where one fails and the other remains usable; and cross-origin 429 with/without exposed Retry-After.

AI assistance disclosure: Codex performed the source comparison and drafted this proposal. No browser integration test or production deployment is claimed.

[flober81]

Agree on scoping the failure to the entry, and the CORS point is a good catch: without exposing Retry-After, a browser client cannot see the delay even when the operator sets it correctly.