Version: 0.9.2 (Docker), also present on main @ 7e80152
What happens
handle_markdown_request in deploy/docker/api.py promotes any unsuccessful crawl into a server error:
if not result.success:
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=result.error_message
)handle_llm_qa carries the identical block.
Since 0.8.5, is_blocked() sets success = False and error_message = "Blocked by anti-bot protection: <reason>" whenever the escalation chain cannot rescue a page. So a remote site serving a Cloudflare, DataDome or Akamai challenge — an ordinary and fully expected outcome — surfaces from the Docker API as a 500.
Why the caller can't tell
_http_exception_handler in deploy/docker/server.py genericizes every 5xx to avoid leaking internals:
{"error": "Internal server error", "correlation_id": "<random>"}That hardening is right, and MIGRATION.md documents it deliberately. But because a bot-block is routed through 5xx, the sanitiser strips the one field the caller needs. error_message is computed, logged, then discarded before it reaches the client. Two individually correct decisions collide.
The result: an HTTP client cannot distinguish "the target site blocked us" from "the service is unhealthy" without shell access to the container logs.
Expected
A blocked page is a crawl outcome, not a server fault. The anti-bot docs define the terminal state as success=False plus an error_message naming the block reason.
handle_crawl_request already honours this — it returns 200 with per-URL success flags and reasons intact. The single-URL endpoints disagree with both the documentation and their sibling handler in the same file.
Suggested: 502, or 200 with a structured failure body; either way preserving error_message.
Repro
curl -sS -X POST http://localhost:11235/md \
-H "Content-Type: application/json" \
-d '{"url":"https://www.sagaftra.org/"}'
# HTTP 500
# {"error":"Internal server error","correlation_id":"e2996592a617"}
curl -sS -X POST http://localhost:11235/crawl \
-H "Content-Type: application/json" \
-d '{"urls":["https://www.sagaftra.org/"]}'
# HTTP 200
# results[0].success = false
# results[0].status_code = 403
# results[0].error_message = "Blocked by anti-bot protection: DataDome captcha"Same page, same crawler, same block. One endpoint says what happened; the other says nothing.
Why this bites harder over REST than in the library
The docs state the terminal condition as "When all attempts fail and blocking is still detected, the result is returned with success=False and error_message describing the block reason." That is the contract /crawl honours and /md breaks.
The escalation those attempts refer to is unreachable over REST. async_configs.py:203-213 places both escape hatches in UNTRUSTED_FORBIDDEN_FIELDS:
fallback_fetch_function — additionally typed Optional[Callable[...]] (line 1705), so it could not cross a JSON boundary regardlessproxy_config, proxy_rotation_strategy, proxy_session_id
So on a stock Docker deployment no request body can supply either one. Detection runs, escalation cannot, and every block lands as an opaque 500. Discussion #1905 — Avada/WordPress pages flagged Structural: no tag with no anti-bot present, still unanswered — is this same failure seen from the other end: the reporter cannot inspect what came back, because the response body no longer contains it.
Impact
One deployment logged 132 of these over six days across roughly fifteen distinct block reasons, including several Structural: classifications on sites we believe are not protected at all. Per-domain retry policy or extraction routing can't be built on the single-URL endpoints while the reason is unavailable to the caller.
Note on the environment
These figures come from a locally derived image built on the pinned 0.9.2 digest, carrying local operational patches unrelated to the paths below. The three code paths above — the /md failure branch, the /llm/{url} failure branch and the 5xx sanitiser — are unmodified upstream, and the /md branch is byte-identical on current main.
Version: 0.9.2 (Docker), also present on
main@7e80152What happens
handle_markdown_requestindeploy/docker/api.pypromotes any unsuccessful crawl into a server error:handle_llm_qacarries the identical block.Since 0.8.5,
is_blocked()setssuccess = Falseanderror_message = "Blocked by anti-bot protection: <reason>"whenever the escalation chain cannot rescue a page. So a remote site serving a Cloudflare, DataDome or Akamai challenge — an ordinary and fully expected outcome — surfaces from the Docker API as a 500.Why the caller can't tell
_http_exception_handlerindeploy/docker/server.pygenericizes every 5xx to avoid leaking internals:{"error": "Internal server error", "correlation_id": "<random>"}That hardening is right, and MIGRATION.md documents it deliberately. But because a bot-block is routed through 5xx, the sanitiser strips the one field the caller needs.
error_messageis computed, logged, then discarded before it reaches the client. Two individually correct decisions collide.The result: an HTTP client cannot distinguish "the target site blocked us" from "the service is unhealthy" without shell access to the container logs.
Expected
A blocked page is a crawl outcome, not a server fault. The anti-bot docs define the terminal state as
success=Falseplus anerror_messagenaming the block reason.handle_crawl_requestalready honours this — it returns 200 with per-URLsuccessflags and reasons intact. The single-URL endpoints disagree with both the documentation and their sibling handler in the same file.Suggested: 502, or 200 with a structured failure body; either way preserving
error_message.Repro
Same page, same crawler, same block. One endpoint says what happened; the other says nothing.
Why this bites harder over REST than in the library
The docs state the terminal condition as "When all attempts fail and blocking is still detected, the result is returned with
success=Falseanderror_messagedescribing the block reason." That is the contract/crawlhonours and/mdbreaks.The escalation those attempts refer to is unreachable over REST.
async_configs.py:203-213places both escape hatches inUNTRUSTED_FORBIDDEN_FIELDS:fallback_fetch_function— additionally typedOptional[Callable[...]](line 1705), so it could not cross a JSON boundary regardlessproxy_config,proxy_rotation_strategy,proxy_session_idSo on a stock Docker deployment no request body can supply either one. Detection runs, escalation cannot, and every block lands as an opaque 500. Discussion #1905 — Avada/WordPress pages flagged
Structural: no tagwith no anti-bot present, still unanswered — is this same failure seen from the other end: the reporter cannot inspect what came back, because the response body no longer contains it.Impact
One deployment logged 132 of these over six days across roughly fifteen distinct block reasons, including several
Structural:classifications on sites we believe are not protected at all. Per-domain retry policy or extraction routing can't be built on the single-URL endpoints while the reason is unavailable to the caller.Note on the environment
These figures come from a locally derived image built on the pinned 0.9.2 digest, carrying local operational patches unrelated to the paths below. The three code paths above — the
/mdfailure branch, the/llm/{url}failure branch and the 5xx sanitiser — are unmodified upstream, and the/mdbranch is byte-identical on currentmain.