fix(antibot): detect blocks behind a redirect chain

[Aitosoft]

Summary

Block detection is given the first hop of a redirect chain instead of the last, so a site that redirects and then serves a block page is reported as a clean success with the block page as its content.

CrawlResult.status_code deliberately carries the first hop (#660, documented in docs/md_v2/api/crawl-result.md §1.3) while the HTML always comes from the last hop, kept separately as redirected_status_code (#1435, §1.4). async_webcrawler.py passes status_code to antibot_detector.is_blocked() at all three call sites, so a 301 is judged instead of the 403 it led to — and none of is_blocked's status rules (429, 403/503, >=400, ==200) can fire on a 3xx.

Live result on unmodified v0.9.2:

{ "url": "https://konecranes.com", "success": true,
  "status_code": 301, "redirected_status_code": 403,
  "redirected_url": "https://www.konecranes.com/", "error_message": "",
  "markdown": "# Error 403 Forbidden … Varnish cache server" }

This is the missing half of #1434 ("no way to tell if a redirected result was successful"). #1435 fixed the user-visible half by exposing redirected_status_code; the internal consumer was never updated. Two details make that concrete:

• redirected_status_code was merged 2026-02-06 (Feat: Add redirected_status_code to CrawlResult #1435). crawl4ai/antibot_detector.py and this retry loop were added 8 days later (72b546c) still wired to status_code — the correct field already existed.
• AsyncHTTPCrawlerStrategy sets status_code to the post-redirect final status (allow_redirects) and never sets redirected_status_code. So today the same URL gets a different block verdict depending on which crawler strategy runs. This patch makes them agree.

Fixing the consumer rather than the producer is deliberate: status_code's first-hop semantics are documented and pinned by tests/test_pr_1435_redirected_status_code.py, and callers may branch on it.

Why this is safe

Redirect hops are 3xx by construction, and no status rule in is_blocked matches a 3xx. So the change can only add blocked verdicts, never remove one. When there is no redirect, status_code == redirected_status_code and it is a literal no-op. redirected_status_code is None on the non-HTTP paths (raw:, file://, js_only) that were hand-patched when #1435 was merged, so the helper falls back to status_code there.

Callers should expect their observed failure rate to rise after this lands — results that were silently wrong become correctly failed.

List of files changed and why

• crawl4ai/antibot_detector.py — adds effective_status(status_code, redirected_status_code): one documented helper so the three call sites cannot drift apart.
• crawl4ai/async_webcrawler.py — the three is_blocked() call sites (attempt loop, _needs_fallback, and the final "mark blocked results as failed" pass) now pass effective_status(...). Nothing else changes.
• tests/proxy/test_antibot_redirect_status.py — new, 8 tests, no browser and no network.

How Has This Been Tested?

$ pytest tests/proxy/test_antibot_redirect_status.py -q
8 passed in 1.83s

$ python tests/proxy/test_antibot_detector.py
RESULTS: 47 passed, 0 failed out of 47 tests

$ pytest tests/test_pr_1435_redirected_status_code.py -q -k Model
4 passed, 3 deselected

The new tests drive arun() with a fake crawler strategy and crawler.ready = True, so no Chromium is launched and nothing is fetched. Verified they fail on the unpatched tree: 3 of the 8 fail (the block-behind-redirect case), the other 5 are regression guards that pass either way — benign 301 → 200, no-redirect 200, no-redirect 403, raw: URLs, and is_blocked's own unchanged behaviour.

Also exercised end-to-end through the Docker /crawl path against a local fixture origin (Chromium, wait_until: domcontentloaded, max_retries: 1):

Case	Before	After
301 → 200 benign	200, success	200, success — no false positive
301 → 403 block page	200, success: true, block page as content	correctly failed
direct 403 block page	correctly failed	correctly failed (unchanged)

black --check and ruff check are clean on the files this touches. (ruff reports 9 pre-existing errors in async_webcrawler.py that are present on unmodified develop; I left them alone.)

Note on overlap with #2088

This touches the same three lines as #2088 (check_blocked opt-out). The two are orthogonal — #2088 gates whether the calls happen, this fixes what they are passed — but they will conflict textually. Happy to rebase on whichever lands first.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes