fix(crawler): bound the Playwright calls that carry no protocol timeout

[Aitosoft]

Summary

page.content() and page.evaluate() can block forever, and page_timeout does not cover them.

Neither method accepts a timeout, so the Python client sends none — _connection.py::_augment_params injects params["timeout"] only when a timeout calculator is passed. The driver then arms no timer at all: progress.js does const deadline = timeout ? monotonicTime() + timeout : 0, and with a falsy timeout progress.race() races against nothing but _forceAbortPromise, which is rejected only by page/context/connection close.

What they wait on is the frame's execution-context promise (frames.js::_context(world)), and frames.js::_setContext(world, null) discards the resolved promise and installs a fresh, unresolved ManualPromise on every _contextDestroyed. So a call issued while the main frame is between documents never returns.

Observed in production on a WordPress + Cloudflare Turnstile site that keeps committing navigations after domcontentloaded:

• page_timeout is irrelevant. It reaches page.goto and the wait_* family only. Lowering it 80s → 30s changed nothing.
• Nothing is logged. Both call sites sit inside swallow-all try/except, so the crawl simply stopped for 172 seconds until an external deadline fired.

The same line produces two different symptoms depending on timing: if the context dies during the call, frames.js:726 converts it to Unable to retrieve content because the page is navigating and changing the content; if the context is already gone at the call, it stalls silently instead. I could find no prior report of either.

page.close() is unbounded for the same reason, which matters because an outer deadline cancels straight into the cleanup path.

The fix — three bounds, outermost last

1. browser_adapter.bounded_evaluate() plus a timeout kwarg on all three adapters. 30 s default; 10 s for the optional cosmetic DOM steps (image dimensions, consent/overlay removal) which already degrade gracefully; 300 s for the virtual-scroll loop, which is legitimately slow. It raises Playwright's own TimeoutError — the same type Playwright raises for operations that do carry a timeout — so every existing except Error / except Exception handler keeps working unchanged, and it stays distinguishable from an outer asyncio deadline.

2. AsyncPlaywrightCrawlerStrategy._capture_html() — bounded page.content() with settle-and-retry. On the navigation error it waits for the new document to reach domcontentloaded and captures again, which is the documented remedy and far cheaper than re-running the whole crawl. Retries share a group budget (25 s) rather than one timeout each: a recoverable race raises instantly so retries stay nearly free, while a wedged page pays once instead of once per attempt. It also bails out early if the page cannot even reach domcontentloaded — that means stuck, not "between documents", and another attempt buys nothing.

3. CrawlerRunConfig.total_timeout (ms, default None) — one budget shared by every attempt and every proxy in arun()'s fetch loop, so max_retries cannot silently multiply the caller's deadline.

Relationship to #1923: complementary, not overlapping. That PR adds a total_timeout at the dispatcher level (MemoryAdaptiveDispatcher / SemaphoreDispatcher), which covers arun_many only. A direct arun() call — including the Docker /crawl path, which is where we hit this — gets no protection from it. Its problem statement is otherwise the same as this one, and I kept the name deliberately so the two can converge. Happy to rebase either way.

Default behaviour is unchanged. total_timeout defaults to None (off), and bounds 1 and 2 only fire where the call would otherwise never return.

List of files changed and why

• crawl4ai/browser_adapter.py — bounded_evaluate() helper + timeout kwarg on BrowserAdapter.evaluate and all three implementations (PlaywrightAdapter, StealthAdapter, UndetectedAdapter). One funnel bounds ~15 evaluate call sites.
• crawl4ai/async_crawler_strategy.py — _capture_html() used at all three page.content() sites (main capture, shadow-DOM fallback, get_delayed_content); explicit ceilings on the optional DOM steps, page.close() and virtual scroll; csp_compliant_wait passes its own timeout plus headroom so the adapter bound never races the JS polling loop it wraps; module-level constants so every ceiling is tunable in one place.
• crawl4ai/async_configs.py — CrawlerRunConfig.total_timeout (field, docstring, to_dict).
• crawl4ai/async_webcrawler.py — the attempt loop computes one deadline and wraps each crawler_strategy.crawl() in it; a later attempt that cannot start is skipped with an attributable message rather than run.
• docs/md_v2/api/parameters.md, docs/md_v2/advanced/anti-bot-and-fallback.md — document total_timeout and why page_timeout is not a substitute.
• tests/async/test_render_call_bounds.py — new, 14 tests, no browser and no network.

How Has This Been Tested?

$ pytest tests/async/test_render_call_bounds.py -q
14 passed in 4.00s

The unit tests use a fake page whose content() hangs / raises the navigation error on demand, and a fake crawler strategy with crawler.ready = True, so no Chromium is launched.

Beyond unit tests, I built a fixture origin that reproduces the race deterministically — a page that commits a navigation to a URL whose response headers are delayed — and located the original hang by walking the suspended task's real await chain via cr_await (Task.get_stack() returns only one frame for a suspended coroutine and is useless here). The blocked frame was adapter.evaluate(update_image_dimensions_js) → _frame.py:320 evaluate() → a protocol future that never resolves.

Measured through the full Docker /crawl path against that fixture, with wait_until: domcontentloaded, page_timeout: 80s, max_retries: 1, delay_before_return_html: 2.0:

Case	Before	After
page commits a navigation during capture	hung until an external 180 s deadline	HTTP 200, full content, 5.0 s
page permanently between documents	hung until an external 180 s deadline, no diagnostic	bounded failure with the exact reason logged
arun() with a 0.4 s budget and 4 attempts	unbounded	≤ 2 attempts, bounded

Cancellation was checked separately: after a bounded attempt the browser still serves the next crawl (0.24 s) and the page count does not grow.

black --check and ruff check are clean on the new test. I did not run black over the touched crawl4ai/ modules — they are not black-formatted on develop and reformatting them would bury the change in noise.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes