fix(scrapy): stop silently dropping in-flight requests and redirects

[vdusek]

Two request-loss bugs in the Scrapy integration, both from next_request() marking a request handled the moment it is fetched, before Scrapy has downloaded anything.

In-flight requests were lost on migration. Everything sitting in the downloader or the parse pipeline was already handled_at-stamped, so an interrupted run dropped all of it, and reclaim_request was never called anywhere in the integration. A request now stays unresolved in the queue until Scrapy is done with it, tracked through downloader.active and scraper.slot.active — a request leaves both only once its download, the middleware chain, the callback and the item pipeline have finished, so the dataset push is covered too. Reconstruction failures are still consumed immediately.

Killing a process with three requests in flight: 3 handled / 0 pending before, all three lost; 0 handled / 3 pending after, all fetchable next run. A graceful SIGINT resolves exactly what completed: 6 responses, 6 handled, 5 pending, no duplicates.

Every redirect on a discovered link was dropped. RedirectMiddleware derives the new request with request.replace(url=...), which inherits meta['apify_request_unique_key'], so add_request answered was_already_present=True and enqueue_request returned False — a silent drop with no error. Start URLs and retries were masked only by their dont_filter=True. to_scrapy_request now stamps meta['apify_request_url'] alongside the key, and the key is reused only while that URL still matches. A hand-set stamp without a URL is still honored.

Reactor stalls. next_request() did two serial blocking API round trips per request on the reactor thread; now one. Marks go through a fire-and-forget AsyncThread.submit_coro, drained before is_finished() is consulted and before the event loop closes. A hard kill can still lose a mark, leaving that request pending for the next run — at-least-once is the safe direction.

✍️ Drafted by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 91.25000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.51%. Comparing base (fcc767a) to head (31391ed).

Files with missing lines	Patch %	Lines
src/apify/scrapy/scheduler.py	88.23%	6 Missing ⚠️
src/apify/scrapy/_async_thread.py	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1097      +/-   ##
==========================================
+ Coverage   92.31%   92.51%   +0.20%     
==========================================
  Files          51       51              
  Lines        3317     3382      +65     
==========================================
+ Hits         3062     3129      +67     
+ Misses        255      253       -2     

Flag	Coverage Δ
e2e	34.06% <0.00%> (-0.67%)	⬇️
integration	55.47% <0.00%> (-1.09%)	⬇️
unit	87.25% <91.25%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.