How can I re-queue a URL?

[matecsaj]

Crawlee optimizes requests by suppressing redundant URLs and giving up on a URL after reaching a configurable retry limit. This is great, but I encountered an edge case.

Situation: You're extracting data from a page and realize that it wasn’t downloaded properly.

Ideal Goal: Re-queue the URL while decrementing its retry counter.

Alternative: Add the URL back to the queue with a fresh retry counter.

How can I achieve either of these?

[B4nan]

You can enqueue it again with a different unique_key, that's what we use for deduplication (and it defaults to the URL itself).

[vdusek]

Ideal Goal: Re-queue the URL while decrementing its retry counter.

I am afraid this is not possible.

Add the URL back to the queue with a fresh retry counter.

You can use the always_enqueue=True:

request = Request.from_url('https://crawlee.dev/', always_enqueue=True)

which will generate "unique" unique_key under the hood. And add the request to the request queue.

[janbuchar]

Actually, I think you can do this:

request = Request.from_url('https://crawlee.dev/', always_enqueue=True)
request.max_retries = failed_request.max_retries - 1  # TODO check if this is not negative
await queue.add_request(request)

It does not enqueue the same request per se, but it's close enough if you copy relevant parameters.

[matecsaj]

Thank you for the helpful answer.

[matecsaj]

I searched the documentation but couldn’t find an example of ‘await queue.add_request(request)’, so I’m unsure how to import or instantiate ‘queue’. Nothing breaks when I try the following—would this be considered acceptable?

await context.add_requests([Request.from_url(url=requested_url, always_enqueue=True)])

[janbuchar]

Yes, this is a reasonable way to enqueue new requests

[vdusek]

@matecsaj Have you seen this page ?