feat(httpx): Use httpx2 instead httpx

docs/guides/architecture_overview.mdx

@@ -76,7 +76,7 @@ PlaywrightCrawler --|> StagehandCrawler
 
 ### HTTP crawlers
 
-HTTP crawlers use HTTP clients to fetch pages and parse them with HTML parsing libraries. They are fast and efficient for sites that do not require JavaScript rendering. HTTP clients are Crawlee components that wrap around HTTP libraries like [httpx](https://www.python-httpx.org/), [curl-impersonate](https://github.com/lwthiker/curl-impersonate) or [impit](https://github.com/apify/impit) and handle HTTP communication for requests and responses. You can learn more about them in the [HTTP clients guide](./http-clients).
+HTTP crawlers use HTTP clients to fetch pages and parse them with HTML parsing libraries. They are fast and efficient for sites that do not require JavaScript rendering. HTTP clients are Crawlee components that wrap around HTTP libraries like [httpx2](https://httpx2.pydantic.dev/), [curl-impersonate](https://github.com/lwthiker/curl-impersonate) or [impit](https://github.com/apify/impit) and handle HTTP communication for requests and responses. You can learn more about them in the [HTTP clients guide](./http-clients).
 
 HTTP crawlers inherit from <ApiLink to="class/AbstractHttpCrawler">`AbstractHttpCrawler`</ApiLink> and there are five crawlers that belong to this category:
 

docs/guides/code_examples/http_clients/parsel_httpx_example.py

@@ -6,7 +6,7 @@
 
 async def main() -> None:
     http_client = HttpxHttpClient(
-        # Optional additional keyword arguments for `httpx.AsyncClient`.
+        # Optional additional keyword arguments for `httpx2.AsyncClient`.
         timeout=10,
         follow_redirects=True,
     )

docs/guides/http_clients.mdx

@@ -13,7 +13,7 @@ import ParselHttpxExample from '!!raw-loader!roa-loader!./code_examples/http_cli
 import ParselCurlImpersonateExample from '!!raw-loader!roa-loader!./code_examples/http_clients/parsel_curl_impersonate_example.py';
 import ParselImpitExample from '!!raw-loader!roa-loader!./code_examples/http_clients/parsel_impit_example.py';
 
-HTTP clients are utilized by HTTP-based crawlers (e.g., <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> and <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>) to communicate with web servers. They use external HTTP libraries for communication rather than a browser. Examples of such libraries include [httpx](https://pypi.org/project/httpx/), [aiohttp](https://pypi.org/project/aiohttp/), [curl-cffi](https://pypi.org/project/curl-cffi/), and [impit](https://pypi.org/project/impit/). After retrieving page content, an HTML parsing library is typically used to facilitate data extraction. Examples of such libraries include [beautifulsoup](https://pypi.org/project/beautifulsoup4/), [parsel](https://pypi.org/project/parsel/), [selectolax](https://pypi.org/project/selectolax/), and [pyquery](https://pypi.org/project/pyquery/). These crawlers are faster than browser-based crawlers but cannot execute client-side JavaScript.
+HTTP clients are utilized by HTTP-based crawlers (e.g., <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> and <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>) to communicate with web servers. They use external HTTP libraries for communication rather than a browser. Examples of such libraries include [httpx2](https://pypi.org/project/httpx2/), [aiohttp](https://pypi.org/project/aiohttp/), [curl-cffi](https://pypi.org/project/curl-cffi/), and [impit](https://pypi.org/project/impit/). After retrieving page content, an HTML parsing library is typically used to facilitate data extraction. Examples of such libraries include [beautifulsoup](https://pypi.org/project/beautifulsoup4/), [parsel](https://pypi.org/project/parsel/), [selectolax](https://pypi.org/project/selectolax/), and [pyquery](https://pypi.org/project/pyquery/). These crawlers are faster than browser-based crawlers but cannot execute client-side JavaScript.
 
 ```mermaid
 ---
@@ -53,7 +53,7 @@ HttpClient --|> CurlImpersonateHttpClient
 
 ## Switching between HTTP clients
 
-Crawlee currently provides three main HTTP clients: <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>, which uses the `impit` library, <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink>, which uses the `httpx` library with `browserforge` for custom HTTP headers and fingerprints, and <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>, which uses the `curl-cffi` library. You can switch between them by setting the `http_client` parameter when initializing a crawler class. The default HTTP client is <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
+Crawlee currently provides three main HTTP clients: <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>, which uses the `impit` library, <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink>, which uses the `httpx2` library with `browserforge` for custom HTTP headers and fingerprints, and <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>, which uses the `curl-cffi` library. You can switch between them by setting the `http_client` parameter when initializing a crawler class. The default HTTP client is <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
 
 Below are examples of how to configure the HTTP client for the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>:
 

docs/guides/http_headers.mdx

@@ -60,7 +60,7 @@ A header is a request, not a command. The server decides what to do with it. A h
 
 ## Default headers in Crawlee
 
-All built-in HTTP clients impersonate a browser by default. Instead of a bare library `User-Agent` like `python-httpx/0.27`, they send a realistic set of browser-like headers: a browser `User-Agent`, an `Accept`, an `Accept-Language`, and client hints where the client supports them. Such headers make a crawl look like normal browser traffic and avoid the simplest forms of blocking.
+All built-in HTTP clients impersonate a browser by default. Instead of a bare library `User-Agent` like `python-httpx2/2.12`, they send a realistic set of browser-like headers: a browser `User-Agent`, an `Accept`, an `Accept-Language`, and client hints where the client supports them. Such headers make a crawl look like normal browser traffic and avoid the simplest forms of blocking.
 
 Each client implements impersonation its own way:
 

pyproject.toml

@@ -67,7 +67,7 @@ cli = [
     "typer>=0.26.0",
 ]
 curl-impersonate = ["curl-cffi>=0.9.0"]
-httpx = ["httpx[brotli,http2,zstd]>=0.27.0", "apify_fingerprint_datapoints>=0.0.2", "browserforge>=1.2.3"]
+httpx = ["httpx2[brotli,http2,zstd]>=2.12.0", "apify_fingerprint_datapoints>=0.0.2", "browserforge>=1.2.3"]
 parsel = ["parsel>=1.10.0"]
 playwright = ["playwright>=1.27.0", "apify_fingerprint_datapoints>=0.0.2", "browserforge>=1.2.3"]
 otel = [

src/crawlee/http_clients/_httpx.py

@@ -5,7 +5,7 @@
 from logging import DEBUG, WARNING, getLogger
 from typing import TYPE_CHECKING, Any, cast
 
-import httpx
+import httpx2
 from typing_extensions import override
 
 from crawlee._log_config import get_configured_log_level
@@ -32,9 +32,9 @@
 
 
 class _HttpxResponse:
-    """Adapter class for `httpx.Response` to conform to the `HttpResponse` protocol."""
+    """Adapter class for `httpx2.Response` to conform to the `HttpResponse` protocol."""
 
-    def __init__(self, response: httpx.Response) -> None:
+    def __init__(self, response: httpx2.Response) -> None:
         self._response = response
 
     @property
@@ -62,16 +62,16 @@ async def read_stream(self) -> AsyncIterator[bytes]:
                 yield chunk
 
 
-class _HttpxTransport(httpx.AsyncHTTPTransport):
+class _HttpxTransport(httpx2.AsyncHTTPTransport):
     """HTTP transport adapter that stores response cookies in a `Session`.
 
     This transport adapter modifies the handling of HTTP requests to update the session cookies
     based on the response cookies, ensuring that the cookies are stored in the session object
-    rather than the `HTTPX` client itself.
+    rather than the `HTTPX2` client itself.
     """
 
     @override
-    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
+    async def handle_async_request(self, request: httpx2.Request) -> httpx2.Response:
         response = await super().handle_async_request(request)
         response.request = request
 
@@ -86,9 +86,9 @@ async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
 
 @docs_group('HTTP clients')
 class HttpxHttpClient(HttpClient):
-    """HTTP client based on the `HTTPX` library.
+    """HTTP client based on the `HTTPX2` library.
 
-    This client uses the `HTTPX` library to perform HTTP requests in crawlers (`BasicCrawler` subclasses)
+    This client uses the `HTTPX2` library to perform HTTP requests in crawlers (`BasicCrawler` subclasses)
     and to manage sessions, proxies, and error handling.
 
     See the `HttpClient` class for more common information about HTTP clients.
@@ -124,15 +124,15 @@ def __init__(
             http2: Whether to enable HTTP/2 support.
             verify: SSL certificates used to verify the identity of requested hosts.
             header_generator: Header generator instance to use for generating common headers.
-            async_client_kwargs: Additional keyword arguments for `httpx.AsyncClient`.
+            async_client_kwargs: Additional keyword arguments for `httpx2.AsyncClient`.
         """
         super().__init__(
             persist_cookies_per_session=persist_cookies_per_session,
         )
 
-        # `httpx` logs one INFO line per request, which is too noisy for the default log level. Silence it down to
+        # `httpx2` logs one INFO line per request, which is too noisy for the default log level. Silence it down to
         # WARNING unless the user has explicitly opted into DEBUG.
-        httpx_logger = getLogger('httpx')
+        httpx_logger = getLogger('httpx2')
         httpx_logger.setLevel(DEBUG if get_configured_log_level() <= DEBUG else WARNING)
 
         self._http1 = http1
@@ -141,11 +141,11 @@ def __init__(
         self._async_client_kwargs = async_client_kwargs
         self._header_generator = header_generator
 
-        self._ssl_context = httpx.create_ssl_context(verify=verify)
+        self._ssl_context = httpx2.create_ssl_context(verify=verify)
 
         self._transport: _HttpxTransport | None = None
 
-        self._client_by_proxy_url = dict[str | None, httpx.AsyncClient]()
+        self._client_by_proxy_url = dict[str | None, httpx2.AsyncClient]()
 
     @override
     async def crawl(
@@ -167,14 +167,14 @@ async def crawl(
             content=request.payload,
             cookies=session.cookies.jar if session else None,
             extensions={'crawlee_session': session if self._persist_cookies_per_session else None},
-            timeout=timeout.total_seconds() if timeout is not None else httpx.USE_CLIENT_DEFAULT,
+            timeout=timeout.total_seconds() if timeout is not None else httpx2.USE_CLIENT_DEFAULT,
         )
 
         try:
             response = await client.send(http_request)
-        except httpx.TimeoutException as exc:
+        except httpx2.TimeoutException as exc:
             raise asyncio.TimeoutError from exc
-        except httpx.TransportError as exc:
+        except httpx2.TransportError as exc:
             if self._is_proxy_error(exc):
                 raise ProxyError from exc
             raise
@@ -211,14 +211,14 @@ async def send_request(
             headers=headers,
             payload=payload,
             session=session,
-            timeout=httpx.Timeout(timeout.total_seconds()) if timeout is not None else None,
+            timeout=httpx2.Timeout(timeout.total_seconds()) if timeout is not None else None,
         )
 
         try:
             response = await client.send(http_request)
-        except httpx.TimeoutException as exc:
+        except httpx2.TimeoutException as exc:
             raise asyncio.TimeoutError from exc
-        except httpx.TransportError as exc:
+        except httpx2.TransportError as exc:
             if self._is_proxy_error(exc):
                 raise ProxyError from exc
             raise
@@ -249,12 +249,12 @@ async def stream(
             headers=headers,
             payload=payload,
             session=session,
-            timeout=httpx.Timeout(None, connect=timeout.total_seconds()) if timeout else None,
+            timeout=httpx2.Timeout(None, connect=timeout.total_seconds()) if timeout else None,
         )
 
         try:
             response = await client.send(http_request, stream=True)
-        except httpx.TimeoutException as exc:
+        except httpx2.TimeoutException as exc:
             raise asyncio.TimeoutError from exc
 
         try:
@@ -265,15 +265,15 @@ async def stream(
     def _build_request(
         self,
         *,
-        client: httpx.AsyncClient,
+        client: httpx2.AsyncClient,
         url: str,
         method: HttpMethod,
         headers: HttpHeaders | dict[str, str] | None,
         payload: HttpPayload | None,
         session: Session | None = None,
-        timeout: httpx.Timeout | None = None,
-    ) -> httpx.Request:
-        """Build an `httpx.Request` using the provided parameters."""
+        timeout: httpx2.Timeout | None = None,
+    ) -> httpx2.Request:
+        """Build an `httpx2.Request` using the provided parameters."""
         if isinstance(headers, dict) or headers is None:
             headers = HttpHeaders(headers or {})
 
@@ -285,18 +285,18 @@ def _build_request(
             headers=dict(headers) if headers else None,
             content=payload,
             extensions={'crawlee_session': session if self._persist_cookies_per_session else None},
-            timeout=timeout or httpx.USE_CLIENT_DEFAULT,
+            timeout=timeout or httpx2.USE_CLIENT_DEFAULT,
         )
 
-    def _get_client(self, proxy_url: str | None) -> httpx.AsyncClient:
+    def _get_client(self, proxy_url: str | None) -> httpx2.AsyncClient:
         """Retrieve or create an HTTP client for the given proxy URL.
 
         If a client for the specified proxy URL does not exist, create and store a new one.
         """
         if not self._transport:
             # Configure connection pool limits and keep-alive connections for transport
             limits = self._async_client_kwargs.get(
-                'limits', httpx.Limits(max_connections=1000, max_keepalive_connections=200)
+                'limits', httpx2.Limits(max_connections=1000, max_keepalive_connections=200)
             )
 
             self._transport = _HttpxTransport(
@@ -325,7 +325,7 @@ def _get_client(self, proxy_url: str | None) -> httpx.AsyncClient:
                 }
             )
 
-            client = httpx.AsyncClient(**kwargs)
+            client = httpx2.AsyncClient(**kwargs)
             self._client_by_proxy_url[proxy_url] = client
 
         return self._client_by_proxy_url[proxy_url]
@@ -345,13 +345,13 @@ def _combine_headers(self, explicit_headers: HttpHeaders | None) -> HttpHeaders
         return headers or None
 
     @staticmethod
-    def _is_proxy_error(error: httpx.TransportError) -> bool:
+    def _is_proxy_error(error: httpx2.TransportError) -> bool:
         """Determine whether the given error is related to a proxy issue.
 
-        Check if the error is an instance of `httpx.ProxyError` or if its message contains known proxy-related
+        Check if the error is an instance of `httpx2.ProxyError` or if its message contains known proxy-related
         error keywords.
         """
-        if isinstance(error, httpx.ProxyError):
+        if isinstance(error, httpx2.ProxyError):
             return True
 
         if any(needle in str(error) for needle in ROTATE_PROXY_ERRORS):  # noqa: SIM103

tests/unit/http_clients/test_http_clients.py

@@ -308,7 +308,7 @@ async def test_stream_rejects_non_http_scheme(http_client: HttpClient) -> None:
     ('optional_module_name', 'import_path'),
     [
         pytest.param('curl_cffi', 'crawlee.http_clients._curl_impersonate', id='curl_impersonate'),
-        pytest.param('httpx', 'crawlee.http_clients._httpx', id='httpx'),
+        pytest.param('httpx2', 'crawlee.http_clients._httpx', id='httpx'),
     ],
 )
 def test_import_error_handled(optional_module_name: str, import_path: str) -> None:

tests/unit/http_clients/test_httpx.py

@@ -25,8 +25,8 @@ async def http_client() -> AsyncGenerator[HttpClient]:
 
 
 def test_silences_httpx_request_logging() -> None:
-    """Instantiating the client lowers the noisy per-request `httpx` INFO logs to WARNING by default."""
-    httpx_logger = logging.getLogger('httpx')
+    """Instantiating the client lowers the noisy per-request `httpx2` INFO logs to WARNING by default."""
+    httpx_logger = logging.getLogger('httpx2')
     httpx_logger.setLevel(logging.NOTSET)
 
     HttpxHttpClient()
@@ -35,9 +35,9 @@ def test_silences_httpx_request_logging() -> None:
 
 
 async def test_common_headers_and_user_agent(server_url: URL, header_network: dict) -> None:
-    """Test that the relevant headers use header values from header generator instead of default Httpx headers.
+    """Test that the relevant headers use header values from header generator instead of default HTTPX2 headers.
 
-    Httpx uses own headers by default which is not desired as it could increase blocking chances.
+    HTTPX2 uses own headers by default which is not desired as it could increase blocking chances.
     """
     client = HttpxHttpClient()
 
@@ -50,7 +50,7 @@ async def test_common_headers_and_user_agent(server_url: URL, header_network: di
     assert 'accept-language' in response_headers
     assert response_headers['accept-language'] == COMMON_ACCEPT_LANGUAGE
 
-    # By default, HTTPX uses its own User-Agent, which should be replaced by the one from the header generator.
+    # By default, HTTPX2 uses its own User-Agent, which should be replaced by the one from the header generator.
     assert 'user-agent' in response_headers
     assert 'python-httpx' not in response_headers['user-agent']
     assert response_headers['user-agent'] in get_available_header_values(header_network, {'User-Agent', 'user-agent'})