diff --git a/sources/platform/integrations/ai/index.mdx b/sources/platform/integrations/ai/index.mdx index a48316cd05..3e8be55298 100644 --- a/sources/platform/integrations/ai/index.mdx +++ b/sources/platform/integrations/ai/index.mdx @@ -44,6 +44,13 @@ Plug Apify Actors into the AI stack - chat clients like Claude and ChatGPT via t imageUrl="/img/platform/integrations/langchain.png" smallImage /> + For more information on LangChain visit its [documentation](https://docs.langchain.com/oss/python/langchain/overview). The Apify integration lives in the [langchain-apify](https://github.com/apify/langchain-apify) repository. -In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs and extract text content from the web pages. -Then we feed the documents into a vector index and answer questions from it. - -This example demonstrates how to integrate Apify with LangChain in Python. +The `langchain-apify` package connects Apify Actors to LangChain. Use it to pull live web data into a vector index for retrieval, or to give an agent a set of scraping tools it can call on its own. :::info Python only @@ -22,13 +21,52 @@ The `langchain-apify` package is currently available for Python only. ::: +## What's on this page + +- [Quick start](#quick-start) - scrape a single URL to confirm your setup works. +- [Load web data into a vector index](#load-web-data-into-a-vector-index) - crawl a site with `ApifyWrapper`, then answer questions from the crawled documents. +- [Use Actors as LangChain tools](#use-actors-as-langchain-tools) - bind dedicated tools for web search, crawling, and social media to an agent. +- [Tool reference](#tool-reference) - all 19 tools and the Actor each one wraps. + +For stateful or multi-agent workflows, see the [LangGraph integration](/integrations/langgraph), which uses the same package. + +## Quick start + +Install the package: + +```bash +pip install langchain-apify +``` + +Then scrape a page to markdown with a single tool. This needs no LLM and no OpenAI key: + +```python +import os + +from langchain_apify import ApifyScrapeUrlTool + +os.environ["APIFY_TOKEN"] = "Your Apify API token" + +tool = ApifyScrapeUrlTool() +print(tool.invoke({"url": "https://docs.apify.com"})) +``` + +Find your token in [Apify Console](https://console.apify.com/settings/integrations). The tool returns a JSON string holding the run's metadata and the scraped markdown, in the single item's `content` field. + +## Load web data into a vector index + +In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs and extract text content from the web pages. +Then we feed the documents into a vector index and answer questions from it. + +### Install the packages + Before we start with the integration, we need to install all dependencies: ```bash pip install langchain-openai langchain-apify ``` -After successful installation of all dependencies, we can start writing code. +### Import the packages First, import all required packages: @@ -43,16 +81,18 @@ from langchain_openai import ChatOpenAI from langchain_openai.embeddings import OpenAIEmbeddings ``` -Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize these into environment variable: +### Set the environment variables + +Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize them as environment variables: ```python os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" os.environ["APIFY_TOKEN"] = "Your Apify API token" ``` -Run the Actor, wait for it to finish, and fetch its results from the Apify dataset into a LangChain document loader. +### Crawl a website -Note that if you already have some results in an Apify dataset, you can load them directly using `ApifyDatasetLoader`, as shown in [this notebook](https://github.com/langchain-ai/langchain/blob/fe1eb8ca5f57fcd7c566adfc01fa1266349b72f3/docs/modules/indexes/document_loaders/examples/apify_dataset.ipynb). In that notebook, you'll also find the explanation of the `dataset_mapping_function`, which is used to map fields from the Apify dataset records to LangChain `Document` fields. +Run the Actor, wait for it to finish, and fetch its results from the Apify dataset into a LangChain document loader: ```python apify = ApifyWrapper() @@ -73,6 +113,27 @@ The Actor call may take some time as it crawls the LangChain documentation websi ::: +The `dataset_mapping_function` converts each raw Apify dataset item into a LangChain `Document`, mapping dataset fields (for example `text` and `url`) onto the `Document`'s `page_content` and `metadata`. Whatever keys the function assigns to `metadata` are the ones available downstream. + +#### Load results from an existing dataset + +If the results are already in an Apify dataset, skip the Actor call and load them directly with `ApifyDatasetLoader`, passing the dataset ID and the same kind of mapping function: + +```python +from langchain_apify import ApifyDatasetLoader +from langchain_core.documents import Document + +loader = ApifyDatasetLoader( + dataset_id="your-dataset-id", + dataset_mapping_function=lambda item: Document( + page_content=item["text"] or "", metadata={"source": item["url"]} + ), +) +documents = loader.load() +``` + +### Build and query the vector index + Initialize the vector index from the crawled documents: ```python @@ -98,6 +159,8 @@ print("answer:", answer) print("source:", sources) ``` +### Run the complete example + If you want to test the whole example, you can simply create a new file, `langchain_integration.py`, and copy the whole code into it. ```python @@ -151,8 +214,7 @@ answer: LangChain is a framework designed for developing applications powered by source: https://docs.langchain.com/oss/python/langchain/quickstart ``` -LangChain is a standard interface through which you can interact with a variety of large language models (LLMs). -It provides modules you can use to build language model applications as well as chains and agents with memory capabilities. +### Use a different Actor You can use all of Apify’s Actors as document loaders in LangChain. For example, to incorporate web browsing functionality, you can use the [RAG-Web-Browser Actor](https://apify.com/apify/rag-web-browser). @@ -242,7 +304,7 @@ Most tools return a JSON string with two keys: `run` (run metadata such as `stat ### Give the tools to an agent -To let a model decide when to call the tools, bind a tool list to an agent. The example below uses LangGraph's prebuilt ReAct agent, so install it alongside the previous dependencies: +To let a model decide when to call the tools, bind a tool list to an agent. The example below uses LangGraph's prebuilt ReAct agent, so install it alongside the previous dependencies. For a fuller walkthrough of multi-tool agents and streaming, see the [LangGraph integration](/integrations/langgraph). ```bash pip install langgraph @@ -335,7 +397,21 @@ tool = ApifyActorsTool("apify/google-trends-scraper") result = tool.invoke({"run_input": {"searchTerms": ["web scraping", "data extraction"]}}) ``` +## Next steps + + + + + ## Resources +- [Apify Actors](/actors) - [LangChain quickstart](https://docs.langchain.com/oss/python/langchain/quickstart) +- [LangChain Apify provider page](https://docs.langchain.com/oss/python/integrations/providers/apify) - [langchain-apify repository](https://github.com/apify/langchain-apify) diff --git a/sources/platform/integrations/ai/langchain/langgraph.md b/sources/platform/integrations/ai/langchain/langgraph.md new file mode 100644 index 0000000000..9d1edd5174 --- /dev/null +++ b/sources/platform/integrations/ai/langchain/langgraph.md @@ -0,0 +1,221 @@ +--- +title: 🦜🔘➡️ LangGraph integration +sidebar_label: LangGraph +description: Learn how to build stateful multi-agent AI workflows with LangGraph and Apify Actors to search, extract, and analyze real-time web data at scale. +slug: /integrations/langgraph +--- + +import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx'; + +[LangGraph](https://www.langchain.com/langgraph) is a framework for constructing stateful, multi-agent applications with large language models (LLMs). Developers use it to build multi-step agent workflows that call tools, APIs, and databases. For more details, check out the [LangGraph documentation](https://docs.langchain.com/oss/python/langgraph/overview). + + + +LangGraph support comes from the same `langchain-apify` package as the [LangChain integration](/integrations/langchain). This page covers binding Apify tools to a LangGraph agent. See the LangChain page for the [full tool reference](/integrations/langchain#tool-reference), tool set selection, and non-agent uses such as document loading and retrieval. + +## Quick start + +Install the packages: + +```bash +pip install langgraph langchain-apify langchain-openai +``` + +Then give a model one Apify tool and let it answer from live web data: + +```python +import os + +from langchain_apify import ApifyRAGWebBrowserTool +from langchain_openai import ChatOpenAI +from langgraph.prebuilt import create_react_agent + +os.environ["APIFY_TOKEN"] = "Your Apify API token" +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" + +agent = create_react_agent(ChatOpenAI(model="gpt-5.4-mini"), [ApifyRAGWebBrowserTool()]) +result = agent.invoke({"messages": [("human", "Search the web and tell me what Apify is.")]}) +print(result["messages"][-1].content) +``` + +The rest of this page builds on that: [several tools with streamed steps](#build-the-tiktok-profile-search-and-analysis-agent), [a whole tool set at once](#bind-a-whole-tool-set), and [any other Actor](#run-any-other-actor). + +## How to use Apify with LangGraph + +This guide will demonstrate how to use Apify Actors with LangGraph by building a ReAct agent that searches the web for TikTok profiles and extracts data from them, using two dedicated Apify tools: `ApifyRAGWebBrowserTool` for the search and `ApifyTikTokScraperTool` for the profile data. + +### Prerequisites + +- **Apify API token**: To use Apify Actors in LangGraph, you need an Apify API token. If you don't have one, you can learn how to obtain it in the [Apify documentation](/integrations/api). + +- **OpenAI API key**: In order to work with agents in LangGraph, you need an OpenAI API key. If you don't have one, you can get it from the [OpenAI platform](https://platform.openai.com/account/api-keys). + +- **Python packages**: You need to install the following Python packages: + + ```bash + pip install langgraph langchain-apify langchain-openai + ``` + +### Build the TikTok profile search and analysis agent + +First, import all required packages: + +```python +import os + +from langchain_apify import ApifyRAGWebBrowserTool, ApifyTikTokScraperTool +from langchain_core.messages import HumanMessage +from langchain_openai import ChatOpenAI +from langgraph.prebuilt import create_react_agent +``` + +Next, set the environment variables for the Apify API token and OpenAI API key: + +```python +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" +os.environ["APIFY_TOKEN"] = "Your Apify API token" +``` + +Instantiate the LLM and the Apify tools: + +```python +llm = ChatOpenAI(model="gpt-5.4-mini") + +browser = ApifyRAGWebBrowserTool() +tiktok = ApifyTikTokScraperTool() +``` + +Each tool wraps one Actor behind a simplified input schema, so the model calls it without knowing Actor IDs or Actor input schemas. + +:::tip Register only the tools you need + +The `langchain-apify` package ships 19 tools grouped into three sets. Every tool you register widens the model's decision space, which can cause wrong tool selection, slower responses, and higher token usage. See [choosing the right tool set](/integrations/langchain#choose-the-right-tool-set) for the full list and the tool set imports. + +::: + +Create the ReAct agent with the LLM and Apify tools: + +```python +tools = [browser, tiktok] +agent_executor = create_react_agent(llm, tools) +``` + +Finally, run the agent and stream the messages: + +```python +for state in agent_executor.stream( + stream_mode="values", + input={ + "messages": [ + HumanMessage(content="Search the web for OpenAI TikTok profile and analyze their profile.") + ] + }): + state["messages"][-1].pretty_print() +``` + +:::note Search and analysis may take some time + +Each tool call runs a real Actor on the Apify platform, so the agent may take from seconds to minutes to finish. + +::: + +You will see the agent's messages in the console, which will show each step of the agent's workflow. The output below is abbreviated: + +```text +================================ Human Message ================================= + +Search the web for OpenAI TikTok profile and analyze their profile. +================================== AI Message ================================== +Tool Calls: + apify_rag_web_browser (call_y2rbmQ6gYJYC2lHzWJAoKDaq) + Call ID: call_y2rbmQ6gYJYC2lHzWJAoKDaq + Args: + query: OpenAI TikTok profile + max_results: 1 + +... + +================================== AI Message ================================== +Tool Calls: + apify_tiktok_scraper (call_yQ0mLqXvRp8bT3nZKcWuHsAe) + Call ID: call_yQ0mLqXvRp8bT3nZKcWuHsAe + Args: + search_query: https://www.tiktok.com/@openai + search_type: user + max_results: 5 + +... + +================================== AI Message ================================== + +The OpenAI TikTok profile is "OpenAI (@openai) Official". Here are some key details +about the profile: + +- **Description**: The profile features "low key research previews" and includes + videos that showcase their various projects and research developments. +- **Content focus**: The posts primarily involve previews of OpenAI's research and + various AI-related innovations. + +... + +``` + +If you want to test the whole example, you can simply create a new file, `langgraph_integration.py`, and copy the whole code into it. + +```python +import os + +from langchain_apify import ApifyRAGWebBrowserTool, ApifyTikTokScraperTool +from langchain_core.messages import HumanMessage +from langchain_openai import ChatOpenAI +from langgraph.prebuilt import create_react_agent + +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" +os.environ["APIFY_TOKEN"] = "Your Apify API token" + +llm = ChatOpenAI(model="gpt-5.4-mini") + +browser = ApifyRAGWebBrowserTool() +tiktok = ApifyTikTokScraperTool() + +tools = [browser, tiktok] +agent_executor = create_react_agent(llm, tools) + +for state in agent_executor.stream( + stream_mode="values", + input={ + "messages": [ + HumanMessage(content="Search the web for OpenAI TikTok profile and analyze their profile.") + ] + }): + state["messages"][-1].pretty_print() +``` + +### Bind a whole tool set + +Instead of importing tools one by one, you can give the agent an entire category. Each list holds tool *classes*, so instantiate them before passing them to the agent: + +```python +from langchain_apify import APIFY_SEARCH_TOOLS, APIFY_SOCIAL_TOOLS + +tools = [tool_cls() for tool_cls in APIFY_SEARCH_TOOLS + APIFY_SOCIAL_TOOLS] +agent_executor = create_react_agent(llm, tools) +``` + +### Run any other Actor + +Actors without a dedicated tool go through [`ApifyActorsTool`](/integrations/langchain#run-any-other-actor), which binds to an agent the same way: + +```python +from langchain_apify import ApifyActorsTool + +trends = ApifyActorsTool("apify/google-trends-scraper") +agent_executor = create_react_agent(llm, [trends]) +``` + +## Resources + +- [Apify Actors](/actors) +- [LangChain integration](/integrations/langchain) - installation, full tool reference, loaders, and retrievers +- [LangGraph documentation](https://docs.langchain.com/oss/python/langgraph/overview) +- [LangChain Apify provider page](https://docs.langchain.com/oss/python/integrations/providers/apify) diff --git a/sources/platform/integrations/ai/langgraph.md b/sources/platform/integrations/ai/langgraph.md deleted file mode 100644 index 8dee446edf..0000000000 --- a/sources/platform/integrations/ai/langgraph.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: 🦜🔘➡️ LangGraph integration -sidebar_label: LangGraph -description: Learn how to build stateful multi-agent AI workflows with LangGraph and Apify Actors to search, extract, and analyze real-time web data at scale. -slug: /integrations/langgraph ---- - -import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx'; - -[LangGraph](https://www.langchain.com/langgraph) is a framework for constructing stateful, multi-agent applications with large language models (LLMs). It allows developers to build complex AI agent workflows that can leverage tools, APIs, and databases. For more details, check out the [LangGraph documentation](https://langchain-ai.github.io/langgraph/). - - - -## How to use Apify with LangGraph - -This guide will demonstrate how to use Apify Actors with LangGraph by building a ReAct agent that will use the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor to search Google for TikTok profiles and [TikTok Data Extractor](https://apify.com/clockworks/free-tiktok-scraper) Actor to extract data from the TikTok profiles to analyze the profiles. - -### Prerequisites - -- **Apify API token**: To use Apify Actors in LangGraph, you need an Apify API token. If you don't have one, you can learn how to obtain it in the [Apify documentation](https://docs.apify.com/integrations/api). - -- **OpenAI API key**: In order to work with agents in LangGraph, you need an OpenAI API key. If you don't have one, you can get it from the [OpenAI platform](https://.openai.com/account/api-keys). - -- **Python packages**: You need to install the following Python packages: - - ```bash - pip install langgraph langchain-apify langchain-openai - ``` - -### Build the TikTok profile search and analysis agent - -First, import all required packages: - -```python -import os - -from langchain_apify import ApifyActorsTool -from langchain_core.messages import HumanMessage -from langchain_openai import ChatOpenAI -from langgraph.prebuilt import create_react_agent -``` - -Next, set the environment variables for the Apify API token and OpenAI API key: - -```python -os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" -os.environ["APIFY_API_TOKEN"] = "Your Apify API token" -``` - -Instantiate LLM and Apify Actors tools: - -```python -llm = ChatOpenAI(model="gpt-4o-mini") - -browser = ApifyActorsTool("apify/rag-web-browser") -tiktok = ApifyActorsTool("clockworks/free-tiktok-scraper") -``` - -Create the ReAct agent with the LLM and Apify Actors tools: - -```python -tools = [browser, tiktok] -agent_executor = create_react_agent(llm, tools) -``` - -Finally, run the agent and stream the messages: - -```python -for state in agent_executor.stream( - stream_mode="values", - input={ - "messages": [ - HumanMessage(content="Search the web for OpenAI TikTok profile and analyze their profile.") - ] - }): - state["messages"][-1].pretty_print() -``` - -:::note Search and analysis may take some time - -The agent tool call may take some time as it searches the web for OpenAI TikTok profiles and analyzes them. - -::: - -You will see the agent's messages in the console, which will show each step of the agent's workflow. - -```text -================================ Human Message ================================= - -Search the web for OpenAI TikTok profile and analyze their profile. -================================== AI Message ================================== -Tool Calls: - apify_actor_apify_rag-web-browser (call_y2rbmQ6gYJYC2lHzWJAoKDaq) - Call ID: call_y2rbmQ6gYJYC2lHzWJAoKDaq - Args: - run_input: {"query":"OpenAI TikTok profile","maxResults":1} - -... - -================================== AI Message ================================== - -The OpenAI TikTok profile is titled "OpenAI (@openai) Official." Here are some key details about the profile: - -- **Followers**: 592.3K -- **Likes**: 3.3M -- **Description**: The profile features "low key research previews" and includes videos that showcase their various projects and research developments. - -### Profile Overview: -- **Profile URL**: [OpenAI TikTok Profile](https://www.tiktok.com/@openai?lang=en) -- **Content Focus**: The posts primarily involve previews of OpenAI's research and various AI-related innovations. - -... - -``` - -If you want to test the whole example, you can simply create a new file, `langgraph_integration.py`, and copy the whole code into it. - -```python -import os - -from langchain_apify import ApifyActorsTool -from langchain_core.messages import HumanMessage -from langchain_openai import ChatOpenAI -from langgraph.prebuilt import create_react_agent - -os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" -os.environ["APIFY_API_TOKEN"] = "Your Apify API token" - -llm = ChatOpenAI(model="gpt-4o-mini") - -browser = ApifyActorsTool("apify/rag-web-browser") -tiktok = ApifyActorsTool("clockworks/free-tiktok-scraper") - -tools = [browser, tiktok] -agent_executor = create_react_agent(llm, tools) - -for state in agent_executor.stream( - stream_mode="values", - input={ - "messages": [ - HumanMessage(content="Search the web for OpenAI TikTok profile and analyze their profile.") - ] - }): - state["messages"][-1].pretty_print() -``` - -## Resources - -- [Apify Actors](https://docs.apify.com/actors) -- [LangGraph - How to Create a ReAct Agent](https://langchain-ai.github.io/langgraph/how-tos/create-react-agent/)