Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions sources/platform/integrations/ai/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ Plug Apify Actors into the AI stack - chat clients like Claude and ChatGPT via t
imageUrl="/img/platform/integrations/langchain.png"
smallImage
/>
<Card
title="LangGraph"
desc="Bind Apify tools to stateful, multi-agent LangGraph workflows."
to="/integrations/langgraph"
imageUrl="/img/platform/integrations/langgraph.png"
smallImage
/>
<Card
title="Vercel AI SDK"
desc="Call Apify Actors from agents built on the Vercel AI SDK."
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
label: 'LangChain'
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,68 @@ description: Learn how to integrate Apify with LangChain to feed vector database
slug: /integrations/langchain
---

import Card from '@site/src/components/Card';
import CardGrid from '@site/src/components/CardGrid';
import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx';

> For more information on LangChain visit its [documentation](https://docs.langchain.com/oss/python/langchain/overview). The Apify integration lives in the [langchain-apify](https://github.com/apify/langchain-apify) repository.

<ThirdPartyDisclaimer />

In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs and extract text content from the web pages.
Then we feed the documents into a vector index and answer questions from it.

This example demonstrates how to integrate Apify with LangChain in Python.
The `langchain-apify` package connects Apify Actors to LangChain. Use it to pull live web data into a vector index for retrieval, or to give an agent a set of scraping tools it can call on its own.

:::info Python only

The `langchain-apify` package is currently available for Python only.

:::

## What's on this page

- [Quick start](#quick-start) - scrape a single URL to confirm your setup works.
- [Load web data into a vector index](#load-web-data-into-a-vector-index) - crawl a site with `ApifyWrapper`, then answer questions from the crawled documents.
- [Use Actors as LangChain tools](#use-actors-as-langchain-tools) - bind dedicated tools for web search, crawling, and social media to an agent.
- [Tool reference](#tool-reference) - all 19 tools and the Actor each one wraps.

For stateful or multi-agent workflows, see the [LangGraph integration](/integrations/langgraph), which uses the same package.

## Quick start

Install the package:

```bash
pip install langchain-apify
```

Then scrape a page to markdown with a single tool. This needs no LLM and no OpenAI key:

```python
import os

from langchain_apify import ApifyScrapeUrlTool

os.environ["APIFY_TOKEN"] = "Your Apify API token"

tool = ApifyScrapeUrlTool()
print(tool.invoke({"url": "https://docs.apify.com"}))
```

Find your token in [Apify Console](https://console.apify.com/settings/integrations). The tool returns a JSON string holding the run's metadata and the scraped markdown, in the single item's `content` field.

## Load web data into a vector index

In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs and extract text content from the web pages.
Then we feed the documents into a vector index and answer questions from it.

### Install the packages

Before we start with the integration, we need to install all dependencies:

```bash
pip install langchain-openai langchain-apify
```

After successful installation of all dependencies, we can start writing code.
### Import the packages

First, import all required packages:

Expand All @@ -43,16 +81,18 @@ from langchain_openai import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings
```

Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize these into environment variable:
### Set the environment variables

Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize them as environment variables:

```python
os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_TOKEN"] = "Your Apify API token"
```

Run the Actor, wait for it to finish, and fetch its results from the Apify dataset into a LangChain document loader.
### Crawl a website

Note that if you already have some results in an Apify dataset, you can load them directly using `ApifyDatasetLoader`, as shown in [this notebook](https://github.com/langchain-ai/langchain/blob/fe1eb8ca5f57fcd7c566adfc01fa1266349b72f3/docs/modules/indexes/document_loaders/examples/apify_dataset.ipynb). In that notebook, you'll also find the explanation of the `dataset_mapping_function`, which is used to map fields from the Apify dataset records to LangChain `Document` fields.
Run the Actor, wait for it to finish, and fetch its results from the Apify dataset into a LangChain document loader:

```python
apify = ApifyWrapper()
Expand All @@ -73,6 +113,27 @@ The Actor call may take some time as it crawls the LangChain documentation websi

:::

The `dataset_mapping_function` converts each raw Apify dataset item into a LangChain `Document`, mapping dataset fields (for example `text` and `url`) onto the `Document`'s `page_content` and `metadata`. Whatever keys the function assigns to `metadata` are the ones available downstream.

#### Load results from an existing dataset

If the results are already in an Apify dataset, skip the Actor call and load them directly with `ApifyDatasetLoader`, passing the dataset ID and the same kind of mapping function:

```python
from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
dataset_id="your-dataset-id",
dataset_mapping_function=lambda item: Document(
page_content=item["text"] or "", metadata={"source": item["url"]}
),
)
documents = loader.load()
```

### Build and query the vector index

Initialize the vector index from the crawled documents:

```python
Expand All @@ -98,6 +159,8 @@ print("answer:", answer)
print("source:", sources)
```

### Run the complete example

If you want to test the whole example, you can simply create a new file, `langchain_integration.py`, and copy the whole code into it.

```python
Expand Down Expand Up @@ -151,8 +214,7 @@ answer: LangChain is a framework designed for developing applications powered by
source: https://docs.langchain.com/oss/python/langchain/quickstart
```

LangChain is a standard interface through which you can interact with a variety of large language models (LLMs).
It provides modules you can use to build language model applications as well as chains and agents with memory capabilities.
### Use a different Actor

You can use all of Apify’s Actors as document loaders in LangChain.
For example, to incorporate web browsing functionality, you can use the [RAG-Web-Browser Actor](https://apify.com/apify/rag-web-browser).
Expand Down Expand Up @@ -242,7 +304,7 @@ Most tools return a JSON string with two keys: `run` (run metadata such as `stat

### Give the tools to an agent

To let a model decide when to call the tools, bind a tool list to an agent. The example below uses LangGraph's prebuilt ReAct agent, so install it alongside the previous dependencies:
To let a model decide when to call the tools, bind a tool list to an agent. The example below uses LangGraph's prebuilt ReAct agent, so install it alongside the previous dependencies. For a fuller walkthrough of multi-tool agents and streaming, see the [LangGraph integration](/integrations/langgraph).

```bash
pip install langgraph
Expand Down Expand Up @@ -335,7 +397,21 @@ tool = ApifyActorsTool("apify/google-trends-scraper")
result = tool.invoke({"run_input": {"searchTerms": ["web scraping", "data extraction"]}})
```

## Next steps

<CardGrid>
<Card
title="LangGraph"
desc="Bind Apify tools to a stateful ReAct agent and stream its steps."
to="/integrations/langgraph"
imageUrl="/img/platform/integrations/langgraph.png"
smallImage
/>
</CardGrid>

## Resources

- [Apify Actors](/actors)
- [LangChain quickstart](https://docs.langchain.com/oss/python/langchain/quickstart)
- [LangChain Apify provider page](https://docs.langchain.com/oss/python/integrations/providers/apify)
- [langchain-apify repository](https://github.com/apify/langchain-apify)
Loading
Loading