Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions pods/configuration/expose-ports.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the SMOG grade (12.53) below 10.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the LIX score (43.96) below 35.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the Flesch reading ease score (42.83) above 70.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the Automated Readability Index (10.12) below 8.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the Coleman–Liau Index grade (13.36) below 9.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the Gunning-Fog index (12.17) below 10.

Check warning on line 1 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L1

Try to keep the Flesch–Kincaid grade level (10.42) below 8.
title: "Expose ports"
description: "Learn how to make your Pod services accessible from the internet using HTTP proxy and TCP port forwarding"
---
Expand All @@ -15,19 +15,19 @@
uvicorn main:app --host 0.0.0.0 --port 4000
```

The external port users connect to will be different, depending on your chosen exposure method.

Check warning on line 18 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L18

Avoid using 'will'.

## HTTP access via Runpod proxy

Check warning on line 20 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L20

'HTTP access via Runpod proxy' should use sentence-style capitalization.

Runpod's HTTP proxy provides the easiest way to expose web services from your Pod. This method works well for REST APIs, web applications, and any HTTP-based service.

Check warning on line 22 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L22

Spell out 'REST', if it's unfamiliar to the audience.

### Configure external HTTP ports

Check warning on line 24 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L24

'Configure external HTTP ports' should use sentence-style capitalization.

To configure HTTP ports during Pod deployment, click **Edit Template** and add a comma-separated list of ports to the **Expose HTTP Ports (Max 10)** field.

Check warning on line 26 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L26

Use parentheses judiciously.

To configure HTTP ports for an existing Pod, navigate to the [Pod page](https://www.console.runpod.io/pods), expand your Pod, click the hamburger menu on the bottom-left, select **Edit Pod**, then add your port(s) to the **Expose HTTP Ports (Max 10)** field.

Check warning on line 28 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L28

Don't use plurals in parentheses such as in 'port(s)'.

Check warning on line 28 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L28

Use parentheses judiciously.

Check warning on line 28 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L28

Use parentheses judiciously.

You can also configure HTTP ports for a Pod template in the [My Templates](https://www.console.runpod.io/user/templates) section of the console.

Check warning on line 30 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L30

Avoid first-person pronouns such as 'My'.

### Access your service

Expand Down Expand Up @@ -61,16 +61,30 @@
This architecture introduces important limitations:

- **100-second timeout**: Cloudflare enforces a maximum connection time of 100 seconds. If your service doesn't respond within this time, the connection closes with a `524` error.
- **HTTPS only**: All connections are secured with HTTPS, even if your internal service uses HTTP.

Check warning on line 64 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L64

In general, use active voice instead of passive voice ('are secured').
- **Public accessibility**: Your service becomes publicly accessible. While the Pod ID provides some obscurity, implement proper authentication in your application.

Check warning on line 65 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L65

Use 'app' instead of 'application'.

Design your application with these constraints in mind. For long-running operations, consider:

Check warning on line 67 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L67

Use 'app' instead of 'application'.
- Implementing progress endpoints that return status updates.
- Using background job queues with status polling.
- Breaking large operations into smaller chunks.
- Returning immediate responses with job IDs for later retrieval.

### Restrict access by IP address

By default, anyone who knows your Pod's proxy URL can reach your service. To limit access to specific source IP addresses, set the `RUNPOD_ALLOW_IP` environment variable on your Pod to a comma-separated list of IP addresses or IP address ranges:

```bash
RUNPOD_ALLOW_IP=203.0.113.7,198.51.100.0/24
```

The proxy checks the source IP of each request against this list before forwarding it to your Pod. Requests from addresses that aren't on the list receive a `400 Bad Request` response with an empty body. If the variable is empty or unset, the proxy allows all IP addresses.

You can set `RUNPOD_ALLOW_IP` in a template or on an existing Pod under **Environment Variables**. See [Environment variables](/pods/templates/environment-variables#runpod-reserved-variables) for details.

`RUNPOD_ALLOW_IP` only applies to ports exposed through the HTTP proxy. It doesn't restrict [TCP ports exposed via public IP](#tcp-access-via-public-ip), including SSH. To restrict access on TCP ports, configure a firewall inside your container or add authentication to your application.

Check warning on line 85 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L85

Use 'app' instead of 'application'.

## TCP access via public IP

Check warning on line 87 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L87

'TCP access via public IP' should use sentence-style capitalization.

<Warning>
Pods do not support UDP connections. If your application relies on UDP, you'll need to modify your application to use TCP-based communication instead.
Expand All @@ -78,7 +92,7 @@

For services requiring direct TCP connections, lower latency, or protocols other than HTTP, use TCP port exposure with public IP addresses.

### Configure TCP ports

Check warning on line 95 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L95

'Configure TCP ports' should use sentence-style capitalization.

In your Pod or template configuration, follow the same steps as for [HTTP ports](#configure-external-http-ports), but add ports to the **Expose TCP Ports** field. This enables direct TCP forwarding with a public IP address.

Expand All @@ -102,20 +116,20 @@

### Requesting symmetrical ports

To request symmetrical mapping, specify port numbers above 70000 in your TCP configuration. These aren't valid port numbers, but signal Runpod to allocate matching internal and external ports.

Check warning on line 119 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L119

Use 'preceding' instead of 'above'.

After Pod creation, check the **Connect** menu to see which symmetrical ports were assigned under **Direct TCP Ports**.

Check warning on line 121 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L121

In general, use active voice instead of passive voice ('were assigned').

### Accessing port mappings programmatically

Your application can discover assigned ports through environment variables. For example, if you specify `70000` and `70001` in your Pod configuration, you could use the following commands to retrieve the assigned ports:

Check warning on line 125 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L125

Use 'app' instead of 'application'.

```bash
echo $RUNPOD_TCP_PORT_70000
echo $RUNPOD_TCP_PORT_70001
```

You can use these environment variables in your application configuration to automatically adapt to assigned ports:

Check warning on line 132 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L132

Use 'app' instead of 'application'.

**Python example:**
```python
Expand All @@ -140,13 +154,14 @@
### Security considerations

- **Implement authentication**: Both HTTP proxy and TCP access make your services publicly accessible. Always implement proper authentication and authorization in your applications.
- **Restrict access by IP**: For HTTP proxy ports, set [`RUNPOD_ALLOW_IP`](#restrict-access-by-ip-address) to limit access to known IP addresses or ranges. This doesn't cover TCP ports.
- **Use HTTPS for sensitive data**: While the proxy automatically provides HTTPS, TCP connections do not. Implement TLS in your application when handling sensitive data over TCP.

Check warning on line 158 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L158

Use 'don't' instead of 'do not'.

Check warning on line 158 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L158

Spell out 'TLS', if it's unfamiliar to the audience.

Check warning on line 158 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L158

Use 'app' instead of 'application'.
- **Validate input**: Public endpoints are targets for malicious traffic. Implement robust input validation and rate limiting.

### Performance optimization

- **Choose the right method**: Use HTTP proxy for web services and TCP for everything else. The proxy adds latency but provides automatic HTTPS and load balancing.
- **Handle timeouts gracefully**: Design your application to work within the 100-second proxy timeout or use TCP for long-running connections.

Check warning on line 164 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L164

Use 'app' instead of 'application'.
- **Monitor your services**: Implement health checks and monitoring to ensure your exposed services remain accessible.

### Configuration tips
Expand All @@ -159,7 +174,7 @@

Different types of applications benefit from different exposure methods:

- **Web APIs and REST services**: Use HTTP proxy for automatic HTTPS and simple configuration.

Check warning on line 177 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L177

Spell out 'REST', if it's unfamiliar to the audience.
- **WebSocket applications**: TCP exposure often works better for persistent connections that might exceed timeout limits.
- **Database connections**: Use TCP with proper security measures. Consider using Runpod's global networking for internal-only databases.
- **Development environments**: HTTP proxy works well for web-based IDEs and development servers.
Expand All @@ -168,11 +183,12 @@

Try these fixes if you're having issues with port exposure:

- **Service not accessible via proxy**: Ensure your service binds to `0.0.0.0` (all interfaces) not just `localhost` or `127.0.0.1`.

Check warning on line 186 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L186

Use parentheses judiciously.
- **400 Bad Request with an empty body from the proxy URL**: Your IP address isn't in the Pod's [`RUNPOD_ALLOW_IP`](#restrict-access-by-ip-address) list. Add your address or remove the variable.
- **524 timeout errors**: If your service takes longer than 100 seconds to respond, consider using TCP or restructuring your application for faster responses.

Check warning on line 188 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L188

Use 'app' instead of 'application'.
- **Connection refused**: Verify your service is running and listening on the correct port inside the Pod.
- **Port already in use**: Check that no other services in your Pod are using the same port.
- **Unstable connections**: For Community Cloud Pods, implement reconnection logic to handle IP address changes.

Check warning on line 191 in pods/configuration/expose-ports.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/configuration/expose-ports.mdx#L191

Use 'Google Cloud Platform' or 'GCP' instead of 'Cloud'.

## Next steps

Expand Down
8 changes: 8 additions & 0 deletions pods/templates/environment-variables.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L1

Try to keep the Flesch reading ease score (41.90) above 70.

Check warning on line 1 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L1

Try to keep the LIX score (40.44) below 35.

Check warning on line 1 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L1

Try to keep the Flesch–Kincaid grade level (9.03) below 8.

Check warning on line 1 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L1

Try to keep the Gunning-Fog index (10.46) below 10.

Check warning on line 1 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L1

Try to keep the Coleman–Liau Index grade (12.08) below 9.
title: "Environment variables"
description: "Configure Pods with environment variables for settings, secrets, and runtime information. See setup and usage details for Runpod Pods."
---
Expand All @@ -16,7 +16,7 @@
2. Click **Add Environment Variable** and enter the key-value pair.

**In Pod templates:**
1. Navigate to [My Templates](https://www.console.runpod.io/user/templates).

Check warning on line 19 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L19

Avoid first-person pronouns such as 'My'.
2. Create or edit a template and add variables in the **Environment Variables** section.

**Using secrets:**
Expand Down Expand Up @@ -83,18 +83,26 @@
| `RUNPOD_POD_HOSTNAME` | Server hostname. |
| `RUNPOD_GPU_COUNT` | Number of GPUs available. |
| `RUNPOD_CPU_COUNT` | Number of CPUs available. |
| `RUNPOD_PUBLIC_IP` | Public IP address (if available). |

Check warning on line 86 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L86

Use parentheses judiciously.
| `RUNPOD_TCP_PORT_22` | Public port mapped to SSH. |
| `RUNPOD_VOLUME_ID` | Attached network volume ID. |
| `RUNPOD_API_KEY` | Pod-scoped API key. |
| `PUBLIC_KEY` | Authorized SSH public keys. |
| `CUDA_VERSION` | Installed CUDA version. |

Check warning on line 91 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L91

Spell out 'CUDA', if it's unfamiliar to the audience.
| `PYTORCH_VERSION` | Installed PyTorch version. |

## Runpod-reserved variables

Runpod reads these variables if you set them. They change how Runpod handles your Pod rather than configuring your application:

Check warning on line 96 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L96

Use 'app' instead of 'application'.

| Variable | Description |
| --- | --- |
| `RUNPOD_ALLOW_IP` | Comma-separated list of IP addresses or IP address ranges allowed to reach your Pod's HTTP proxy ports, such as `203.0.113.7,198.51.100.0/24`. Requests from other addresses receive a `400` response. Doesn't apply to TCP ports. See [Restrict access by IP address](/pods/configuration/expose-ports#restrict-access-by-ip-address). |

## Best practices

- **Use secrets for sensitive data**: Never hardcode API keys or passwords. Use [Runpod secrets](/pods/templates/secrets).
- **Validate required variables**: Check that critical variables are set before your application starts.

Check warning on line 105 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L105

In general, use active voice instead of passive voice ('are set').

Check warning on line 105 in pods/templates/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

pods/templates/environment-variables.mdx#L105

Use 'app' instead of 'application'.
- **Provide defaults**: Use fallback values for non-critical configuration.
- **Use descriptive names**: Prefer `DATABASE_PASSWORD` over `DB_PASS`.
- **Group related variables**: Use consistent prefixes like `DB_HOST`, `DB_PORT`, `DB_NAME`.
8 changes: 8 additions & 0 deletions serverless/development/environment-variables.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the Flesch–Kincaid grade level (9.80) below 8.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the LIX score (48.04) below 35.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the Flesch reading ease score (40.79) above 70.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the Automated Readability Index (9.28) below 8.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the Coleman–Liau Index grade (13.64) below 9.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the Gunning-Fog index (11.60) below 10.

Check warning on line 1 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L1

Try to keep the SMOG grade (11.12) below 10.
title: "Environment variables"
description: "Configure your Serverless endpoints with environment variables. Review configuration and operations guidance for Runpod Serverless."
---
Expand All @@ -7,19 +7,19 @@

## How environment variables work

Environment variables are set in the Runpod console and are available to your handler at runtime through `os.environ`. Your handler can read these variables to configure its behavior.

Check warning on line 10 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L10

In general, use active voice instead of passive voice ('are set').

## Set environment variables

You can set environment variables in the Runpod console when [creating or editing your endpoint](/serverless/endpoints/overview):

1. Navigate to your endpoint in the [Runpod console](https://console.runpod.io/serverless).
2. Click on the **Settings** tab.

Check warning on line 17 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L17

Use 'click' or 'click in' instead of 'Click on'.
3. Scroll to the **Environment Variables** section.
4. Add your variables as key-value pairs.
5. Click **Save** to apply the changes.

These environment variables will be available to all workers running on your endpoint.

Check warning on line 22 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L22

Avoid using 'will'.

### Access environment variables in your handler

Expand Down Expand Up @@ -48,7 +48,7 @@

### Build-time variables

Build-time variables are set in your Dockerfile using the `ENV` instruction. These are baked into your Docker image during the build:

Check warning on line 51 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L51

In general, use active voice instead of passive voice ('are set').

Check warning on line 51 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L51

In general, use active voice instead of passive voice ('are baked').

```dockerfile title="Dockerfile"
FROM runpod/base:0.4.0-cuda11.8.0
Expand All @@ -65,14 +65,22 @@

### Runtime variables

Runtime variables are set in the Runpod console and can be changed without rebuilding your image. These override build-time variables with the same name:

Check warning on line 68 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L68

In general, use active voice instead of passive voice ('are set').

Check warning on line 68 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L68

In general, use active voice instead of passive voice ('be changed').

Runtime variables are useful for:
- API keys and secrets.
- Environment-specific configuration (dev, staging, prod).

Check warning on line 72 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L72

Use parentheses judiciously.
- Values that change frequently.
- Sensitive information that shouldn't be in your image.

## Runpod-reserved variables

Runpod reserves some variable names. It reads them to change how it handles your endpoint, in addition to passing them to your workers:

| Variable | Description |
| --- | --- |
| `RUNPOD_ALLOW_IP` | Comma-separated list of IP addresses or IP address ranges allowed to send requests to your endpoint, such as `203.0.113.7,198.51.100.0/24`. Requests from other addresses receive a `403` response before API key validation. See [IP allowlist](/serverless/endpoints/endpoint-configurations#ip-allowlist). |

## Common use cases

### API keys and secrets
Expand Down Expand Up @@ -131,7 +139,7 @@
```

Set these variables in the Runpod console:
- `BUCKET_ENDPOINT_URL`: Your bucket endpoint (e.g., `https://your-bucket.s3.us-west-2.amazonaws.com`)

Check warning on line 142 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L142

Use parentheses judiciously.

Check warning on line 142 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L142

Use 'for example' instead of 'e.g.'.
- `BUCKET_ACCESS_KEY_ID`: Your access key ID
- `BUCKET_SECRET_ACCESS_KEY`: Your secret access key

Expand All @@ -141,7 +149,7 @@

### Feature flags

Use environment variables to enable or disable features:

Check warning on line 152 in serverless/development/environment-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/development/environment-variables.mdx#L152

Use 'turn off' or 'off' instead of 'disable'.

```python title="handler.py"
import os
Expand Down
12 changes: 12 additions & 0 deletions serverless/endpoints/endpoint-configurations.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the Coleman–Liau Index grade (12.52) below 9.

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the SMOG grade (11.28) below 10.

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the Flesch–Kincaid grade level (9.72) below 8.

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the LIX score (45.19) below 35.

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the Flesch reading ease score (46.71) above 70.

Check warning on line 1 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L1

Try to keep the Automated Readability Index (9.22) below 8.
title: "Endpoint settings"
sidebarTitle: "Endpoint settings"
description: "Reference guide for all Serverless endpoint settings and parameters. Review configuration and operations guidance for Runpod Serverless."
Expand All @@ -17,27 +17,27 @@

| Setting | Default | Description |
|---------|---------|-------------|
| **Active workers** | 0 | Always-on workers (eliminates cold starts) |

Check warning on line 20 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L20

Use parentheses judiciously.
| **Max workers** | 3 | Maximum concurrent workers |
| **GPUs per worker** | 1 | GPU count per worker instance |
| **Idle timeout** | 5s | Time before idle worker shuts down |

Check warning on line 23 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L23

Put a nonbreaking space between the number and the unit in '5s'.
| **Execution timeout** | 600s (10 min) | Max job duration |

Check warning on line 24 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L24

Put a nonbreaking space between the number and the unit in '600s'.

Check warning on line 24 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L24

Use parentheses judiciously.
| **Job TTL** | 24h | Total job lifespan in system |

Check warning on line 25 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L25

Spell out 'TTL', if it's unfamiliar to the audience.

Check warning on line 25 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L25

Put a nonbreaking space between the number and the unit in '24h'.
| **FlashBoot** | Enabled | Faster cold starts via state retention |

## General configuration

### Endpoint name

Display name for identifying your endpoint in the console. Does not affect the endpoint ID used for API requests.

Check warning on line 32 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L32

Use 'doesn't' instead of 'Does not'.

### Endpoint type

**Queue-based endpoints** use a built-in queueing system with guaranteed execution and automatic retries. Ideal for async tasks, batch processing, and long-running jobs. Implemented using [handler functions](/serverless/workers/handler-functions).

**Load balancing endpoints** route traffic directly to workers, bypassing the queue. Designed for low-latency applications like real-time <InferenceTooltip /> or custom REST APIs. See [Load balancing endpoints](/serverless/load-balancing/overview).

Check warning on line 38 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L38

Spell out 'REST', if it's unfamiliar to the audience.

### GPU configuration

Check warning on line 40 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L40

'GPU configuration' should use sentence-style capitalization.

Determines the hardware tier for your workers. Select multiple GPU categories to create a prioritized fallback list. If your first choice is unavailable, Runpod automatically uses the next option. Selecting multiple types improves availability during high demand.

Expand All @@ -47,7 +47,7 @@

Specify up to three GPU types in priority order when configuring an endpoint. Runpod uses this ranking to distribute workers across available GPUs, improving availability during high demand.

For endpoints with five or more workers, Runpod distributes workers across your selected GPU priorities. Most workers run on your primary GPU type, with fewer assigned to secondary and tertiary selections. This reduces throttling when your primary GPU is constrained.

Check warning on line 50 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L50

Use the Oxford comma in ', with fewer assigned to secondary and tertiary selections.'.

Check warning on line 50 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L50

In general, use active voice instead of passive voice ('is constrained').

For endpoints with fewer than five workers, all workers use the highest-priority GPU type available.

Expand All @@ -61,15 +61,15 @@

Maximum concurrent instances your endpoint can scale to. Acts as a cost safety limit and concurrency cap. Set ~20% higher than expected max concurrency to handle traffic spikes smoothly.

### GPUs per worker

Check warning on line 64 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L64

'GPUs per worker' should use sentence-style capitalization.

Number of GPUs assigned to each worker instance. Default is 1. Generally prioritize fewer high-end GPUs over multiple lower-tier GPUs.

### Auto-scaling type

**Queue delay**: Adds workers when requests wait longer than the threshold (default: 4 seconds). Best when slight delays are acceptable for higher utilization.

Check warning on line 70 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L70

'Adds' should be in lowercase.

Check warning on line 70 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L70

Use parentheses judiciously.

Check warning on line 70 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L70

Avoid the unverifiable claim 'Best'.

**Request count**: More aggressive scaling based on pending + active work. Formula: `Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)`. Use scaler value of 1 for max responsiveness. Recommended for LLM workloads or frequent short requests.

Check warning on line 72 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L72

'More' should be in lowercase.

Check warning on line 72 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L72

Spell out 'LLM', if it's unfamiliar to the audience.

## Lifecycle and timeouts

Expand All @@ -80,24 +80,24 @@
### Idle endpoint scale-down

Runpod automatically scales down endpoints that go a long time without any requests, so unused endpoints don't keep consuming your account balance.
* After 3 days with no requests, the endpoint's max workers is reduced to 2 and Runpod sends you an email notification.

Check warning on line 83 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L83

In general, use active voice instead of passive voice ('is reduced').
* After 7 days with no requests, max workers is set to 0.

Check warning on line 84 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L84

In general, use active voice instead of passive voice ('is set').

This scale-down is automatic and system-driven, and the timer is based on request activity, so any incoming request resets it.

Check warning on line 86 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L86

In general, use active voice instead of passive voice ('is based').

Once an endpoint has been scaled down this way, it stays at its reduced max workers until you raise the value yourself. To use the endpoint again, increase its max workers in the Runpod console. To prevent an endpoint from scaling down in the first place, make sure it continues to receive requests.

Check warning on line 88 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L88

In general, use active voice instead of passive voice ('been scaled').

### Execution timeout

Maximum duration for a single job. When exceeded, the job fails and the worker stops. Keep enabled to prevent runaway jobs. Default: 600s (10 min). Range: 5s to 7 days.

Check warning on line 92 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L92

Put a nonbreaking space between the number and the unit in '600s'.

Check warning on line 92 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L92

Use parentheses judiciously.

Check warning on line 92 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L92

Put a nonbreaking space between the number and the unit in '5s'.

Configure in **Advanced** settings, or override per-request via `executionTimeout` in the [job policy](/serverless/endpoints/send-requests#execution-policies).

### Job TTL (time-to-live)

Check warning on line 96 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L96

'Job TTL (time-to-live)' should use sentence-style capitalization.

Check warning on line 96 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L96

Spell out 'TTL', if it's unfamiliar to the audience.

Check warning on line 96 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L96

Use parentheses judiciously.

Total lifespan of a job in the system. When TTL expires, job data is deleted regardless of state (queued, running, or completed). Default: 24 hours. Range: 10s to 7 days.

Check warning on line 98 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L98

Spell out 'TTL', if it's unfamiliar to the audience.

Check warning on line 98 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L98

In general, use active voice instead of passive voice ('is deleted').

Check warning on line 98 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L98

Use parentheses judiciously.

Check warning on line 98 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L98

Put a nonbreaking space between the number and the unit in '10s'.

The timer starts at submission, not execution. If a job queues for 45 minutes with a 1-hour TTL, only 15 minutes remain for execution.

Check warning on line 100 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L100

Spell out 'TTL', if it's unfamiliar to the audience.

<Warning>
TTL is a hard limit. If it expires while a job is running, the job is immediately removed and status checks return 404. Set TTL to cover both expected queue time and execution time.
Expand All @@ -110,16 +110,16 @@
| Request type | Retention | Notes |
|--------------|-----------|-------|
| Async (`/run`) | 30 min | Retrieve via `/status/{job_id}` |
| Sync (`/runsync`) | 1 min | Returned in response; also available via `/status/{job_id}` |

Check warning on line 113 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L113

Use semicolons judiciously.

Results are permanently deleted after retention expires.

## Performance features

### FlashBoot

Check warning on line 119 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L119

'FlashBoot' should use sentence-style capitalization.

Reduces cold starts by retaining worker state after spin-down, allowing faster "revival" than fresh boots. Most effective on endpoints with consistent traffic where workers frequently cycle between active and idle.
Both new GPU and CPU endpoints will have FlashBoot enabled by default, and you can edit existing endpoints to enable or disable FlashBoot.

Check warning on line 122 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L122

Avoid using 'will'.

Check warning on line 122 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L122

Use 'turn off' or 'off' instead of 'disable'.

### Model

Expand All @@ -135,10 +135,22 @@

[Network volumes](/storage/network-volumes) provide persistent storage across worker restarts. Tradeoffs: adds network latency and restricts your endpoint to the volume's data center. Use only when you need shared persistence or datasets exceeding container limits.

### CUDA version selection

Check warning on line 138 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L138

Spell out 'CUDA', if it's unfamiliar to the audience.

Check warning on line 138 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L138

'CUDA version selection' should use sentence-style capitalization.

Ensures workers run on <MachinesTooltip /> with compatible drivers. Select your required version plus all newer versions, since CUDA is backward compatible and a wider range increases available hardware.

Check warning on line 140 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L140

Spell out 'CUDA', if it's unfamiliar to the audience.

### Expose HTTP/TCP ports

Check warning on line 142 in serverless/endpoints/endpoint-configurations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/endpoint-configurations.mdx#L142

'Expose HTTP/TCP ports' should use sentence-style capitalization.

Exposes the worker's public IP and port for direct external communication. Required for persistent connections like WebSockets.

### IP allowlist

To restrict which source IP addresses can send requests to your endpoint, add a `RUNPOD_ALLOW_IP` environment variable to your endpoint with a comma-separated list of IP addresses or IP address ranges:

```bash
RUNPOD_ALLOW_IP=203.0.113.7,198.51.100.0/24
```

Runpod checks the source IP of every request before validating the API key. Requests from addresses that aren't on the list receive a `403 Forbidden` response with the message `ip address not allowed`. This applies to every endpoint operation, including `/run`, `/runsync`, `/status`, and `/health`, and to both queue-based and load balancing endpoints. Changes take effect within a few seconds and don't restart your workers. If the variable is empty or unset, Runpod allows all IP addresses.

To set the variable, open your endpoint in the [Runpod console](https://console.runpod.io/serverless), go to the **Settings** tab, and add it under **Environment Variables**. See [Environment variables](/serverless/development/environment-variables#runpod-reserved-variables) for details.
1 change: 1 addition & 0 deletions serverless/endpoints/send-requests.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the Gunning-Fog index (10.79) below 10.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the Coleman–Liau Index grade (12.45) below 9.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the Automated Readability Index (9.80) below 8.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the Flesch reading ease score (49.06) above 70.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the LIX score (45.24) below 35.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the Flesch–Kincaid grade level (9.84) below 8.

Check warning on line 1 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L1

Try to keep the SMOG grade (11.51) below 10.
title: "Send API requests"
sidebarTitle: "Send API requests"
description: "Submit and manage jobs for your queue-based endpoints by sending HTTP requests. Review configuration and operations guidance for Runpod Serverless."
Expand All @@ -21,13 +21,13 @@

## How requests work

A **job** is a unit of work containing the input data from the request, packaged for processing by your [workers](/serverless/workers/overview). If no worker is immediately available, the job is queued. Once a worker is available, the job is processed using your worker's [handler function](/serverless/workers/handler-functions).

Check warning on line 24 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L24

In general, use active voice instead of passive voice ('is queued').

Check warning on line 24 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L24

In general, use active voice instead of passive voice ('is processed').

## Sync vs. async

- `/runsync` submits a **synchronous** job.
- Client waits for the job to complete before returning the result.
- Results are available for 1 minute (5 minutes max).

Check warning on line 30 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L30

Use parentheses judiciously.
- Ideal for quick responses and interactive applications.
- `/run` submits an **asynchronous** job.
- The job processes in the background; retrieve results via `/status`.
Expand Down Expand Up @@ -71,7 +71,7 @@
| `/cancel` | POST | Stop a job in progress or waiting in the queue. |
| `/retry` | POST | Requeue a failed or timed-out job with the same job ID and input. |
| `/purge-queue` | POST | Clear all pending jobs from the queue. |
| `/health` | GET | Monitor endpoint status, including worker and job statistics. |

Check warning on line 74 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L74

Use the Oxford comma in ', including worker and job statistics.'.

See the [operation reference](/serverless/endpoints/operation-reference) for detailed examples using cURL and the Runpod SDK.

Expand All @@ -81,7 +81,7 @@

## Advanced options

Beyond the required `input` object, you can include optional top-level parameters for additional functionality.

Check warning on line 84 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L84

Use 'capability' or 'feature' instead of 'functionality'.

### Webhook notifications

Expand Down Expand Up @@ -113,17 +113,17 @@

| Option | Description | Default | Constraints |
|--------------------|----------------------------------------------------------|----------------------|-------------------------|
| `executionTimeout` | Maximum time a job can run while being processed | 600000 (10 minutes) | Min 5 sec, max 7 days |

Check warning on line 116 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L116

In general, use active voice instead of passive voice ('being processed').

Check warning on line 116 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L116

Use parentheses judiciously.
| `lowPriority` | When true, job won't trigger worker scaling | false | - |
| `ttl` | Total lifespan of the job before deletion | 86400000 (24 hours) | Min 10 sec, max 7 days |

Check warning on line 118 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L118

Use parentheses judiciously.

<Info>
Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only.
</Info>

#### TTL vs. execution timeout

Check warning on line 124 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L124

Spell out 'TTL', if it's unfamiliar to the audience.

Check warning on line 124 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L124

'TTL vs. execution timeout' should use sentence-style capitalization.

- **`ttl`**: Total lifespan of the job. Timer starts when submitted and covers queue time, execution time, and everything in between. When TTL expires, the job is deleted regardless of state.

Check warning on line 126 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L126

Spell out 'TTL', if it's unfamiliar to the audience.

Check warning on line 126 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L126

In general, use active voice instead of passive voice ('is deleted').
- **`executionTimeout`**: Maximum time the job can actively run once a worker picks it up. Only enforced during execution.

<Warning>
Expand All @@ -132,7 +132,7 @@

#### Long-running jobs

For jobs that need to run longer than the default 24-hour TTL:

Check warning on line 135 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L135

Spell out 'TTL', if it's unfamiliar to the audience.

1. Set `executionTimeout` to your desired maximum runtime.
2. Set `ttl` to cover **both expected queue time and execution time**.
Expand All @@ -147,7 +147,7 @@
}
```

This allows up to 48 hours of active runtime with 72 hours total lifespan (24 hours headroom for queue time).

Check warning on line 150 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L150

Use parentheses judiciously.

<Warning>
Both `ttl` and `executionTimeout` have a maximum of 7 days. A job with 7-day TTL that queues for 2 days only has 5 days remaining for execution.
Expand All @@ -155,12 +155,12 @@

#### Result retention

After completion, results are retained for a fixed period separate from TTL:

Check warning on line 158 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L158

In general, use active voice instead of passive voice ('are retained').

Check warning on line 158 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L158

Spell out 'TTL', if it's unfamiliar to the audience.

| Request type | Retention period |
|--------------------|------------------|
| `/run` (async) | 30 minutes |

Check warning on line 162 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L162

Use parentheses judiciously.
| `/runsync` (sync) | 1 minute |

Check warning on line 163 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L163

Use parentheses judiciously.

### S3-compatible storage

Expand Down Expand Up @@ -199,7 +199,7 @@

Rate limits scale with your endpoint's worker count. The system uses whichever is higher between:

1. **Base limit**: Fixed rate limit per user per endpoint (shown above)

Check warning on line 202 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L202

Use parentheses judiciously.

Check warning on line 202 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L202

Use 'preceding' instead of 'above'.
2. **Worker-based limit**: `number_of_running_workers × requests_per_worker`

Requests exceeding the effective limit return `429 (Too Many Requests)`. Implement retry logic with exponential backoff to handle rate limiting gracefully.
Expand All @@ -212,15 +212,16 @@
|-------------|-----------------------|---------------------------------------------------|
| 400 | Bad Request | Check your request format and parameters |
| 401 | Unauthorized | Verify your API key is correct and has permission |
| 403 | Forbidden | Your IP address isn't in the endpoint's [IP allowlist](/serverless/endpoints/endpoint-configurations#ip-allowlist) |
| 404 | Not Found | Check your endpoint ID |
| 429 | Too Many Requests | Implement backoff and retry logic |
| 500 | Internal Server Error | Check endpoint logs; worker may have crashed |

Check warning on line 218 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L218

Use semicolons judiciously.

| Issue | Possible Causes | Solutions |
|--------------------|------------------------------------------|---------------------------------------------------------------------|
| Job stuck in queue | No available workers, max workers reached | Increase max workers, check endpoint health |
| Timeout errors | Job takes longer than execution timeout | Increase timeout in job policy, optimize processing |
| Failed jobs | Worker errors, input validation issues | Check [endpoint logs](/serverless/development/logs), verify input |
| Missing results | Results expired | Retrieve within expiration window (30 min async, 1 min sync) |

Check warning on line 225 in serverless/endpoints/send-requests.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/endpoints/send-requests.mdx#L225

Use parentheses judiciously.

See [error handling](/serverless/workers/handler-functions#error-handling) for implementation details.
1 change: 1 addition & 0 deletions serverless/load-balancing/overview.mdx
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the Flesch reading ease score (43.31) above 70.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the Gunning-Fog index (11.09) below 10.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the Coleman–Liau Index grade (13.46) below 9.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the Automated Readability Index (11.01) below 8.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the SMOG grade (12.21) below 10.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the Flesch–Kincaid grade level (10.89) below 8.

Check warning on line 1 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L1

Try to keep the LIX score (46.80) below 35.
title: "Overview"
sidebarTitle: "Overview"
description: "Deploy custom direct-access REST APIs with load balancing Serverless endpoints. Review configuration and operations guidance for Runpod Serverless."

Check warning on line 4 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L4

Spell out 'REST', if it's unfamiliar to the audience.
---

import { RequestsTooltip, QueueBasedEndpointsTooltip } from "/snippets/tooltips.jsx";

Load balancing endpoints route incoming traffic directly to available workers, bypassing the queueing system. Unlike <QueueBasedEndpointsTooltip /> that process requests sequentially, load balancing distributes requests across your worker pool for lower latency.

You can create custom REST endpoints accessible via a unique URL:

Check warning on line 11 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L11

Spell out 'REST', if it's unfamiliar to the audience.

```
https://ENDPOINT_ID.api.runpod.ai/YOUR_CUSTOM_PATH
Expand All @@ -28,15 +28,15 @@

### Queue-based endpoints

With queue-based endpoints, <RequestsTooltip /> are placed in a queue and processed in order. They use the standard handler pattern (`def handler(job)`) and are accessed through fixed endpoints like `/run` and `/runsync`.

Check warning on line 31 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L31

Use parentheses judiciously.

Check warning on line 31 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L31

In general, use active voice instead of passive voice ('are placed').

Check warning on line 31 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L31

In general, use active voice instead of passive voice ('are accessed').

These endpoints are better for tasks that can be processed asynchronously and guarantee request processing, similar to how TCP guarantees packet delivery in networking.

Check warning on line 33 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L33

In general, use active voice instead of passive voice ('be processed').

Check warning on line 33 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L33

Avoid the unverifiable claim 'guarantee'.

Check warning on line 33 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L33

Avoid the unverifiable claim 'guarantees'.

### Load balancing endpoints (new)

Check warning on line 35 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L35

Use parentheses judiciously.

Load balancing endpoints send requests directly to workers without queuing. You can use any HTTP framework such as FastAPI or Flask, and define custom URL paths and API contracts to suit your specific needs.

These endpoints are ideal for real-time applications and streaming, but provide no queuing mechanism for request backlog, similar to UDP's behavior in networking.

Check warning on line 39 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L39

Spell out 'UDP', if it's unfamiliar to the audience.


## Endpoint type comparison table
Expand All @@ -44,15 +44,15 @@
Aspect | Load balancing | Queue-based |
|--------|----------------|-------------|
| **Request flow** | Direct to worker HTTP server | Through queueing system |
| **Implementation** | Custom HTTP server (FastAPI, Flask, etc.) | Handler function |

Check warning on line 47 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L47

Use parentheses judiciously.
| **API flexibility** | Custom URL paths, any HTTP capability | Fixed `/run` and `/runsync` endpoints |
| **Backpressure** | Drops requests when overloaded | Queue buffering |
| **Latency** | Lower (single-hop) | Higher (queue + worker) |

Check warning on line 50 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L50

Use parentheses judiciously.

Check warning on line 50 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L50

Use parentheses judiciously.
| **Error handling** | No built-in retry | Automatic retries |

## Worker comparison

**Queue-based worker** (traditional):

Check warning on line 55 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L55

Use parentheses judiciously.

```python
import runpod
Expand All @@ -64,7 +64,7 @@
runpod.serverless.start({"handler": handler})
```

**Load balancing worker** (custom HTTP server):

Check warning on line 67 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L67

Use parentheses judiciously.

```python
from fastapi import FastAPI
Expand Down Expand Up @@ -116,19 +116,20 @@

| Variable | Default | Description |
|----------|---------|-------------|
| `PORT` | `80` | Main application server port |

Check warning on line 119 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L119

Use 'app' instead of 'application'.
| `PORT_HEALTH` | Same as `PORT` | Health check endpoint port |
| `HEALTH_CHECK_PATH` | `/ping` | Path the load balancer polls to check worker health |
| `RUNPOD_ALLOW_IP` | Unset, allows all | Comma-separated IP addresses or IP address ranges allowed to send requests to the endpoint. See [IP allowlist](/serverless/endpoints/endpoint-configurations#ip-allowlist) |

If using a custom port, add it to your endpoint's environment variables and expose it in container configuration (under **Expose HTTP Ports (Max 10)**).

Check warning on line 124 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L124

Use parentheses judiciously.

## Timeouts and limits

| Limit | Value |
|-------|-------|
| **Request timeout** | 2 min (no worker available) |

Check warning on line 130 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L130

Use parentheses judiciously.
| **Processing timeout** | 5.5 min (per request) |

Check warning on line 131 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L131

Use parentheses judiciously.
| **Payload limit** | 30 MB (request and response) |

Check warning on line 132 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L132

Use parentheses judiciously.

For payloads larger than 30 MB, use [network volumes](/storage/network-volumes) or implement chunking.

Expand Down Expand Up @@ -172,6 +173,6 @@
Use load balancing endpoints when you need:

- Direct access to your model's HTTP server.
- Internal batching systems (like vLLM).

Check warning on line 176 in serverless/load-balancing/overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

serverless/load-balancing/overview.mdx#L176

Use parentheses judiciously.
- Non-JSON payloads.
- Multiple endpoints within a single worker.
Expand Down
Loading