Skip to content

docs: Fix max_tokens placement and response schema on Public Endpoints model pages #847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
promptless wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Fix max_tokens placement and response schema on Public Endpoints model pages #847

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 29 additions & 28 deletions public-endpoints/models/granite-4.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---

Check warning on line 1 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L1 

Try to keep the LIX score (36.44) below 35.

Check warning on line 1 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L1 

Try to keep the Flesch reading ease score (63.33) above 70.
title: "IBM Granite 4.0"
sidebarTitle: "IBM Granite 4.0"

Check warning on line 3 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L3 

Spell out 'IBM', if it's unfamiliar to the audience.
description: "A 32-billion-parameter long-context instruction model for text generation. See model inputs and outputs on Runpod Public Endpoints."
---

IBM Granite-4.0-H-Small is a 32B parameter long-context instruct model. It excels at general text generation, instruction following, and conversational AI tasks with support for extended context lengths.

Check warning on line 7 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L7 

Spell out 'IBM', if it's unfamiliar to the audience.

Check warning on line 7 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L7 

Put a nonbreaking space between the number and the unit in '32B'.

<Card title="Try in playground" icon="play" href="https://console.runpod.io/hub/playground/text/granite-4-0-h-small" horizontal>
Test IBM Granite 4.0 in the Runpod Hub playground.
Expand All @@ -18,7 +18,7 @@

## Request

All parameters are passed within the `input` object in the request body.

Check warning on line 21 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L21 

In general, use active voice instead of passive voice ('are passed').

<ParamField body="input.messages" type="array" required>
Array of message objects with role and content.
Expand All @@ -32,26 +32,30 @@
The content of the message.
</ParamField>

<ParamField body="input.sampling_params.max_tokens" type="integer" default="512">
Maximum number of tokens to generate.
<ParamField body="input.max_tokens" type="integer" default="512">
Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`.

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Customer's empirical test of the top-level-vs-nested max_tokens bug was performed on kimi-k3; extrapolated here to Granite 4 on the same Public Endpoints native-route input contract. Not independently verified against a Granite-4-specific test (see open question in verification report).

View source

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Internal corroboration: sibling Public Endpoints model pages (Cogito 671B, GPT-OSS 120B, Qwen3 32B) already document max_tokens/temperature/top_p/top_k at the top level of input on the native route, with no sampling_params wrapper, consistent with the corrected placement.

Source: cogito-671b.mdx:31 · 1ac8c64

</ParamField>

<ParamField body="input.sampling_params.temperature" type="float" default="0.7">
<ParamField body="input.temperature" type="float" default="0.7">
Controls randomness in generation. Lower values make output more deterministic. Range: 0.0-1.0.
</ParamField>

<ParamField body="input.sampling_params.seed" type="integer" default="-1">
<ParamField body="input.seed" type="integer" default="-1">
Seed for reproducible results. Set to -1 for random.
</ParamField>

<ParamField body="input.sampling_params.top_k" type="integer" default="-1">
<ParamField body="input.top_k" type="integer" default="-1">
Restricts sampling to the top K most probable tokens.
</ParamField>

<ParamField body="input.sampling_params.top_p" type="float" default="1">
<ParamField body="input.top_p" type="float" default="1">
Nucleus sampling threshold. Range: 0.0-1.0.
</ParamField>

<Note>
Sampling parameters such as `max_tokens` must sit at the top level of the `input` object on the native route. Nesting them inside a `sampling_params` object has no effect: the values are ignored.
</Note>

<RequestExample>
```bash cURL
curl -X POST "https://api.runpod.ai/v2/granite-4-0-h-small/runsync" \
Expand All @@ -69,20 +73,21 @@
"content": "What is Runpod?"
}
],
"sampling_params": {
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1
}
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1
}
}'
```

```python Python
import os
import requests

RUNPOD_API_KEY = os.environ["RUNPOD_API_KEY"]

response = requests.post(
"https://api.runpod.ai/v2/granite-4-0-h-small/runsync",
headers={
Expand All @@ -98,13 +103,11 @@
},
{"role": "user", "content": "What is Runpod?"},
],
"sampling_params": {
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1,
},
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1,
}
},
)
Expand All @@ -119,7 +122,7 @@
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
Authorization: `Bearer ${process.env.RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
Expand All @@ -131,13 +134,11 @@
},
{ role: "user", content: "What is Runpod?" },
],
sampling_params: {
max_tokens: 512,
temperature: 0.7,
seed: -1,
top_k: -1,
top_p: 1,
},
max_tokens: 512,
temperature: 0.7,
seed: -1,
top_k: -1,
top_p: 1,
},
}),
}
Expand Down Expand Up @@ -222,7 +223,7 @@

## Cost calculation

IBM Granite 4.0 charges \$10.00 per 1M tokens. Example costs:

Check warning on line 226 in public-endpoints/models/granite-4.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/granite-4.mdx#L226 

Spell out 'IBM', if it's unfamiliar to the audience.

| Tokens | Cost |
|--------|------|
Expand Down
149 changes: 91 additions & 58 deletions public-endpoints/models/moonshot-kimi.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L1 

Try to keep the SMOG grade (10.47) below 10.

Check warning on line 1 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L1 

Try to keep the Flesch reading ease score (62.48) above 70.

Check warning on line 1 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L1 

Try to keep the Flesch–Kincaid grade level (8.79) below 8.
title: "Moonshot Kimi"
sidebarTitle: "Moonshot Kimi"
description: "Use Moonshot Kimi models for reasoning, chat, and coding through Runpod Public Endpoints, with supported inputs, outputs, and request examples."
Expand All @@ -22,19 +22,19 @@

## Model variants

Choose a variant by setting the `model` field in the request body (default `kimi-k2.6`). The endpoint slug stays `moonshot-kimi` for every variant.

Check warning on line 25 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L25 

Use parentheses judiciously.

| `model` value | Description | Context window | Input price | Output price |
|---|---|---|---|---|
| `kimi-k2.6` (default) | General-purpose model with thinking and non-thinking modes, agentic capabilities, and multimodal input. | 256K (262,144 tokens) | \$0.95 per 1M tokens | \$4.00 per 1M tokens |

Check warning on line 29 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L29 

Use parentheses judiciously.

Check warning on line 29 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L29 

Use parentheses judiciously.
| `kimi-k2.7-code` | Coding-focused model with thinking mode and agentic capabilities. | 256K (262,144 tokens) | \$0.95 per 1M tokens | \$4.00 per 1M tokens |

Check warning on line 30 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L30 

Use parentheses judiciously.
| `kimi-k3` | Flagship model with always-on reasoning and configurable reasoning effort for long-horizon coding and knowledge work. | 1M (1,048,576 tokens) | \$3.00 per 1M tokens | \$15.00 per 1M tokens |

Check warning on line 31 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L31 

Use parentheses judiciously.

To target a variant other than the default, set the `model` field to its ID. For example, to use the flagship model, set `"model": "kimi-k3"` in the request body (or `model="kimi-k3"` with the OpenAI SDK).

Check warning on line 33 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L33 

Use parentheses judiciously.

## Request

All parameters are passed within the `input` object in the request body.

Check warning on line 37 in public-endpoints/models/moonshot-kimi.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

public-endpoints/models/moonshot-kimi.mdx#L37 

In general, use active voice instead of passive voice ('are passed').

<ParamField body="input.messages" type="array" required>
Array of message objects with role and content.
Expand All @@ -52,55 +52,58 @@
The Kimi variant to use. One of `kimi-k2.6`, `kimi-k2.7-code`, or `kimi-k3`.
</ParamField>

<ParamField body="input.sampling_params.max_tokens" type="integer" default="2048">
Maximum number of tokens to generate.
<ParamField body="input.max_tokens" type="integer" default="2048">
Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`. This cap counts reasoning tokens as well as visible output tokens.

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Customer empirical A/B test on kimi-k3 native /runsync route: nested input.sampling_params.max_tokens=200 was silently ignored (23,094 completion tokens, finish_reason "stop", $0.3464), while input.max_tokens=200 at the top level was honored (200 completion tokens, finish_reason "length", $0.0030). Basis for requiring top-level placement of max_tokens on the native route.

View source

</ParamField>

<ParamField body="input.sampling_params.temperature" type="float" default="1">
<ParamField body="input.reasoning_effort" type="string" default="max">
Controls how much reasoning the model performs before answering. One of `low`, `high`, or `max`. Specific to the `kimi-k3` variant.

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Moonshot AI's own docs confirm reasoning_effort accepts low/high/max (default max) and is specific to Kimi K3 as the top-level replacement for the K2.x thinking configuration.

View source

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Runpod's Kimi K3 technical FAQ corroborates reasoning_effort values low/high/max for the K3 variant served on this endpoint.

View source

</ParamField>

<ParamField body="input.temperature" type="float" default="1">
Controls randomness in generation. Lower values make output more deterministic.
</ParamField>

<ParamField body="input.sampling_params.seed" type="integer">
<ParamField body="input.seed" type="integer">
Seed for reproducible results.
</ParamField>

<ParamField body="input.sampling_params.top_k" type="integer">
<ParamField body="input.top_k" type="integer">
Restricts sampling to the top K most probable tokens.
</ParamField>

<ParamField body="input.sampling_params.top_p" type="float">
<ParamField body="input.top_p" type="float">
Nucleus sampling threshold. Range: 0.0-1.0.
</ParamField>

<Note>
Sampling parameters such as `max_tokens` must sit at the top level of the `input` object on the native route. Nesting them inside a `sampling_params` object has no effect: the values are ignored, and generation continues until the model stops on its own. Check `finish_reason` in the response (`"length"` means the `max_tokens` cap was reached, `"stop"` means the model finished on its own) to confirm the cap took effect.

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Same customer test: nesting sampling parameters inside a sampling_params object on the native route produces no error but has no effect, so generation runs unbounded; finish_reason distinguishes the two cases ("length" = cap applied, "stop" = cap ignored / model stopped on its own).

View source

</Note>

<RequestExample>
```bash cURL
curl -X POST "https://api.runpod.ai/v2/moonshot-kimi/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"model": "kimi-k3",
"reasoning_effort": "max",
"max_tokens": 2048,
"messages": [
{
"role": "system",
"content": "You are Kimi."
},
{
"role": "user",
"content": "What is Runpod?"
}
],
"sampling_params": {
"max_tokens": 2048,
"temperature": 1
},
"model": "kimi-k2.6"
{ "role": "system", "content": "You are Kimi." },
{ "role": "user", "content": "What is Runpod?" }
]
}
}'
```

```python Python
import os
import requests

RUNPOD_API_KEY = os.environ["RUNPOD_API_KEY"]

response = requests.post(
"https://api.runpod.ai/v2/moonshot-kimi/runsync",
headers={
Expand All @@ -109,21 +112,26 @@
},
json={
"input": {
"model": "kimi-k3",
"reasoning_effort": "max",
"max_tokens": 2048,
"messages": [
{"role": "system", "content": "You are Kimi."},
{"role": "user", "content": "What is Runpod?"},
],
"sampling_params": {
"max_tokens": 2048,
"temperature": 1,
},
"model": "kimi-k2.6",
}
},
)

result = response.json()
print(result["output"])

# output is a list; the completion sits under result
completion = result["output"][0]["result"]
print(completion["choices"][0]["message"]["content"])
print(completion["choices"][0]["message"].get("reasoning_content"))
print(completion["choices"][0]["finish_reason"]) # "length" if max_tokens was reached
print(completion["usage"])
print(result["output"][0]["cost"])
```

```javascript JavaScript
Expand All @@ -132,27 +140,32 @@
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
Authorization: `Bearer ${process.env.RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
model: "kimi-k3",
reasoning_effort: "max",
max_tokens: 2048,
messages: [
{ role: "system", content: "You are Kimi." },
{ role: "user", content: "What is Runpod?" },
],
sampling_params: {
max_tokens: 2048,
temperature: 1,
},
model: "kimi-k2.6",
},
}),
}
);

const result = await response.json();
console.log(result.output);

// output is an array; the completion sits under result
const completion = result.output[0].result;
console.log(completion.choices[0].message.content);
console.log(completion.choices[0].message.reasoning_content);
console.log(completion.choices[0].finish_reason); // "length" if max_tokens was reached
console.log(completion.usage);
console.log(result.output[0].cost);
```
</RequestExample>

Expand All @@ -178,19 +191,23 @@
Identifier of the worker that processed the request.
</ResponseField>

<ResponseField name="output" type="object">
The generation result containing the text and usage information.

<ResponseField name="output.choices" type="array">
Array containing the generated text.
</ResponseField>
<ResponseField name="output" type="array">
Array of result objects. Each element contains the cost and the generated chat completion.

<ResponseField name="output.cost" type="float">
<ResponseField name="output[].cost" type="float">
Cost of the generation in USD.
</ResponseField>

<ResponseField name="output.usage" type="object">
Token usage information.
<ResponseField name="output[].result" type="object">
The chat completion. Contains `choices`, `usage`, and metadata (`id`, `object`, `model`, `created`).

<ResponseField name="output[].result.choices" type="array">
Generated choices. Each has an `index`, a `finish_reason` (`"stop"` or `"length"`), and a `message` with `role`, `content`, and, for reasoning variants such as `kimi-k3`, `reasoning_content`.
</ResponseField>

<ResponseField name="output[].result.usage" type="object">
Token usage: `prompt_tokens`, `completion_tokens`, `total_tokens`, `completion_tokens_details` (including `reasoning_tokens`), `prompt_tokens_details` (including `cached_tokens`), and a top-level `cached_tokens` count. The `completion_tokens` count includes reasoning tokens.
</ResponseField>
</ResponseField>
</ResponseField>

Expand All @@ -199,23 +216,39 @@
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 2345,
"workerId": "oqk7ao1uomckye",
"output": {
"choices": [
{
"tokens": [
"Runpod is a cloud computing platform that provides GPU resources for AI and machine learning workloads..."
]
"delayTime": 89,
"executionTime": 10078,
"workerId": "wp3m0d84tlio47",
"output": [
{
"cost": 0.0030309,
"result": {
"id": "chatcmpl-6aaa2dce25da6d26c138da0d",

@promptless promptless Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Response shape (output is an array; output[].result is an OpenAI-style chat completion with choices[0].message.content/reasoning_content and finish_reason; usage uses prompt_tokens/completion_tokens plus completion_tokens_details.reasoning_tokens; cost at output[].cost) comes from the customer's verified kimi-k3 run with top-level max_tokens=200.

View source

"object": "chat.completion",
"model": "kimi-k3",
"created": 1789537742,
"choices": [
{
"index": 0,
"finish_reason": "length",
"message": {
"role": "assistant",
"content": "...",
"reasoning_content": "..."
}
}
],
"usage": {
"prompt_tokens": 103,
"completion_tokens": 200,
"completion_tokens_details": { "reasoning_tokens": 197 },
"prompt_tokens_details": { "cached_tokens": 103 },
"cached_tokens": 103,
"total_tokens": 303
}
}
],
"cost": 0.00074,
"usage": {
"input": 35,
"output": 150
}
}
]
}
```

Expand All @@ -241,7 +274,7 @@
)

response = client.chat.completions.create(
model="kimi-k2.6",
model="kimi-k3",
messages=[
{
"role": "system",
Expand Down
Loading