docs(inspect): add Inspect AI adapter documentation - #96
Conversation
|
Warning Review limit reached
Next review available in: 42 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds a complete Inspect AI adapter documentation section covering architecture, configuration, benchmarks, deployment examples, and sidebar navigation. ChangesInspect AI adapter documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (1)
src/content/docs/adapters/inspect/index.mdx (1)
172-185: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winPin the container image in production-facing examples.
The Provider Details and
podmanexamples prescribequay.io/evalhub/community-inspect:latest, while the build instructions publish versioned images. A moving tag can silently change the adapter and prevents reproducible rollback. Use a release tag or digest for deployment examples, reservinglatestfor explicitly local/development usage.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/content/docs/adapters/inspect/index.mdx` around lines 172 - 185, Update the inspect adapter’s Provider Details and Container Image examples to reference a versioned release tag or immutable digest instead of quay.io/evalhub/community-inspect:latest. Keep latest only where explicitly identified for local or development usage, and align the deployment examples with the versioned images produced by the build instructions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/content/docs/adapters/inspect/benchmarks.md`:
- Line 18: Correct the benchmark-count statement in the documentation to match
the catalog: either add the five missing Petri category IDs so there are 40
named categories, or revise the wording to reflect 35 named category audits plus
inspect/petri-full, totaling 36 Petri benchmarks.
In `@src/content/docs/adapters/inspect/configuration.md`:
- Line 187: Update the realism_filter entry in the configuration table to
declare the accepted type as boolean | number and explain how a numeric value
acts as the realism-score threshold, preserving its experimental status and
default.
- Line 67: Clarify the guidance around the api_key and target_api_key parameter
documentation: state that plaintext API-key parameters are only for local or
development use and must not be included in persisted Kubernetes/API job
submissions. Recommend model.auth.secret_ref or environment-based credentials
for submitted jobs, while preserving the existing Secret reference instructions.
- Around line 114-115: Resolve the contradiction between the Bloom scenarios
note and the scenarios_base_url/scenarios_api_key documentation: verify the
adapter behavior and update the notes and parameter descriptions consistently.
If these parameters are mapped to global OPENAI_BASE_URL and OPENAI_API_KEY
before bloom scenarios runs, state that explicitly; otherwise remove or clarify
the per-role override claims so the documentation matches actual behavior in
both affected sections.
In `@src/content/docs/adapters/inspect/examples.mdx`:
- Around line 359-363: Update the inspect-petri-sycophancy-001 example command
to remove the --model-auth-secret maas-creds argument when targeting the
unauthenticated vLLM endpoint at http://vllm:8080/v1; keep the existing endpoint
and other arguments unchanged.
- Around line 150-152: Remove role API keys from both example JobSpecs: at
src/content/docs/adapters/inspect/examples.mdx lines 150-152, replace
auditor_api_key with the supported secret or environment-based configuration;
make the equivalent replacement for judge_anthropic_api_key at lines 221-224.
Keep the examples consistent with the credential guidance near lines 14-15.
---
Nitpick comments:
In `@src/content/docs/adapters/inspect/index.mdx`:
- Around line 172-185: Update the inspect adapter’s Provider Details and
Container Image examples to reference a versioned release tag or immutable
digest instead of quay.io/evalhub/community-inspect:latest. Keep latest only
where explicitly identified for local or development usage, and align the
deployment examples with the versioned images produced by the build
instructions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7408d44e-a3bf-4249-829c-d5e200c63465
📒 Files selected for processing (5)
astro.config.mjssrc/content/docs/adapters/inspect/benchmarks.mdsrc/content/docs/adapters/inspect/configuration.mdsrc/content/docs/adapters/inspect/examples.mdxsrc/content/docs/adapters/inspect/index.mdx
2 participants
What and why
Adds upstream documentation for the Inspect AI adapter so EvalHub users can discover and run Petri/Bloom alignment audits and inspect-evals benchmarks through the service.
Covers overview (architecture, quick start via SDK CLI / Python / REST), configuration (model roles, parameters, env vars), a benchmarks catalog (75 IDs across Petri, Bloom, and inspect-evals), and JobSpec examples. Registers the section in the Adapters sidebar.
Type
Testing
Summary by CodeRabbit