Add interception example environments - #2178
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d41b94349e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review 2 blocking correctness issues found. This PR introduces a complete request/response interception framework with significant new core library code across dialects, session handling, rollout, and harness components. As a substantial new feature introducing new capabilities and workflows, it warrants human review. Additionally, unresolved High and Medium severity findings regarding edge cases in the example interceptors require attention. You can customize Macroscope's approvability policy. Learn more. |
04864d3 to
3bd972d
Compare
295cec1 to
7aae47f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7aae47f960
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7aae47f to
b1f77a4
Compare
3bd972d to
257ddbc
Compare
257ddbc to
f9f354e
Compare
b1f77a4 to
3113425
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 311342505e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3113425 to
4b92fe4
Compare
e0271d7 to
ee897f4
Compare
8dad0d9 to
b12342b
Compare
fd9617b to
63c88b9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b12342b8be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b12342b to
ecb3b4b
Compare
63c88b9 to
b9cb943
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
2 similar comments
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 41741ae. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41741aec7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7dbd7b9b8b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04e3c3bf86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0bf72b38c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if image.has_transparency_data | ||
| else None | ||
| ) | ||
| grayscale_image = ImageOps.grayscale(image) |
There was a problem hiding this comment.
🟡 Medium grayscale_interception/taskset.py:40
When a multi-frame image (e.g., animated GIF, APNG, or animated WebP) is sent as a data URL, ImageOps.grayscale(image) converts only the first frame and image_data_url serializes that single static frame. All subsequent frames and animation timing are silently discarded, so the model receives a static first frame instead of a grayscale version of the supplied animated image. Consider checking getattr(image, "is_animated", False) and either preserving frame iteration/timing metadata through the grayscale conversion or documenting that only static images are supported.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/grayscale_interception/grayscale_interception/taskset.py around line 40:
When a multi-frame image (e.g., animated GIF, APNG, or animated WebP) is sent as a data URL, `ImageOps.grayscale(image)` converts only the first frame and `image_data_url` serializes that single static frame. All subsequent frames and animation timing are silently discarded, so the model receives a static first frame instead of a grayscale version of the supplied animated image. Consider checking `getattr(image, "is_animated", False)` and either preserving frame iteration/timing metadata through the grayscale conversion or documenting that only static images are supported.
| def grayscale(self, request: vf.Request) -> vf.Request | None: | ||
| # The last request message is the new user input, before the harness stores it. | ||
| message = request.messages[-1] |
There was a problem hiding this comment.
🟠 High grayscale_interception/taskset.py:12
grayscale accesses request.messages[-1] without first checking that messages is non-empty, so a request with an empty messages list raises IndexError instead of being returned unchanged. Add an if not request.messages: guard before the indexing, as is done in the analogous Bash interceptor.
| def grayscale(self, request: vf.Request) -> vf.Request | None: | |
| # The last request message is the new user input, before the harness stores it. | |
| message = request.messages[-1] | |
| def grayscale(self, request: vf.Request) -> vf.Request | None: | |
| if not request.messages: | |
| return None | |
| # The last request message is the new user input, before the harness stores it. |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/grayscale_interception/grayscale_interception/taskset.py around lines 12-14:
`grayscale` accesses `request.messages[-1]` without first checking that `messages` is non-empty, so a request with an empty `messages` list raises `IndexError` instead of being returned unchanged. Add an `if not request.messages:` guard before the indexing, as is done in the analogous Bash interceptor.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96f1468746
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Macroscope skipped reviewing this pull request. Per-PR cost limit exceeded (workspace setting). Reviews on this PR have cost $95.21 so far. This review would add an estimated $4.84, bringing the total to $100.04 — above your per-PR limit of $100.00. Tip To get this pull request reviewed, you can:
|
2 participants
Overview
Adds four small environments that show the typed interception API at the point where each hook runs.
Examples
interceptionshows a deterministic response rewrite, a judge-backed response rewrite, a response stop, and an observer hook that records a metric while returningNone.grayscale-interceptionrewrites embedded base64 user images to grayscale before the harness or model stores them. It preserves transparency and passes unsupported or malformed image data through unchanged.bash-interceptionshows two pre-execution outcomes: stop a proposed Bash command, or return a syntheticToolMessageso the command is skipped and the model continues.web-search-interceptioninspects provider-native Codex web-search citation URLs and stops when a returned source contains the configured word.The examples use
vf.Request,vf.Response, andvf.Tracedirectly, with short comments beside the relevant code.Stack
These PRs are stacked in merge order. Each PR targets the one above it.
@vf.interceptforvf.Requestandvf.Response@vf.stopforvf.Request,vf.Response, andvf.TraceNote
Low Risk
Documentation-style example packages and dependency/lockfile registration only; no changes to core verifiers runtime behavior.
Overview
Adds four installable v1 example environments that demonstrate typed
@vf.interceptand@vf.stopat different rollout boundaries, and wires them into the repo’s defaultexamplesuv group and lockfile.interceptioncovers response-side hooks: deterministic and judge-backed rewrites, a stop when the assistant saysSTOP, and an intercept that only records a metric.grayscale-interceptionrewrites base64 user images to grayscale on incomingvf.Request(with alpha preserved and bad data left alone).bash-interceptionshows pre-execution control—stop a proposedbashcall vs inject a syntheticToolMessageso the command never runs—with rewards checked via sentinel files.web-search-interceptiondefaults to Codex and stops when native web-search citation URLs contain a blocked domain.Each package is a small editable
environments/*project with README and taskset; rootpyproject.tomllists them inexamplesand[tool.uv.sources].Reviewed by Cursor Bugbot for commit 96f1468. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add interception example environments for request, response, image, bash, and web-search hooks
interception,grayscale-interception,bash-interception,web-search-interception) demonstrating@vf.interceptand@vf.stopdecorators across different rollout boundaries.RolloutSessiongains typed request/response interceptors and stop hooks;Rollout.openandRollout.steprun interceptors before model/harness calls and can rewrite or halt execution early.rewrite_request,rewrite_response, andstream_eventsmethods so rewritten content is reflected in native wire formats and SSE streams.BashHarnessand its program runner gain opt-in tool interception viaSUPPORTS_TOOL_INTERCEPTIONand--tool-interception-url, enabling before/after hooks on bash tool calls.InterceptRecordmodel is added toTraceso request and response rewrites are captured and queryable after rollout.Rollout.opennow returnsFalsewhen a stop fires during interception (previously only returnedFalsefor harness failures), which changes existing semantics for callers that infer session health from the return value.Macroscope summarized 96f1468.