feat(core): add response-side structured output validation with error-feedback retry - #2858
Conversation
|
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…-feedback retry The request side of structured outputs was already supported (ResponseFormat.jsonSchema + JsonSchema + GenerateOptions.responseFormat), but the model's actual output was never validated: the javadoc claimed that strict mode validates the response against the schema, yet no validation implementation existed anywhere in the codebase. This adds the missing response-side piece, following the industry-standard remediation pattern used by Instructor, Guardrails re-ask and Spring AI 2.0 self-correcting structured output: - StructuredOutputValidator: validate parsed output against a JsonSchema, with compiled-schema caching (Networknt Draft 2020-12; the dependency was already declared in agentscope-core but unused) - ValidationError record (instance location + message) for actionable errors - StructuredOutputValidationException carrying schema name + error list - StructuredOutputGenerator.generateWithRetry: generate -> validate -> feed validation errors back into the next generation attempt instead of blindly retrying; markdown fence stripping and brace-matched JSON extraction - JsonSchema.validate(output): convenience shortcut 15 new unit tests; full agentscope-core suite passes (2314 tests).
3d30848 to
93bec48
Compare
| * @throws StructuredOutputValidationException when all attempts fail | ||
| */ | ||
| public static JsonNode generateWithRetry( | ||
| Function<List<StructuredOutputValidator.ValidationError>, String> generate, |
There was a problem hiding this comment.
[P1] 新增的校验重试入口没有接入现有结构化输出调用链。
当前 ReActAgent.call(..., Class/JsonNode) 的 native 路径仍直接调用 wrapNativeStructuredResult()，只解析 JSON，没有调用这里的 validator/generator。实测 required-number schema 收到 {"answer":"wrong"} 时，标准 agent.call() 仍然成功。因此这个 PR 目前没有为现有 AgentScope 结构化输出 API 增加 response-side validation 或 retry。建议将校验和纠错重试集成到 ReActAgent 的 Reactor 调用链，并保留 usage、thinking、hooks、session、取消及状态回滚语义。
| text = text.substring(0, closingFence).trim(); | ||
| } | ||
| } | ||
| int start = text.indexOf('{'); |
There was a problem hiding this comment.
P1] 解析失败时不能把原始文本合成为业务对象。
当前找不到 JSON 或 JSON 解析失败时会返回 {"result": raw}。如果目标 schema 合法要求 result:string，纯文本或畸形 JSON 就会直接校验成功。实测输入 this is not JSON 会成功返回 {"result":"this is not JSON"}，绕过重试和“模型实际输出必须是 JSON”的保证。解析失败应生成独立的 parse error 并进入 retry/exception，不能构造可能符合用户 schema 的字段。另外，第一个 {...} 解析失败后也不应直接放弃后面可能存在的合法 JSON。
| * @param schema the schema to validate against | ||
| * @return the list of validation errors; empty when the output conforms | ||
| */ | ||
| public static List<ValidationError> validate(JsonNode output, JsonSchema schema) { |
There was a problem hiding this comment.
[P2] null output 或缺失 schema 当前会 fail-open。
这里返回空列表，但该 API 的契约将空列表解释为“output conforms”。实测 schema.validate(null) 返回 []；JsonSchema.builder().name("x").build() 传给 generator 后，任意输出都会在第一次尝试成功。缺失 schema 应抛出配置错误；null output 应返回明确的 validation error 或抛出解析异常，不能与校验通过使用相同结果。
| */ | ||
| public final class StructuredOutputValidator { | ||
|
|
||
| private static final SchemaRegistry REGISTRY = |
There was a problem hiding this comment.
[P2] 当前 Networknt 配置不会执行 format assertions。
使用当前 2.0.0 依赖实测，strict schema 声明 format: email 时，{"email":"not an email"} 仍返回零错误。如果本地校验承诺 strict/schema-conforming，建议通过 SchemaRegistryConfig 启用 formatAssertionsEnabled(true)，并补充 email、date-time 等回归测试；如果有意只把 format 作为 annotation，则需要在 API 文档中明确说明限制。
|
关于这个 PR 在 native 路径上补响应侧校验的必要性, 持保留意见,想先对齐一下设计前提: 能力契约已经建立了信任边界。 Model.supportsNativeStructuredOutput() 返回 true,意味着 provider 侧通过 response_format / strict mode 在生成阶段(constrained decoding)就强制输出符合 schema。schema 在解码时就被约束,框架侧再做一次 JSON Schema 校验是重复劳动——只增加延迟和成本,几乎拦截不到任何真实错误。我们不应该在框架层默认"不信任 provider 的能力声明"。 fallback 路径已经有完整的"校验 + 错误反馈重试"闭环。 模型不支持原生结构化输出时,ReActAgent 注入 generate_response 合成工具,参数 schema 内嵌用户 schema;工具执行前会做运行时校验,失败时把带字段路径的可操作错误信息(每行 path: message)作为 tool result 回灌给模型自我修正(受 maxIters 约束)。这正是本 PR 所描述的 error-feedback retry 模式,已存在于主调用链中,所以"response side 完全没有校验"的空白比 PR 描述的要小。 如果某个 provider 声称支持但实际不严格遵守, 那是该 Model 实现的 supportsNativeStructuredOutput() 声明不诚实,正确修复是把它的能力开关关掉、让它降级走 fallback 路径(那里已有校验),而不是给所有 native 调用方加一层防御性校验。 基于 1,我认为把校验接入 native 主路径没有必要。 对独立工具类 StructuredOutputGenerator,从框架角度我也持保留态度:它本质上只是一个"生成 → 校验 → 把错误拼回 prompt 重试"的薄循环,用户基于 networknt 几十行即可自行实现;框架在 agent 层已经提供了完整的结构化输出能力(两条路径均含校验保障)。为一个自拼 prompt 的小众场景在 core 中增加 API 面积,收益不足、维护成本却在。除非能给出框架内的具体集成点或真实用户需求,否则建议不引入。 另外我在核对代码时发现一个与本 PR 无关的性能问题: |
Wire response-side validation into the actual structured-output call
chain via an opt-in middleware, and address the maintainer review
findings:
- StructuredOutputValidationMiddleware hooks onModelCall and activates
only for calls carrying a json_schema response format (the native
structured-output path) and only when explicitly attached: the
framework keeps trusting supportsNativeStructuredOutput() capability
declarations. Failed attempts are never surfaced as conforming
content; one structured_output.failed_attempt CustomEvent is emitted
per failed attempt (disable via emitAttemptEvents(false)), and only
the final conforming attempt's events are released. Because it wraps
the model call at the event-stream level, usage, thinking, hooks,
session, cancellation and state-rollback semantics are preserved.
- StructuredOutputRetryPolicy: maxAttempts, cumulative tokenBudget,
onFailedAttempt listener, emitAttemptEvents toggle; per-call override
via GenerateOptions.builder().structuredOutputPolicy(...).
- Fail-closed extraction (review P1): parse failures throw
StructuredOutputParseException and enter the retry loop instead of
synthesizing a {"result": raw} payload; a malformed balanced
candidate no longer aborts extraction — later candidates are scanned.
- Fail-closed validation (review P2): a missing schema throws a
configuration error and null output yields an explicit validation
error instead of silently passing.
- Format assertions (review P2): formatAssertionsEnabled(true) in the
Networknt registry so format: email / date-time are actually checked.
- StructuredOutputValidationException now carries the parse error and
every failed attempt chronologically (FailedAttempt record).
- Correction-prompt wording aligned with the existing fallback path
(English "path: message" lines, same shape as ToolValidator).
29 structured-output tests green; full agentscope-core suite: 2328 tests.
…hub.com/helloworldtang/agentscope-java into feat/structured-output-validation-retry
|
感谢两位的详细评审,逐条回应如下。对应实现已完成并推送(29 个 structured-output 测试 + agentscope-core 全量 2328 通过,spotless 干净)。
你的设计前提我完全认同,这正是本次改法调整的依据:
|
|
看一下框架中已有的 generate_response 原理，以及在后面版本合入的 #2697 增强处理。 且：这些类怎么会全部塞到 core 模块的 formatter 包中呢？从功能上来看是 middleware。 |
|
发现新的 P1：失败轮输出进入 middleware 校验前，已经被 ReasoningContext 累计并触发 hooks；后续仅回写 text，没有回滚 thinking/tool 状态。隔离集成测试可稳定复现：最终成功结果会混入失败轮 thinking，usage 也没有正确累计。 |
|
感谢评审，4 条意见已全部处理并 force-push（commit 0d4d57a）：
|
|
感谢指出，这个 P1 成立：失败轮的 thinking/tool 状态已写入 ReasoningContext 并触发过 hooks，middleware 重试只替换了 text，状态污染会导致最终结果混入失败轮 thinking、usage 累计也不对。我正在处理——方案方向是让重试路径具备状态回滚能力（回滚 ReasoningContext 到校验前快照，或把校验循环下沉到 reason 状态提交点之前），确认方案后会更新 PR。隔离复现如果方便贴一下，我可以对照验证修复。 |
|
可以，我这边有隔离复现，基于当前 head 69de1a6。
|
Move response-side validation from the onModelCall middleware into the ReActAgent reasoning loop. The middleware approach could not roll back per-turn ReasoningContext state: by the time the middleware saw the stream, thinking/tool-call accumulation and reasoning hooks had already fired, so a retried turn leaked failed-attempt thinking into the final message and lost its usage. Now the validation wraps reasoning per iteration: - a non-conforming attempt is discarded before its reasoning context is committed, so failed-turn thinking/tool state never reaches the final message; - token usage from failed attempts is carried forward and aggregated into the final message so retried tokens remain accounted for; - an error-feedback correction turn is appended to the conversation and reasoning restarts with a fresh context; exhausting maxAttempts fails with StructuredOutputValidationException; - failed_attempt CustomEvent is emitted per rejected attempt. The StructuredOutputValidationMiddleware is removed accordingly. Adds regression tests: failed-attempt thinking isolation, usage aggregation across attempts, and error-feedback retry. Full agentscope-core suite: 2321 tests, 0 failures.
|
感谢提供隔离复现，已按该场景修复并推送（commit faa0559）。 根因：middleware 工作在 model call 层，收到流时 ReasoningContext 已完成 thinking/tool 状态累计并触发过 reasoning hooks，架构上无法回滚——所以把校验循环下沉到了 ReActAgent reasoning 层（ 修复行为：
已按你的复现场景新增回归测试 |
4 participants
Summary
Structured outputs currently have only the request side:
ResponseFormat.jsonSchema+JsonSchema(+GenerateOptions.responseFormat) send the schema constraint to the model, but the model's actual response is never validated — the javadoc onJsonSchema.strictclaims the response "will be validated against the schema", yet no validation implementation exists anywhere in the codebase (json-schema-validatoris declared inagentscope-core/pom.xmlbut unused).This PR adds the missing response side, following the industry-standard remediation pattern used by Instructor (error context fed back to the LLM), Guardrails re-ask, and Spring AI 2.0 self-correcting structured output:
StructuredOutputValidator— validates a parsed output against aJsonSchema, with compiled-schema caching (Networknt Draft 2020-12)ValidationErrorrecord (instance location + message) for actionable, machine-readable errorsStructuredOutputValidationException— carries schema name + error list after retries are exhaustedStructuredOutputGenerator.generateWithRetry(generate, schema, maxRetries)— generate → validate → feed validation errors back into the next generation attempt (instead of blindly retrying); includes markdown-fence stripping and brace-matched JSON extraction so loosely-formatted model output still gets validatedJsonSchema.validate(output)— convenience shortcutWhy error-feedback retry
Blind retry just re-rolls the dice; feeding validation errors back into the prompt materially raises correction rates and reduces token waste. This is now a cross-framework standard mechanism.
Design notes
json-schema-validatorTesting
agentscope-coresuite passes: 2314 testsmvn spotless:applyclean