fix(integration-platform): classify code-based checks statically on manual run (CS-715)#3376
Merged
Merged
Conversation
…anual run (CS-715) The manual "run a check" endpoint decided whether a run was dynamic (held as an 'inconclusive' result for the self-heal agent, and hidden from the run history) versus static (shown as failed/pass) by checking only whether an active DynamicIntegration row existed for the provider slug. Several providers ship a code-based manifest AND an active DynamicIntegration row of the same slug for their extra DB-backed checks. A code manifest always wins in the registry, so the check that actually runs is the code one and must be classified statically. Keying `isDynamic` off the DB row alone marked those code-check runs 'inconclusive', so their results were withheld from the manual run while the scheduled and server-run paths (which guard on the code manifest) kept showing them correctly. Add a registry helper `isCodeManifest` and gate the manual-run `isDynamic` on it, mirroring the scheduled/server-run paths: a provider with a code manifest is never dynamic; only a provider with no code manifest is. Add unit tests covering the bundled manifests, the mixed code+dynamic providers, and the dynamic-only case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CRPjA4fQxdXuELyfTjYF8b
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
cubic analysis
No issues found across 5 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Linked issue analysis
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Statically classify manual runs for code-based integration checks (use isCodeManifest so code manifests are never treated as dynamic/inconclusive) | The manual-run path now imports and uses isCodeManifest to decide isDynamic, mirroring scheduled/server-run paths. |
| ✅ | Treat providers that ship both a code manifest and a DynamicIntegration row (github, vercel, aikido, rippling) as code manifests so manual runs show real failed/success results instead of being hidden | Tests explicitly assert these slugs are code manifests and registry prevents a dynamic registration from overriding a code manifest. |
| ✅ | Add regression tests and verify suite/typecheck to guard against recurrence | A new test file was added and the PR reports the integration-platform suite passing and api typecheck showing no errors for changed files. |
…code-based checks (CS-715) Extend the isCodeManifest guard from the manual run-check path to the self-heal agent's re-run (rerunAndPersistCheck). A code-based provider that also has a dynamic integration row of the same slug (github/vercel/aikido/rippling) was re-held as 'inconclusive' on re-run — and since the agent cannot patch a code check, it stayed hidden in a loop. It is now classified statically and its real result is shown. Dynamic integrations are unaffected (isCodeManifest is false for them, so the dynamic lookup runs exactly as before). Tests: - self-heal re-run reveals a code-based check as 'failed' (not re-held), even when a dynamic row shares the slug. - manual run-check treats a code-based provider as static under the same condition; adds the isCodeManifest mock the controller spec now requires. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CRPjA4fQxdXuELyfTjYF8b
claudfuen
pushed a commit
that referenced
this pull request
Jul 8, 2026
## [3.99.3](v3.99.2...v3.99.3) (2026-07-08) ### Bug Fixes * **integration-platform:** classify code-based checks statically on manual run (CS-715) ([#3376](#3376)) ([200b191](200b191))
Contributor
|
🎉 This PR is included in version 3.99.3 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a user clicks Run on a built-in (code-based) integration check, the manual run-check endpoint could stamp a legitimate result as
inconclusiveinstead offailed/success. Aninconclusiverun is intentionally hidden from the customer and handed to the self-heal agent — so real findings silently disappeared from the run history (no new run shown, no error), while the scheduled and auto-run paths kept showing the same checks correctly.Root cause
decideRunStatusholds any non-success result asinconclusivewhenisDynamicis true. The manual path (task-integrations.controller.ts) computedisDynamicas "does an activeDynamicIntegrationrow exist for this provider slug?" — without checking for a code manifest.Several providers ship both a code-based manifest and an active
DynamicIntegrationrow of the same slug (for extra DB-backed checks):github,vercel,aikido,rippling. A code manifest always wins in the registry, so the check that actually executes is the code one and must be classified statically. Keying off the DB row alone marked those code-check runsinconclusive.The scheduled path (
run-task-integration-checks.ts) and server-run path (run-connection-checks.ts) already guard on the code manifest (manifest ? false : …) — only the manual path was missing the guard (introduced inb374c89b6).Fix
isCodeManifest(slug)(exposes the registry's existingcodeManifestIdsset).isDynamicon it: a provider with a code manifest is never dynamic; only a provider with no code manifest is. This mirrors the scheduled/server-run paths.Tests
is-code-manifest.test.ts: bundled manifests →true; the mixed code+dynamic providers (github/vercel/aikido/rippling) →true; unknown slug →false; a dynamic-only manifest → resolvable butfalse; a dynamic registration cannot override a code manifest of the same id.@trycompai/integration-platformsuite: 435 pass.@trycompai/apitypecheck: no errors in any changed file (pre-existing unrelated spec-file failures remain onmain).Impact
Manual runs of code-based checks for
github/vercel/aikido/ripplingare shown as realfailed/successresults again. Truly dynamic integrations (no code manifest) are unaffected — they still hold asinconclusivefor the self-heal agent. No schema or API-contract change.🤖 Generated with Claude Code
https://claude.ai/code/session_01CRPjA4fQxdXuELyfTjYF8b
Summary by cubic
Statically classify code-based checks on manual runs and self-heal re-runs using a new
isCodeManifestguard. Fixes hidden inconclusive results sogithub,vercel,aikido, andripplingchecks show real success/fail again (CS-715 “manual Run starts then stops silently”).rerunAndPersistCheck) applies the same guard so code-based checks aren’t re-held as inconclusive.isCodeManifestfrom@trycompai/integration-platformand added tests for registry, manual-run controller, and self-heal re-run (bundled, mixed code+dynamic, and dynamic-only cases).Written for commit ec3646b. Summary will update on new commits.