Conversation
Integrate @inference/tracing to send OpenTelemetry spans to Inference.net's Catalyst dashboard for AI call observability (model usage, token counts, tool calls, latency). Instruments the GRC assistant chat agent with a labeled function ID, and auto-instruments all other AI SDK calls in the API process. Tracing is a no-op when CATALYST_OTLP_TOKEN is unset — zero impact on existing behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Snapshot a dynamic check's logic (definition + variables) before every edit so changes — manual, via the internal API, or by the future self-heal agent — are reversible and auditable. This is the safety net that makes auto-fixing dynamic checks safe: a bad change can be rolled back to the prior snapshot. - New DynamicCheckVersion table (additive; CREATE TABLE only, no change to existing tables or data). - PATCH /:id/checks/:checkId now records the pre-change snapshot first. Best-effort: a versioning failure NEVER blocks the actual update. - New GET /:id/checks/:checkId/versions (history) and POST /:id/checks/:checkId/restore/:versionId (rollback, which itself snapshots current state so it's reversible). - Only the check's logic is versioned; never credentials/secrets. Verified: prisma generate (schema valid) + @db rebuilt; API typecheck clean for all changed files; 7/7 unit tests pass (snapshot-before-update, ordering, best-effort-on-failure, source/note stripping, restore, list, 404s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
Pure function that classifies a dynamic-check failure as compliance / transient / customer_side / our_side. Core guarantee: ambiguous EXECUTION failures default to our_side (never blame the customer without positive proof), and a customer- looking error that is actually failing fleet-wide is treated as our_side. Additive — nothing calls it yet; it's the shared brain for the upcoming hold-layer (mark our-side/transient runs as inconclusive instead of failed) and the self-heal agent. 30/30 unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
Adds the building blocks for the hold-layer (additive — not wired into the live run path yet, so no behavior change): - New `inconclusive` IntegrationRunStatus enum value (ALTER TYPE ADD VALUE) for a run held because it failed for a non-customer reason (our bug / changed vendor endpoint / transient). - `splitFailuresByDisposition()` next to `decideTaskStatus()`: classifies each failing finding and partitions into `effective` (real compliance + proven customer-side → fail the task + show) vs `held` (our-side/transient → hold as inconclusive). Conservative — never blames the customer without proof; a fleet-wide failure is held even if it looks customer-like. Reuses the existing "indeterminate run doesn't fail the task" semantics. 36/36 unit tests (classifier 30 + partition 6). errorText is documented as pre-redacted by the caller — no secrets enter classification or storage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
… run path) Wires the self-heal hold into the scheduled run path. For DYNAMIC integrations, failing findings are classified and our-side/transient ones are HELD as inconclusive — they no longer fail the task or trigger a "task failed" email. Static/AWS behavior is byte-identical. - failingFindings now carry REDACTED error signals (failureSignalsFromEvidence + redactSecrets) so the classifier can run; no secrets/PII are stored or logged. - splitFailuresByDisposition gates the task-status decision (dynamic only). Held findings are excluded from BOTH the failure count and the finding count, so an all-held run is indeterminate (status unchanged) rather than failed or done. - Safe degradation: a finding with no readable error signal classifies as a real compliance failure -> exactly today's behavior. Non-dynamic providers untouched. Verified: typecheck clean for touched files; 116 tests pass across 10 suites (existing run-path + controller specs + new classifier/redact/signals/split). NOTE: end-to-end needs a staging run (the Trigger.dev pipeline isn't runnable locally). The manual run-check path still needs the same hold wired (follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…n path) Applies the same hold to the manual "Run check" path as the scheduled path: for DYNAMIC integrations, our-side/transient failures are held as inconclusive and do not fail the task (or flip it to done). Static/AWS behavior is unchanged. - ConnectionCheckOutcome now returns classifiable failures (resourceId + redacted error signals) instead of bare resourceIds. - The task-status decision splits held findings out (dynamic only), excluding them from both the failure count and the finding count. - 12/12 controller tests pass, incl. 2 new: a dynamic our-side failure is held (task NOT failed); a dynamic REAL finding still fails the task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
For dynamic integrations, a per-check run that failed for an our-side/transient reason (or threw) is now recorded with status 'inconclusive' instead of 'failed' — the self-heal agent's work queue + accurate run history. Static/AWS unchanged. Reuses the per-check failures already built for the task-status decision. Typecheck clean; 81 tests pass (no regressions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…self-heal agent Adds GET /v1/internal/integration-debug/inconclusive-runs — lists check runs held as inconclusive (our-side/transient failures the customer never saw as a red "failed"), filterable by provider/org, newest first, with the failing results + evidence so the agent can classify/diagnose without an extra round-trip. This is the agent's work queue (the agent itself lands in a separate comp-private PR). 10/10 internal-debug service tests pass (incl. the new queue test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…history The only read is listVersions(checkId) ordered by createdAt desc. One composite index serves that exactly and its leading checkId column covers the FK cascade, so it replaces both prior single-column indexes (the standalone createdAt index was never queried alone). Addresses cubic P2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…usive Runs are append-only and an inconclusive status is never cleared, so a fixed check's old held row lingered in the agent's queue forever. Filter the queue to runs that are the LATEST for their (connection, check) via a groupBy on max completedAt — resolved checks drop out instead of being re-attempted every poll. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
The hold marked our-side failures inconclusive (so the task wouldn't go red and no failure email fired), but the per-check run card still showed them: the /runs read path adjusted failedCount for exceptions only, so a held run reached the UI with failedCount>0 and its findings — rendering as a red failure. Exclude inconclusive runs in findLatestPerConnectionAndCheckByTask (the only caller, the customer /runs endpoint). Each account now shows its latest REAL run, or "not run yet" if a check has only ever been held — never a red for our own bug. The rows stay in the DB (the agent's queue + evidence) untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…t w/ scheduled) The manual run path (runCheckForConnection) stored every held dynamic run row as 'failed' and only adjusted task status — so a manual run that failed our-side still showed as failed in the task UI (the /runs hide keys on 'inconclusive') AND was invisible to the self-heal agent's queue. Mirror the scheduled path exactly: per account, a dynamic run that failed only for our-side/transient reasons is recorded as 'inconclusive'. isDynamic is now resolved once before the per-account loop and passed in. Widen CompleteCheckRunDto.status to include 'inconclusive'. Real compliance findings still record 'failed' (verified by test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…omers to green
After the agent applies a fix, the customer should see a fresh successful run
without waiting for the next scheduled run. Add POST /v1/internal/integration-
debug/connections/:id/rerun {checkId, taskId} that executes the SAVED check
in-process and PERSISTS a real run (unlike /run + /test which never persist),
deciding held-vs-shown via the same shared rule. A fixed check → 'success'
(customer sees green); one still failing our-side → re-held 'inconclusive'.
- Extract decideRunStatus() shared by all three run paths (scheduled, manual,
re-run) so run-status classification can never drift. Behavior-preserving for
the scheduled + manual paths (covered by existing tests + new unit tests).
- Add taskId to the inconclusive-runs queue so the agent re-runs the right task.
- Inject CheckRunRepository into the debug service (already a module provider).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
… P1)
The key/value redaction only matched unquoted keys, so `"api_key": "x"` /
`{"pwd":"x"}` leaked short secrets to the LLM/logs (long ones were caught by the
>=24-char catch-all, short ones weren't). Handle optional quotes around both key
and value, and add client_secret/access_token/refresh_token. Also de-escape the
no-useless-escape `\-` in the existing patterns so the file lints clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…y in review ## Problem A policy stuck in "Needs Review" status with grayed-out review date forces all team members to re-accept when republished, even though the policy content was never changed. This breaks the workflow for customers who have an auto-triggered review but no actual content updates. ## Root cause The policy transitions to "Needs Review" via time-based review triggers in policy-schedule.ts without a pending version. However, every exit path in policies.service.ts (updateById, publishVersion, acceptChanges, setActiveVersion, publishAll) unconditionally clears signedBy[] when moving back to published status. This wipes all existing acceptances even when there is no content change (no pendingVersionId). ## Fix Modify policies.service.ts to preserve existing signedBy records when: - Policy is transitioning from "Needs Review" back to "Published" - There is no pending version or content change This allows re-publishing unchanged content without resetting team acceptances. Adds an optional "mark reviewed" exit path for clarity. ## Explicitly NOT touched - UpdatePolicyOverview.tsx reviewDate grayed-out behavior (intentional, commit c966652) - Policy schedule auto-trigger logic - Core version/content change detection - Any other service areas ## Verification ✅ Policy with time-based review but no content change can republish without clearing acceptances ✅ signedBy preserved when transitioning needs_review -> published with no pendingVersionId ✅ Normal content updates still clear acceptances as expected ✅ Review date field remains grayed-out (existing behavior)
…ck 'done'
- failureSignalsFromEvidence (P1): the HTTP-status regex only handled `http_404`
and only searched evidence.error. Now searches BOTH error and message and
tolerates space/colon/_/- separators, so a 401/403 in a message
('HTTP 401 Unauthorized') is parsed and classified customer_side (shown) rather
than missed and defaulted to our_side (held). Won't false-match a URL.
- decideTaskStatus (P2): a held (our-side) failure now BLOCKS 'done' even when
other checks passed — an unresolved held check must not hide behind a green
task. heldCount is 0 for non-dynamic, so static/AWS/GCP/Azure are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
A quoted value with spaces ("password": "a b c") only had its first token
redacted; the tail leaked. Match the full double/single-quoted string (or an
unquoted token) so the whole value is redacted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
… decision - A held (inconclusive) run has no CONFIRMED failures — its findings are our-side/transient, not real fails — so failedCount is now 0 on all three persist paths (scheduled, manual, agent re-run). Defense-in-depth + self- consistent: no consumer can read a held run as a failure even if it doesn't filter by status (the /runs endpoint already excludes inconclusive). The raw findings still persist as results for the agent to diagnose. (cubic P1) - decideTaskStatus on a held outcome returns null (leave unchanged) BY DESIGN: a held check is our-side/transient and the agent re-runs after a fix, so the task reconciles within minutes. We do NOT pull a done task out of done for our bug (that would show the customer a regression they didn't cause, which the hold exists to prevent) — and there's no clean "held" task status to move it to. Added a test documenting this. (cubic P2 — intentional) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…d queue, throw→inconclusive
Four items cubic flagged across earlier commits, now addressed in current code:
- failureSignalsFromEvidence: an empty-string `message` masked the `error` text
(`msgStr ?? errStr` keeps '') — use the message only when non-empty.
- failureSignalsFromEvidence: parse a STRING `evidence.status` ('401',
'403 Forbidden'), not just a numeric one, so auth errors classify correctly.
- inconclusive-runs queue: bound the nested failing results (take 20) so a check
with thousands of findings can't dump an unbounded payload to the agent poller.
- scheduled throw path: a transport blip on a DYNAMIC provider is now recorded
'inconclusive' (held/hidden) instead of 'failed' — a network hiccup must not
show the customer a red. Static/AWS keep 'failed'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
… throw path
- evidence.status string parsing now anchors the code to the START of the string
(^\d{3}) so an unrelated 3-digit run ('build 200') isn't taken as a status.
- The scheduled throw (transport blip) path now routes through decideRunStatus
instead of hardcoding isDynamic?'inconclusive':'failed' — same result, no
duplicated rule. (cubic)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
feat(integration-platform): dynamic-integration self-heal layer
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
…-in-needs fix(policies): preserve acceptances when republishing unchanged policy in review
Email/password is not offered in any of our apps — sign-in is via magic link, email OTP, and OAuth. Disable the better-auth email/password endpoints since they aren't used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsGo9DPoaKf7npxMhY2tkz
…auth chore(auth): disable unused email/password login
…ng-review # Conflicts: # apps/api/src/assistant-chat/assistant-chat.controller.ts
The held-runs queue now carries each run's task title (e.g. "App availability") so the self-heal agent knows the check's INTENT and fixes toward what it should verify, not just toward "passes." Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PAsijjUQ1bMJuw8NuC1oR
…egration feat(api): add inference.net Catalyst tracing for LLM observability
[dev] [tofikwest] tofik/dynamic-check-versioning
Contributor
|
🎉 This PR is included in version 3.93.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated pull request to release the candidate branch into production, which will trigger a deployment.
It was created by the [Production PR] action.
Summary by cubic
Adds a self-heal hold for dynamic integrations and version history with rollback for dynamic checks so customers don’t see reds caused by our bugs and edits are safely reversible. Also preserves policy acceptances when republishing an unchanged policy from Needs Review and adds optional
@inference/tracingCatalyst tracing for AI calls; no-op if unset.New Features
inconclusive(hidden from task status and/runs,failedCount= 0). Applied to scheduled runs, manual runs, throw paths, and the new persist re-run; tasks don’t go “done” while any held.definitionandvariablesare versioned.source/noteare accepted as metadata and never written to the check.@inference/tracingto trace AI SDK calls and instrument the assistant chat with a function ID; controlled byCATALYST_OTLP_TOKEN/CATALYST_OTLP_ENDPOINT.signedByand advances the review date.Migration
DynamicCheckVersiontable and addedinconclusivetoIntegrationRunStatus. Run Prisma migrations and regenerate the client. Static providers are unchanged.Written for commit 496db80. Summary will update on new commits.