diff --git a/.changeset/etl-pipeline-layer-retired.md b/.changeset/etl-pipeline-layer-retired.md new file mode 100644 index 0000000000..d7b7127ce1 --- /dev/null +++ b/.changeset/etl-pipeline-layer-retired.md @@ -0,0 +1,51 @@ +--- +"@objectstack/spec": major +--- + +refactor(spec)!: retire the L2 ETL layer — `automation/etl.zod.ts` had no executor, and the sync architecture doc was recommending it (#6414) + +`ETLPipeline`, `ETLPipelineRun`, `ETLSource`, `ETLDestination`, `ETLTransformation`, +the `ETLEndpointType` / `ETLTransformationType` / `ETLSyncMode` / `ETLRunStatus` +enums and the `ETL` factory are REMOVED under ADR-0049 enforce-or-remove. The whole +file goes, on the same reading #4738 used to retire L1 `DataSyncConfig` one layer up: +**narrative-only**. No engine ever parsed, scheduled or executed an `ETLPipeline`. + +Measured on `origin/main` immediately before the removal: the only non-spec +references in this repo are two fumadocs-generated documentation sources +(`apps/docs/.source/*.ts`), not executors; objectui has no reference at all; and +there is no `packages/spec/liveness/etl.json`, so no ADR-0049 gate ever had a reading +on the surface — while the same file family's EXECUTED half does have one +(`liveness/mapping.json`), which is what makes that absence meaningful rather than an +oversight. + +FROM → TO, layer by layer — with one gap stated plainly instead of redirected: + +| removed | use instead | +|---|---| +| `ETLPipeline.source` + `syncMode` + `schedule` (scheduled extraction from an external system) | `ConnectorSchema.syncConfig` (`integration/connector.zod.ts`) — the live, parsed sync surface: strategy, direction, cron schedule, `conflictResolution`, batching, delete mode | +| `ETLTransformation` of type `map` / `cast`-like per-field work | `mapping.fieldMapping[].transform` (`data/mapping.zod.ts`) — `none`/`constant`/`map`/`split`/`join`/`lookup`, applied row by row by the REST import path | +| `ETLPipeline.schedule` alone | `system/job.zod.ts` | +| `ETLTransformation` of type `join` / `aggregate` / `script` / `merge` / `deduplicate` / … | **nothing.** There is no replacement because there was never an implementation — those ten transformation types named capabilities no runtime had. Do the work where it runs (the destination warehouse's ELT, a `flow`, a scheduled job), and let multi-stage movement return through ADR-0049's ENFORCE route: the engine first, the vocabulary second | + +**The fix:** delete the import. Nothing was ever deployed under an `ETLPipeline` — +that is the finding, not a consolation — so there is no data migration; `tsc` reports +TS2724/TS2305 at every import of a retired name. + +**`packages/spec/docs/SYNC_ARCHITECTURE.md` is rewritten in the same change**, and +that is not incidental. It named `ETLPipeline` as the recommended destination for +authors displaced by the L1 retirement and tabulated ten transformation types with +copyable examples down to `script | Custom JavaScript/Python`. Retiring the schema +while the doc still recommended it would have been self-contradictory, and +forwarding L1's authors to a second layer with no executor was the defect compounding +rather than closing. + +**Absorbed:** the #4962 `etl-retry-converged-onto-retry-policy` entry (`retry.maxAttempts` +→ `maxRetries`, default 3 → 0) — both land in the unreleased protocol 17, so composed, +a rename on a shape that does not survive the major has no observable effect, and its +`retiredKey()` tombstone goes with the shape that carried it. + +The retirement kit — route 3: no tombstone, no D2 conversion. +`RETIRED_DEFS_BY_MAJOR[17]` (9 defs) plus the D3 `SemanticMigration` +`etl-pipeline-layer-retired` are the declaration. + + diff --git a/.changeset/http-server-runtime-vocabulary-retired.md b/.changeset/http-server-runtime-vocabulary-retired.md new file mode 100644 index 0000000000..6f1e0854cf --- /dev/null +++ b/.changeset/http-server-runtime-vocabulary-retired.md @@ -0,0 +1,52 @@ +--- +"@objectstack/spec": major +--- + +refactor(spec)!: retire `system/http-server.zod.ts`'s runtime vocabulary — the event, capability and status shapes nothing ever emitted (#5295) + +`ServerEventType`, `ServerEventSchema` / `ServerEvent`, `ServerCapabilitiesSchema` / +`ServerCapabilities` / `ServerCapabilitiesParsed` and `ServerStatusSchema` / +`ServerStatus` are REMOVED under ADR-0049 enforce-or-remove. This is the second and +final pass over the file: #4938 removed its CONFIG half (`HttpServerConfigSchema`, +nine keys, zero readers, zero authoring entry), and this removes the RUNTIME half — +a 7-member lifecycle event union, an eight-boolean capability report and a +five-state status record with connection and request counters. Nothing ever emitted, +consumed or parsed any of them. + +FROM → TO: + +| removed | what actually decides it | +|---|---| +| `ServerEventType` / `ServerEvent(Schema)` | nothing emits a server event feed. Lifecycle is the transport plugin's own start/stop seam; observability is `system/metrics.zod.ts` + `system/logging.zod.ts`, and `OS_SERVER_TIMING` for timings | +| `ServerCapabilities(Schema/Parsed)` | a transport plugin declares what it provides by implementing the kernel plugin contract — the seams it registers ARE the capability statement | +| `ServerStatus(Schema)` | `/health` for liveness, the metrics surface for counters | + +**The fix:** delete the import. There is no replacement key, because there was +never a key — none of the four was authorable on any shape. Server-level +configuration that IS authorable is untouched: `defineStack({ server: { trustProxy, +security } })` / `StackServerConfigSchema` (#5006) parses exactly as it did in 16.x, +as does the route-registration half of the same module (`RouteHandlerMetadata`, +`MiddlewareType`, `MiddlewareConfig`). + +**Why now, and what unblocked it.** The card was held rather than queued on a real +doubt: a response/capability vocabulary can legitimately be a REFERENCE surface for +host implementers, so "zero consumers in this repo" is weaker evidence for one of +those than for an authorable key. It was lifted by measuring the reference reader +itself — `plugin-hono-server`, the one in-tree host implementation, neither +implements nor reports any of the three: it names no capability record, no status +shape and no event union, and what it registers is routes and middleware. The +control passed in the same sweep (`MiddlewareConfig`, twelve lines away, resolves to +`packages/runtime/src/middleware.ts`). + +The retirement kit — route 3 of the retirement playbook, as #4938 was in this same +file: **no `retiredKey()` tombstone and no D2 conversion**, because a prescription +nobody can receive is noise and there is no authored document to rewrite. +`RETIRED_DEFS_BY_MAJOR[17]` (4 defs) plus the D3 `SemanticMigration` +`http-server-runtime-vocabulary-retired` are the declaration; the generated +baselines (`json-schema.manifest/system.json`, `authorable-surface/system.json`, +`api-surface/system.json`) lose their entries in the same change, deliberately. + +If host-implementer conformance becomes a real requirement it returns through the +ENFORCE route: an adapter contract with a checker behind it, vocabulary second. + + diff --git a/.changeset/view-management-protocol-retired.md b/.changeset/view-management-protocol-retired.md new file mode 100644 index 0000000000..1b36a3cf15 --- /dev/null +++ b/.changeset/view-management-protocol-retired.md @@ -0,0 +1,55 @@ +--- +"@objectstack/spec": major +"@objectstack/client": major +--- + +refactor(spec,client)!: retire `ViewProtocol`'s five viewId-addressed methods and their ten schemas (#6239) + +`listViews`, `getView`, `createView`, `updateView` and `deleteView` — the +`ViewProtocol` interface and `ListViews`/`GetView`/`CreateView`/`UpdateView`/`DeleteView` +Request+Response schemas in `api/protocol.zod.ts` — are REMOVED under ADR-0049 +enforce-or-remove (maintainer ruling 2026-08-07). `@objectstack/client` drops the +five response types it re-exported. + +Measured on `origin/main` immediately before the removal, the surface had none of +the three things a protocol method needs: + +- **no implementation** — `packages/metadata-protocol/src/protocol.ts` declares no + `listViews`/`getView`/`createView`/`updateView`/`deleteView`; its only view + resolver is `getUiView`; +- **no route** — `packages/rest/src/rest-server.ts` never mentions `viewId`, so + nothing viewId-addressed was reachable over HTTP at all; +- **no caller** — the only `ViewProtocol` mention outside its own file was + `content/docs/kernel/services-checklist.mdx`, which already recorded the five as + declared-and-unrouted. + +FROM → TO — both replacements are surfaces that were always the live ones: + +| removed | use instead | +|---|---| +| `listViews` / `getView` / `createView` / `updateView` / `deleteView` (+ their 10 schemas) | the generic metadata methods with `type: 'view'` — `getMetaItem` / `getMetaItems` / `saveMetaItem` / `deleteMetaItem`, served at `/api/v1/meta/view/:name` | +| `GetViewResponse` as "the shape of the resolved view" | `GetUiViewResponse` — `getUiView`, served at `GET /api/v1/ui/view/:object/:type` | + +**The fix:** delete the import and address views by NAME through the metadata API +(`view` is a metadata type), or by object+type through `getUiView`. Nothing +addressed a view by `viewId` before this change either; that is the finding. + +**Why a removal rather than a note.** The declared surface is name-identical and +semantics-adjacent to a real one, which makes it an attractive nuisance in every +grep — and it has already mis-directed a decision: **#5948's issue body AND its +2026-08-07 maintainer ruling both read `GetViewResponseSchema` (zero +implementations) as the contract of `GET /ui/view/:object/:type`**, whose declared +response is `GetUiViewResponseSchema`, 250 lines up and one word different. That +ruling's reasoning happened to survive the mix-up; this removal stops relying on +that luck. + +The retirement kit — route 3: **no tombstone and no D2 conversion** (none of the ten +was a key on an authorable shape, and nothing parsed them, so there is no source or +`sys_metadata` row to rewrite). `RETIRED_DEFS_BY_MAJOR[17]` (10 defs) plus the D3 +`SemanticMigration` `view-management-protocol-retired` are the declaration; the +generated baselines and reference docs lose their entries in the same change. + +If "read and write ONE view by id" becomes a real requirement, it returns +implementation-first. + + diff --git a/content/docs/getting-started/quick-reference.mdx b/content/docs/getting-started/quick-reference.mdx index 0fae0f2fac..dc62d1caf6 100644 --- a/content/docs/getting-started/quick-reference.mdx +++ b/content/docs/getting-started/quick-reference.mdx @@ -145,7 +145,7 @@ REST/GraphQL endpoints, real-time subscriptions, and discovery. | **[Metadata](/docs/references/api/metadata)** | `metadata.zod.ts` | Metadata | API metadata endpoints | | **[Storage](/docs/references/api/storage)** | `storage.zod.ts` | Storage | API storage operations | -## Automation Protocol (5 schemas) +## Automation Protocol (4 schemas) Flows, state machines, approvals, and integrations. @@ -155,7 +155,6 @@ Flows, state machines, approvals, and integrations. | **[Approval](/docs/references/automation/approval)** | `approval.zod.ts` | ApprovalNodeConfig | Flow approval-node config | | **[State Machine](/docs/references/automation/state-machine)** | `state-machine.zod.ts` | StateMachine | State machine definitions | | **[Webhook](/docs/references/automation/webhook)** | `webhook.zod.ts` | Webhook | Outbound webhooks | -| **[ETL](/docs/references/automation/etl)** | `etl.zod.ts` | ETLPipeline | Data transformation pipelines | ## Security Protocol (3 schemas) diff --git a/content/docs/kernel/services-checklist.mdx b/content/docs/kernel/services-checklist.mdx index 9a8e906815..b2f5ff391b 100644 --- a/content/docs/kernel/services-checklist.mdx +++ b/content/docs/kernel/services-checklist.mdx @@ -320,22 +320,34 @@ registered under their own names (`security.permissions`, `security.rls`, ## 5–6. Business Services -### 5. ui Service — 5 declared methods, none routed ❌ -`listViews`, `getView`, `createView`, `updateView`, `deleteView` +### 5. ui Service — 1 routed method ✅ (was 5 declared, none routed) +`getUiView` -These five are **optional members of `ViewProtocol` that nothing implements and no -route reaches** — `view` is a metadata type, so view CRUD goes through the metadata -API (`/api/v1/meta`), not through them. Nothing anywhere registers the `ui` slot -either (#4093 / #4146), so `CORE_SERVICE_PROVIDER.ui` names -`@objectstack/metadata-protocol` rather than a `ui` plugin: the one route the `/ui` -domain serves is `GET /api/v1/ui/view/:object[/:type]`, which calls `getUiView` on -the **`protocol`** service that `assembleMetadataProtocol()` registers (invoked by +Nothing anywhere registers the `ui` slot (#4093 / #4146), so `CORE_SERVICE_PROVIDER.ui` +names `@objectstack/metadata-protocol` rather than a `ui` plugin: the one route the +`/ui` domain serves is `GET /api/v1/ui/view/:object[/:type]`, which calls `getUiView` +on the **`protocol`** service that `assembleMetadataProtocol()` registers (invoked by `ObjectQLPlugin`, or by the standalone `createMetadataProtocolPlugin()`). Without it the domain answers **501** with that remedy spelled out, not a generic "install a ui plugin". +#### Retired in v17: `ViewProtocol`'s five methods +`listViews`, `getView`, `createView`, `updateView`, `deleteView` — and their ten +Request/Response schemas — were removed in +[#6239](https://github.com/objectstack-ai/objectstack/issues/6239) under ADR-0049 +enforce-or-remove. This checklist had recorded them as declared-and-unrouted since it +was written; the removal makes that reading permanent instead of re-derivable. Views +are read and written through the surfaces that always served them: the metadata API +(`/api/v1/meta/view/:name`, `view` being a metadata type) for the stored definition, +and `getUiView` above for the resolved render-time view. + +The concrete cost of leaving it declared is on the record: **#5948's issue body and its +2026-08-07 maintainer ruling both read `GetViewResponseSchema` — this retired block, +zero implementations — as the contract of `GET /ui/view/:object/:type`**, whose declared +response is `GetUiViewResponseSchema`. One word apart, and identical to a grep. + ### Retired in v17: the `workflow` slot The slot, its `IWorkflowService` contract and the three `WorkflowProtocol` methods (`getWorkflowConfig`, `getWorkflowState`, `workflowTransition`) were @@ -509,7 +521,7 @@ a package that cannot be installed is a dead end, which is why | Slot | State | |:-------|:------------| -| **ui** | Nothing registers the slot. `ViewProtocol`'s five methods are declared and unrouted; view CRUD runs through `/api/v1/meta`, and `/api/v1/ui/view/:object` is served by the `protocol` service. | +| **ui** | Nothing registers the slot. `ViewProtocol`'s five declared-and-unrouted methods were **retired in v17** (#6239); view CRUD runs through `/api/v1/meta`, and `/api/v1/ui/view/:object` is served by the `protocol` service. | | **search** | Nothing ships. Contract and engine enum exist in `@objectstack/spec` only. | | **ai** | Nothing in this repo — `service-ai` (chat, completion, models, conversations) is Cloud/EE. | | **realtime transport** | The service exists but no WebSocket/SSE route is mounted, so `routes.realtime` is deliberately never advertised. | diff --git a/content/docs/references/api/protocol.mdx b/content/docs/references/api/protocol.mdx index 8cbc817c82..4aaae09b9f 100644 --- a/content/docs/references/api/protocol.mdx +++ b/content/docs/references/api/protocol.mdx @@ -12,8 +12,8 @@ description: Protocol protocol schemas ## TypeScript Usage ```typescript -import { AiAgentCapabilitiesSchema, AiAgentChatRequestSchema, AiAgentSummarySchema, AiAgentsResponseSchema, AiChatRequestSchema, AiChatResponseSchema, AiCompleteRequestSchema, AiConversationSchema, AiMessageSchema, AiModelsResponseSchema, AiPendingActionSchema, AiPendingActionStatusSchema, AiStreamChunkSchema, ApproveAiPendingActionResponseSchema, AutomationActionsResponseSchema, AutomationTriggerRequestSchema, AutomationTriggerResponseSchema, BatchDataRequestSchema, BatchDataResponseSchema, CheckPermissionRequestSchema, CheckPermissionResponseSchema, CreateAiConversationRequestSchema, CreateDataRequestSchema, CreateDataResponseSchema, CreateManyDataRequestSchema, CreateManyDataResponseSchema, CreateViewRequestSchema, CreateViewResponseSchema, DeleteDataRequestSchema, DeleteDataResponseSchema, DeleteManyDataRequestSchema, DeleteManyDataResponseSchema, DeleteMetaItemRequestSchema, DeleteMetaItemResponseSchema, DeleteViewRequestSchema, DeleteViewResponseSchema, DisablePackageRequestSchema, DisablePackageResponseSchema, EnablePackageRequestSchema, EnablePackageResponseSchema, FindDataRequestSchema, FindDataResponseSchema, GetDataRequestSchema, GetDataResponseSchema, GetDiscoveryRequestSchema, GetDiscoveryResponseSchema, GetEffectivePermissionsRequestSchema, GetEffectivePermissionsResponseSchema, GetFieldLabelsRequestSchema, GetFieldLabelsResponseSchema, GetLocalesRequestSchema, GetLocalesResponseSchema, GetMetaItemCachedRequestSchema, GetMetaItemCachedResponseSchema, GetMetaItemRequestSchema, GetMetaItemResponseSchema, GetMetaItemsRequestSchema, GetMetaItemsResponseSchema, GetMetaTypesRequestSchema, GetMetaTypesResponseSchema, GetNotificationPreferencesRequestSchema, GetNotificationPreferencesResponseSchema, GetObjectPermissionsRequestSchema, GetObjectPermissionsResponseSchema, GetPackageRequestSchema, GetPackageResponseSchema, GetPresenceRequestSchema, GetPresenceResponseSchema, GetTranslationsRequestSchema, GetTranslationsResponseSchema, GetUiViewRequestSchema, GetUiViewResponseSchema, GetViewRequestSchema, GetViewResponseSchema, HttpFindQueryParamsSchema, InstallPackageRequestSchema, InstallPackageResponseSchema, ListAiConversationsRequestSchema, ListAiConversationsResponseSchema, ListAiPendingActionsRequestSchema, ListAiPendingActionsResponseSchema, ListNotificationsRequestSchema, ListNotificationsResponseSchema, ListPackagesRequestSchema, ListPackagesResponseSchema, ListViewsRequestSchema, ListViewsResponseSchema, MarkAllNotificationsReadRequestSchema, MarkAllNotificationsReadResponseSchema, MarkNotificationsReadRequestSchema, MarkNotificationsReadResponseSchema, NotificationSchema, NotificationPreferencesSchema, RealtimeConnectRequestSchema, RealtimeConnectResponseSchema, RealtimeDisconnectRequestSchema, RealtimeDisconnectResponseSchema, RealtimeSubscribeRequestSchema, RealtimeSubscribeResponseSchema, RealtimeUnsubscribeRequestSchema, RealtimeUnsubscribeResponseSchema, RegisterDeviceRequestSchema, RegisterDeviceResponseSchema, RejectAiPendingActionResponseSchema, SaveMetaItemRequestSchema, SaveMetaItemResponseSchema, SetPresenceRequestSchema, SetPresenceResponseSchema, UninstallPackageRequestSchema, UninstallPackageResponseSchema, UnregisterDeviceRequestSchema, UnregisterDeviceResponseSchema, UpdateAiConversationRequestSchema, UpdateDataRequestSchema, UpdateDataResponseSchema, UpdateManyDataRequestSchema, UpdateManyDataResponseSchema, UpdateNotificationPreferencesRequestSchema, UpdateNotificationPreferencesResponseSchema, UpdateViewRequestSchema, UpdateViewResponseSchema, ValidateDataIssueSchema, ValidateDataRequestSchema, ValidateDataResponseSchema } from '@objectstack/spec/api'; -import type { AiAgentCapabilities, AiAgentChatRequest, AiAgentSummary, AiAgentsResponse, AiChatRequest, AiChatResponse, AiCompleteRequest, AiConversation, AiMessage, AiModelsResponse, AiPendingAction, AiPendingActionStatus, AiStreamChunk, ApproveAiPendingActionResponse, AutomationActionsResponse, AutomationTriggerRequest, AutomationTriggerResponse, BatchDataRequest, BatchDataResponse, CheckPermissionRequest, CheckPermissionResponse, CreateAiConversationRequest, CreateDataRequest, CreateDataResponse, CreateManyDataRequest, CreateManyDataResponse, CreateViewRequest, CreateViewResponse, DeleteDataRequest, DeleteDataResponse, DeleteManyDataRequest, DeleteManyDataResponse, DeleteMetaItemRequest, DeleteMetaItemResponse, DeleteViewRequest, DeleteViewResponse, DisablePackageRequest, DisablePackageResponse, EnablePackageRequest, EnablePackageResponse, FindDataRequest, FindDataResponse, GetDataRequest, GetDataResponse, GetDiscoveryRequest, GetDiscoveryResponse, GetEffectivePermissionsRequest, GetEffectivePermissionsResponse, GetFieldLabelsRequest, GetFieldLabelsResponse, GetLocalesRequest, GetLocalesResponse, GetMetaItemCachedRequest, GetMetaItemCachedResponse, GetMetaItemRequest, GetMetaItemResponse, GetMetaItemsRequest, GetMetaItemsResponse, GetMetaTypesRequest, GetMetaTypesResponse, GetNotificationPreferencesRequest, GetNotificationPreferencesResponse, GetObjectPermissionsRequest, GetObjectPermissionsResponse, GetPackageRequest, GetPackageResponse, GetPresenceRequest, GetPresenceResponse, GetTranslationsRequest, GetTranslationsResponse, GetUiViewRequest, GetUiViewResponse, GetViewRequest, GetViewResponse, InstallPackageRequest, InstallPackageResponse, ListAiConversationsRequest, ListAiConversationsResponse, ListAiPendingActionsRequest, ListAiPendingActionsResponse, ListNotificationsRequest, ListNotificationsResponse, ListPackagesRequest, ListPackagesResponse, ListViewsRequest, ListViewsResponse, MarkAllNotificationsReadRequest, MarkAllNotificationsReadResponse, MarkNotificationsReadRequest, MarkNotificationsReadResponse, Notification, NotificationPreferences, RealtimeConnectRequest, RealtimeConnectResponse, RealtimeDisconnectRequest, RealtimeDisconnectResponse, RealtimeSubscribeRequest, RealtimeSubscribeResponse, RealtimeUnsubscribeRequest, RealtimeUnsubscribeResponse, RegisterDeviceRequest, RegisterDeviceResponse, RejectAiPendingActionResponse, SaveMetaItemRequest, SaveMetaItemResponse, SetPresenceRequest, SetPresenceResponse, UninstallPackageRequest, UninstallPackageResponse, UnregisterDeviceRequest, UnregisterDeviceResponse, UpdateAiConversationRequest, UpdateDataRequest, UpdateDataResponse, UpdateManyDataRequest, UpdateManyDataResponse, UpdateNotificationPreferencesRequest, UpdateNotificationPreferencesResponse, UpdateViewRequest, UpdateViewResponse, ValidateDataIssue, ValidateDataRequest, ValidateDataResponse } from '@objectstack/spec/api'; +import { AiAgentCapabilitiesSchema, AiAgentChatRequestSchema, AiAgentSummarySchema, AiAgentsResponseSchema, AiChatRequestSchema, AiChatResponseSchema, AiCompleteRequestSchema, AiConversationSchema, AiMessageSchema, AiModelsResponseSchema, AiPendingActionSchema, AiPendingActionStatusSchema, AiStreamChunkSchema, ApproveAiPendingActionResponseSchema, AutomationActionsResponseSchema, AutomationTriggerRequestSchema, AutomationTriggerResponseSchema, BatchDataRequestSchema, BatchDataResponseSchema, CheckPermissionRequestSchema, CheckPermissionResponseSchema, CreateAiConversationRequestSchema, CreateDataRequestSchema, CreateDataResponseSchema, CreateManyDataRequestSchema, CreateManyDataResponseSchema, DeleteDataRequestSchema, DeleteDataResponseSchema, DeleteManyDataRequestSchema, DeleteManyDataResponseSchema, DeleteMetaItemRequestSchema, DeleteMetaItemResponseSchema, DisablePackageRequestSchema, DisablePackageResponseSchema, EnablePackageRequestSchema, EnablePackageResponseSchema, FindDataRequestSchema, FindDataResponseSchema, GetDataRequestSchema, GetDataResponseSchema, GetDiscoveryRequestSchema, GetDiscoveryResponseSchema, GetEffectivePermissionsRequestSchema, GetEffectivePermissionsResponseSchema, GetFieldLabelsRequestSchema, GetFieldLabelsResponseSchema, GetLocalesRequestSchema, GetLocalesResponseSchema, GetMetaItemCachedRequestSchema, GetMetaItemCachedResponseSchema, GetMetaItemRequestSchema, GetMetaItemResponseSchema, GetMetaItemsRequestSchema, GetMetaItemsResponseSchema, GetMetaTypesRequestSchema, GetMetaTypesResponseSchema, GetNotificationPreferencesRequestSchema, GetNotificationPreferencesResponseSchema, GetObjectPermissionsRequestSchema, GetObjectPermissionsResponseSchema, GetPackageRequestSchema, GetPackageResponseSchema, GetPresenceRequestSchema, GetPresenceResponseSchema, GetTranslationsRequestSchema, GetTranslationsResponseSchema, GetUiViewRequestSchema, GetUiViewResponseSchema, HttpFindQueryParamsSchema, InstallPackageRequestSchema, InstallPackageResponseSchema, ListAiConversationsRequestSchema, ListAiConversationsResponseSchema, ListAiPendingActionsRequestSchema, ListAiPendingActionsResponseSchema, ListNotificationsRequestSchema, ListNotificationsResponseSchema, ListPackagesRequestSchema, ListPackagesResponseSchema, MarkAllNotificationsReadRequestSchema, MarkAllNotificationsReadResponseSchema, MarkNotificationsReadRequestSchema, MarkNotificationsReadResponseSchema, NotificationSchema, NotificationPreferencesSchema, RealtimeConnectRequestSchema, RealtimeConnectResponseSchema, RealtimeDisconnectRequestSchema, RealtimeDisconnectResponseSchema, RealtimeSubscribeRequestSchema, RealtimeSubscribeResponseSchema, RealtimeUnsubscribeRequestSchema, RealtimeUnsubscribeResponseSchema, RegisterDeviceRequestSchema, RegisterDeviceResponseSchema, RejectAiPendingActionResponseSchema, SaveMetaItemRequestSchema, SaveMetaItemResponseSchema, SetPresenceRequestSchema, SetPresenceResponseSchema, UninstallPackageRequestSchema, UninstallPackageResponseSchema, UnregisterDeviceRequestSchema, UnregisterDeviceResponseSchema, UpdateAiConversationRequestSchema, UpdateDataRequestSchema, UpdateDataResponseSchema, UpdateManyDataRequestSchema, UpdateManyDataResponseSchema, UpdateNotificationPreferencesRequestSchema, UpdateNotificationPreferencesResponseSchema, ValidateDataIssueSchema, ValidateDataRequestSchema, ValidateDataResponseSchema } from '@objectstack/spec/api'; +import type { AiAgentCapabilities, AiAgentChatRequest, AiAgentSummary, AiAgentsResponse, AiChatRequest, AiChatResponse, AiCompleteRequest, AiConversation, AiMessage, AiModelsResponse, AiPendingAction, AiPendingActionStatus, AiStreamChunk, ApproveAiPendingActionResponse, AutomationActionsResponse, AutomationTriggerRequest, AutomationTriggerResponse, BatchDataRequest, BatchDataResponse, CheckPermissionRequest, CheckPermissionResponse, CreateAiConversationRequest, CreateDataRequest, CreateDataResponse, CreateManyDataRequest, CreateManyDataResponse, DeleteDataRequest, DeleteDataResponse, DeleteManyDataRequest, DeleteManyDataResponse, DeleteMetaItemRequest, DeleteMetaItemResponse, DisablePackageRequest, DisablePackageResponse, EnablePackageRequest, EnablePackageResponse, FindDataRequest, FindDataResponse, GetDataRequest, GetDataResponse, GetDiscoveryRequest, GetDiscoveryResponse, GetEffectivePermissionsRequest, GetEffectivePermissionsResponse, GetFieldLabelsRequest, GetFieldLabelsResponse, GetLocalesRequest, GetLocalesResponse, GetMetaItemCachedRequest, GetMetaItemCachedResponse, GetMetaItemRequest, GetMetaItemResponse, GetMetaItemsRequest, GetMetaItemsResponse, GetMetaTypesRequest, GetMetaTypesResponse, GetNotificationPreferencesRequest, GetNotificationPreferencesResponse, GetObjectPermissionsRequest, GetObjectPermissionsResponse, GetPackageRequest, GetPackageResponse, GetPresenceRequest, GetPresenceResponse, GetTranslationsRequest, GetTranslationsResponse, GetUiViewRequest, GetUiViewResponse, InstallPackageRequest, InstallPackageResponse, ListAiConversationsRequest, ListAiConversationsResponse, ListAiPendingActionsRequest, ListAiPendingActionsResponse, ListNotificationsRequest, ListNotificationsResponse, ListPackagesRequest, ListPackagesResponse, MarkAllNotificationsReadRequest, MarkAllNotificationsReadResponse, MarkNotificationsReadRequest, MarkNotificationsReadResponse, Notification, NotificationPreferences, RealtimeConnectRequest, RealtimeConnectResponse, RealtimeDisconnectRequest, RealtimeDisconnectResponse, RealtimeSubscribeRequest, RealtimeSubscribeResponse, RealtimeUnsubscribeRequest, RealtimeUnsubscribeResponse, RegisterDeviceRequest, RegisterDeviceResponse, RejectAiPendingActionResponse, SaveMetaItemRequest, SaveMetaItemResponse, SetPresenceRequest, SetPresenceResponse, UninstallPackageRequest, UninstallPackageResponse, UnregisterDeviceRequest, UnregisterDeviceResponse, UpdateAiConversationRequest, UpdateDataRequest, UpdateDataResponse, UpdateManyDataRequest, UpdateManyDataResponse, UpdateNotificationPreferencesRequest, UpdateNotificationPreferencesResponse, ValidateDataIssue, ValidateDataRequest, ValidateDataResponse } from '@objectstack/spec/api'; // Validate data const result = AiAgentCapabilitiesSchema.parse(data); @@ -380,31 +380,6 @@ const result = AiAgentCapabilitiesSchema.parse(data); | **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when'> }[]` | optional | Write-observability (#3407/#3431/#3455): caller-supplied `readonly` fields the #3043 create-ingress strip removed before the rows were written. AGGREGATED across the batch (one event per object/reason with the union of dropped field names) rather than per-row, because the insert-time strip is static-`readonly` only — schema-uniform, so every row drops the same set. Present ONLY when ≥1 field was dropped; the creates still succeeded without them (count/success unchanged). Optional — omit-when-empty keeps the shape backward-compatible. (The per-row `insertMany`/`batch` paths carry per-row `droppedFields` on each result instead — see BatchOperationResultSchema.) | ---- - -## CreateViewRequest - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name (snake_case) | -| **data** | `{ name?: string; label?: string; object?: string; list?: object; … }` | ✅ | View definition to create | - - ---- - -## CreateViewResponse - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name | -| **viewId** | `string` | ✅ | Created view identifier | -| **view** | `{ name?: string; label?: string; object?: string; list?: object; … }` | ✅ | Created view definition | - - --- ## DeleteDataRequest @@ -487,31 +462,6 @@ const result = AiAgentCapabilitiesSchema.parse(data); | **message** | `string` | optional | | ---- - -## DeleteViewRequest - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name (snake_case) | -| **viewId** | `string` | ✅ | View identifier to delete | - - ---- - -## DeleteViewResponse - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name | -| **viewId** | `string` | ✅ | Deleted view identifier | -| **success** | `boolean` | ✅ | Whether deletion succeeded | - - --- ## DisablePackageRequest @@ -975,30 +925,6 @@ Get package response | **_lockDocsUrl** | `string` | optional | Optional documentation link surfaced next to _lockReason. | ---- - -## GetViewRequest - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name (snake_case) | -| **viewId** | `string` | ✅ | View identifier | - - ---- - -## GetViewResponse - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name | -| **view** | `{ name?: string; label?: string; object?: string; list?: object; … }` | ✅ | View definition | - - --- ## HttpFindQueryParams @@ -1156,30 +1082,6 @@ List packages response | **total** | `number` | ✅ | Total package count | ---- - -## ListViewsRequest - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name (snake_case) | -| **type** | `Enum<'list' \| 'form'>` | optional | Filter by view type | - - ---- - -## ListViewsResponse - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name | -| **views** | `{ name?: string; label?: string; object?: string; list?: object; … }[]` | ✅ | Array of view definitions | - - --- ## MarkAllNotificationsReadRequest @@ -1586,32 +1488,6 @@ Uninstall package response | **preferences** | `{ email: boolean; push: boolean; inApp: boolean; digest: Enum<'none' \| 'daily' \| 'weekly'>; … }` | ✅ | Updated notification preferences | ---- - -## UpdateViewRequest - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name (snake_case) | -| **viewId** | `string` | ✅ | View identifier | -| **data** | `{ name?: string; label?: string; object?: string; list?: object; … }` | ✅ | Partial view data to update | - - ---- - -## UpdateViewResponse - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **object** | `string` | ✅ | Object name | -| **viewId** | `string` | ✅ | Updated view identifier | -| **view** | `{ name?: string; label?: string; object?: string; list?: object; … }` | ✅ | Updated view definition | - - --- ## ValidateDataIssue diff --git a/content/docs/references/automation/etl.mdx b/content/docs/references/automation/etl.mdx deleted file mode 100644 index 0275a7ee94..0000000000 --- a/content/docs/references/automation/etl.mdx +++ /dev/null @@ -1,246 +0,0 @@ ---- -title: Etl -description: Etl protocol schemas ---- - -{/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */} - -ETL (Extract, Transform, Load) Pipeline Protocol - LEVEL 2: Data Engineering - -Inspired by modern data integration platforms like Airbyte, Fivetran, and Apache NiFi. - -**Positioning in the sync/integration layering** (L1 "Simple Sync" was -retired in #4738 — narrative-only, zero consumers; see -`packages/spec/docs/SYNC_ARCHITECTURE.md`): -- **ETL Pipeline** (THIS FILE) - Data engineers - Aggregate 10 sources to warehouse -- **Enterprise Connector** ([integration/connector.zod.ts](/docs/references/integration/connector)) - System integrators - Full SAP integration; connector-attached sync via `syncConfig` - -ETL pipelines enable automated data synchronization between systems, transforming -data as it moves from source to destination. - -**SCOPE: Advanced multi-source, multi-stage transformations.** -Supports complex operations: joins, aggregations, filtering, custom SQL. - -## When to Use This Layer - -**Use ETL Pipeline when:** -- Combining data from multiple sources -- Need aggregations, joins, transformations -- Building data warehouses or analytics platforms -- Complex data transformations required - -**Examples:** -- Sales data from Salesforce + Marketing from HubSpot → Data Warehouse -- Multi-region databases → Consolidated reporting -- Legacy system migration with transformation - -**When to upgrade:** -- Need full connector lifecycle (auth, webhooks, rate limits) → Use [Enterprise Connector](/docs/references/integration/connector) - -See also: [../integration/connector.zod.ts](/docs/references/integration/connector) for the Enterprise Connector layer - -## Use Cases - -1. **Data Warehouse Population** - - Extract from multiple operational systems - - Transform to analytical schema - - Load into data warehouse - -2. **System Integration** - - Sync data between CRM and Marketing Automation - - Keep product catalog synchronized across e-commerce platforms - - Replicate data for backup/disaster recovery - -3. **Data Migration** - - Move data from legacy systems to modern platforms - - Consolidate data from multiple sources - - Split monolithic databases into microservices - -See also: https://airbyte.com/ - -See also: https://docs.fivetran.com/ - -See also: https://nifi.apache.org/ - -@example -```typescript -const salesforceToDB: ETLPipeline = { - name: 'salesforce_to_postgres', - label: 'Salesforce Accounts to PostgreSQL', - source: { - type: 'api', - connector: 'salesforce', - config: { object: 'Account' } - }, - destination: { - type: 'database', - connector: 'postgres', - config: { table: 'accounts' } - }, - transformations: [ - { type: 'map', config: { 'Name': 'account_name' } } - ], - schedule: '0 2 * * *' // Daily at 2 AM -} -``` - - -**Source:** `packages/spec/src/automation/etl.zod.ts` - - -## TypeScript Usage - -```typescript -import { ETLDestinationSchema, ETLEndpointTypeSchema, ETLPipelineSchema, ETLPipelineRunSchema, ETLRunStatusSchema, ETLSourceSchema, ETLSyncModeSchema, ETLTransformationSchema, ETLTransformationTypeSchema } from '@objectstack/spec/automation'; -import type { ETLDestination, ETLEndpointType, ETLPipeline, ETLPipelineRun, ETLRunStatus, ETLSource, ETLSyncMode, ETLTransformation, ETLTransformationType } from '@objectstack/spec/automation'; - -// Validate data -const result = ETLDestinationSchema.parse(data); -``` - ---- - -## ETLDestination - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **type** | `Enum<'database' \| 'api' \| 'file' \| 'stream' \| 'object' \| 'warehouse' \| 'storage' \| 'spreadsheet'>` | ✅ | Destination type | -| **connector** | `string` | optional | Connector ID | -| **config** | `Record` | ✅ | Destination configuration | -| **writeMode** | `Enum<'append' \| 'overwrite' \| 'upsert' \| 'merge'>` | ✅ | How to write data | -| **primaryKey** | `string[]` | optional | Primary key fields | - - ---- - -## ETLEndpointType - -### Allowed Values - -* `database` -* `api` -* `file` -* `stream` -* `object` -* `warehouse` -* `storage` -* `spreadsheet` - - ---- - -## ETLPipeline - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **name** | `string` | ✅ | Pipeline identifier (snake_case) | -| **label** | `string` | optional | Pipeline display name | -| **description** | `string` | optional | Pipeline description | -| **source** | `{ type: Enum<'database' \| 'api' \| 'file' \| 'stream' \| 'object' \| 'warehouse' \| 'storage' \| 'spreadsheet'>; connector?: string; config: Record; incremental?: object }` | ✅ | Data source | -| **destination** | `{ type: Enum<'database' \| 'api' \| 'file' \| 'stream' \| 'object' \| 'warehouse' \| 'storage' \| 'spreadsheet'>; connector?: string; config: Record; writeMode?: Enum<'append' \| 'overwrite' \| 'upsert' \| 'merge'>; … }` | ✅ | Data destination | -| **transformations** | `{ name?: string; type: Enum<'map' \| 'filter' \| 'aggregate' \| 'join' \| 'script' \| 'lookup' \| 'split' \| … +3 more>; config: Record; continueOnError?: boolean }[]` | optional | Transformation pipeline | -| **syncMode** | `Enum<'full' \| 'incremental' \| 'cdc'>` | optional | Sync mode | -| **schedule** | `string \| { dialect: Enum<'cel' \| 'cron' \| 'template'>; source?: string; ast?: any; meta?: object }` | optional | Cron schedule expression | -| **enabled** | `boolean` | optional | Pipeline enabled status | -| **retry** | `{ maxRetries?: integer; backoffMs?: integer; backoffMultiplier?: number; maxRetryDelayMs?: integer; … }` | optional | Retry configuration | -| **notifications** | `{ onSuccess?: string[]; onFailure?: string[] }` | optional | Notification settings | -| **tags** | `string[]` | optional | Pipeline tags | -| **metadata** | `Record` | optional | Custom metadata | - - ---- - -## ETLPipelineRun - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **id** | `string` | ✅ | Run identifier | -| **pipelineName** | `string` | ✅ | Pipeline name | -| **status** | `Enum<'pending' \| 'running' \| 'succeeded' \| 'failed' \| 'cancelled' \| 'timeout'>` | ✅ | Run status | -| **startedAt** | `string` | ✅ | Start time | -| **completedAt** | `string` | optional | Completion time | -| **durationMs** | `number` | optional | Duration in ms | -| **stats** | `{ recordsRead: integer; recordsWritten: integer; recordsErrored: integer; bytesProcessed: integer }` | optional | Run statistics | -| **error** | `{ message: string; code?: string; details?: any }` | optional | Error information | -| **logs** | `string[]` | optional | Execution logs | - - ---- - -## ETLRunStatus - -### Allowed Values - -* `pending` -* `running` -* `succeeded` -* `failed` -* `cancelled` -* `timeout` - - ---- - -## ETLSource - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **type** | `Enum<'database' \| 'api' \| 'file' \| 'stream' \| 'object' \| 'warehouse' \| 'storage' \| 'spreadsheet'>` | ✅ | Source type | -| **connector** | `string` | optional | Connector ID | -| **config** | `Record` | ✅ | Source configuration | -| **incremental** | `{ enabled: boolean; cursorField: string; cursorValue?: any }` | optional | Incremental extraction config | - - ---- - -## ETLSyncMode - -### Allowed Values - -* `full` -* `incremental` -* `cdc` - - ---- - -## ETLTransformation - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **name** | `string` | optional | Transformation name | -| **type** | `Enum<'map' \| 'filter' \| 'aggregate' \| 'join' \| 'script' \| 'lookup' \| 'split' \| 'merge' \| 'normalize' \| 'deduplicate'>` | ✅ | Transformation type | -| **config** | `Record` | ✅ | Transformation config | -| **continueOnError** | `boolean` | ✅ | Continue on error | - - ---- - -## ETLTransformationType - -### Allowed Values - -* `map` -* `filter` -* `aggregate` -* `join` -* `script` -* `lookup` -* `split` -* `merge` -* `normalize` -* `deduplicate` - - ---- - diff --git a/content/docs/references/automation/index.mdx b/content/docs/references/automation/index.mdx index 626af5d65d..2b9cac9582 100644 --- a/content/docs/references/automation/index.mdx +++ b/content/docs/references/automation/index.mdx @@ -10,7 +10,6 @@ This section contains all protocol schemas for the automation layer of ObjectSta - diff --git a/content/docs/references/automation/meta.json b/content/docs/references/automation/meta.json index 2ee557f996..c9990e2740 100644 --- a/content/docs/references/automation/meta.json +++ b/content/docs/references/automation/meta.json @@ -10,7 +10,6 @@ "time-relative-trigger", "---Integration & Data---", "bpmn-interop", - "etl", "webhook", "---Approvals & Jobs---", "approval", diff --git a/content/docs/references/index.mdx b/content/docs/references/index.mdx index f7f074d784..7ba22b6797 100644 --- a/content/docs/references/index.mdx +++ b/content/docs/references/index.mdx @@ -1,6 +1,6 @@ --- title: Protocol Reference -description: Every schema published by @objectstack/spec — 1605 schemas across 14 protocol modules +description: Every schema published by @objectstack/spec — 1582 schemas across 14 protocol modules --- {/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */} @@ -20,8 +20,8 @@ counts are sums of the rows they head. Regenerate with | Module | Pages | Schemas | Description | | :--- | ---: | ---: | :--- | | [AI Protocol](/docs/references/ai) | 11 | 66 | Agents, tools, skills, RAG and knowledge sources, model registry, conversations. | -| [API Protocol](/docs/references/api) | 28 | 419 | REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. | -| [Automation Protocol](/docs/references/automation) | 14 | 77 | Flows and their nodes, approvals, ETL pipelines, webhooks, state machines, execution records. | +| [API Protocol](/docs/references/api) | 28 | 409 | REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. | +| [Automation Protocol](/docs/references/automation) | 13 | 68 | Flows and their nodes, approvals, ETL pipelines, webhooks, state machines, execution records. | | [Cloud Protocol](/docs/references/cloud) | 11 | 94 | Environments, packages and versions, marketplace, developer portal, tenancy. | | [Data Protocol](/docs/references/data) | 29 | 164 | Objects, fields, queries, filters, datasources and drivers — the ObjectQL layer. | | [Identity Protocol](/docs/references/identity) | 5 | 28 | Users and accounts, organizations, positions, API keys, SCIM provisioning. | @@ -31,9 +31,9 @@ counts are sums of the rows they head. Regenerate with | [Security Protocol](/docs/references/security) | 5 | 27 | Permission sets, row-level security, sharing rules, tenancy posture. | | [Shared Protocol](/docs/references/shared) | 8 | 31 | Primitives used across every protocol — identifiers, HTTP, expressions, error maps, enums. | | [Studio Protocol](/docs/references/studio) | 3 | 35 | Studio designer metadata — the authoring surfaces for the protocols above. | -| [System Protocol](/docs/references/system) | 37 | 296 | The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. | +| [System Protocol](/docs/references/system) | 37 | 292 | The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. | | [UI Protocol](/docs/references/ui) | 16 | 146 | Apps, pages, views, dashboards, reports, actions and themes — the ObjectUI layer. | -| **Total** | **200** | **1605** | 14 protocol modules | +| **Total** | **199** | **1582** | 14 protocol modules | --- @@ -61,7 +61,7 @@ Agents, tools, skills, RAG and knowledge sources, model registry, conversations. ## API Protocol -**Source:** `packages/spec/src/api/` · **Import:** `@objectstack/spec/api` · **28 pages, 419 schemas** +**Source:** `packages/spec/src/api/` · **Import:** `@objectstack/spec/api` · **28 pages, 409 schemas** REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. @@ -86,7 +86,7 @@ REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. | [`odata.zod.ts`](/docs/references/api/odata) | `ODataConfig`, `ODataError`, `ODataFilterFunction`, `ODataMetadata`, `ODataQuery`, `ODataResponse` | | [`package-api.zod.ts`](/docs/references/api/package-api) | `GetInstalledPackageRequest`, `GetInstalledPackageResponse`, `ListInstalledPackagesRequest`, `ListInstalledPackagesResponse`, `PackageApiErrorCode`, `PackageInstallRequest`, `PackageInstallResponse`, `PackagePathParams`, `PackageRollbackRequest`, `PackageRollbackResponse`, `PackageUpgradeRequest`, `PackageUpgradeResponse`, `ResolveDependenciesRequest`, `ResolveDependenciesResponse`, `UninstallPackageApiRequest`, `UninstallPackageApiResponse`, `UploadArtifactRequest`, `UploadArtifactResponse` | | [`plugin-rest-api.zod.ts`](/docs/references/api/plugin-rest-api) | `ErrorHandlingConfig`, `HandlerStatus`, `OpenApiGenerationConfig`, `RequestValidationConfig`, `ResponseEnvelopeConfig`, `RestApiEndpoint`, `RestApiPluginConfig`, `RestApiRouteCategory`, `RestApiRouteRegistration`, `RouteCoverageEntry`, `RouteCoverageReport`, `ValidationMode` | -| [`protocol.zod.ts`](/docs/references/api/protocol) | `AiAgentCapabilities`, `AiAgentChatRequest`, `AiAgentSummary`, `AiAgentsResponse`, `AiChatRequest`, `AiChatResponse`, `AiCompleteRequest`, `AiConversation`, `AiMessage`, `AiModelsResponse`, `AiPendingAction`, `AiPendingActionStatus`, `AiStreamChunk`, `ApproveAiPendingActionResponse`, `AutomationActionsResponse`, `AutomationTriggerRequest`, `AutomationTriggerResponse`, `BatchDataRequest`, `BatchDataResponse`, `CheckPermissionRequest`, `CheckPermissionResponse`, `CreateAiConversationRequest`, `CreateDataRequest`, `CreateDataResponse`, `CreateManyDataRequest`, `CreateManyDataResponse`, `CreateViewRequest`, `CreateViewResponse`, `DeleteDataRequest`, `DeleteDataResponse`, `DeleteManyDataRequest`, `DeleteManyDataResponse`, `DeleteMetaItemRequest`, `DeleteMetaItemResponse`, `DeleteViewRequest`, `DeleteViewResponse`, `DisablePackageRequest`, `DisablePackageResponse`, `EnablePackageRequest`, `EnablePackageResponse`, `FindDataRequest`, `FindDataResponse`, `GetDataRequest`, `GetDataResponse`, `GetDiscoveryRequest`, `GetDiscoveryResponse`, `GetEffectivePermissionsRequest`, `GetEffectivePermissionsResponse`, `GetFieldLabelsRequest`, `GetFieldLabelsResponse`, `GetLocalesRequest`, `GetLocalesResponse`, `GetMetaItemCachedRequest`, `GetMetaItemCachedResponse`, `GetMetaItemRequest`, `GetMetaItemResponse`, `GetMetaItemsRequest`, `GetMetaItemsResponse`, `GetMetaTypesRequest`, `GetMetaTypesResponse`, `GetNotificationPreferencesRequest`, `GetNotificationPreferencesResponse`, `GetObjectPermissionsRequest`, `GetObjectPermissionsResponse`, `GetPackageRequest`, `GetPackageResponse`, `GetPresenceRequest`, `GetPresenceResponse`, `GetTranslationsRequest`, `GetTranslationsResponse`, `GetUiViewRequest`, `GetUiViewResponse`, `GetViewRequest`, `GetViewResponse`, `HttpFindQueryParams`, `InstallPackageRequest`, `InstallPackageResponse`, `ListAiConversationsRequest`, `ListAiConversationsResponse`, `ListAiPendingActionsRequest`, `ListAiPendingActionsResponse`, `ListNotificationsRequest`, `ListNotificationsResponse`, `ListPackagesRequest`, `ListPackagesResponse`, `ListViewsRequest`, `ListViewsResponse`, `MarkAllNotificationsReadRequest`, `MarkAllNotificationsReadResponse`, `MarkNotificationsReadRequest`, `MarkNotificationsReadResponse`, `Notification`, `NotificationPreferences`, `RealtimeConnectRequest`, `RealtimeConnectResponse`, `RealtimeDisconnectRequest`, `RealtimeDisconnectResponse`, `RealtimeSubscribeRequest`, `RealtimeSubscribeResponse`, `RealtimeUnsubscribeRequest`, `RealtimeUnsubscribeResponse`, `RegisterDeviceRequest`, `RegisterDeviceResponse`, `RejectAiPendingActionResponse`, `SaveMetaItemRequest`, `SaveMetaItemResponse`, `SetPresenceRequest`, `SetPresenceResponse`, `UninstallPackageRequest`, `UninstallPackageResponse`, `UnregisterDeviceRequest`, `UnregisterDeviceResponse`, `UpdateAiConversationRequest`, `UpdateDataRequest`, `UpdateDataResponse`, `UpdateManyDataRequest`, `UpdateManyDataResponse`, `UpdateNotificationPreferencesRequest`, `UpdateNotificationPreferencesResponse`, `UpdateViewRequest`, `UpdateViewResponse`, `ValidateDataIssue`, `ValidateDataRequest`, `ValidateDataResponse` | +| [`protocol.zod.ts`](/docs/references/api/protocol) | `AiAgentCapabilities`, `AiAgentChatRequest`, `AiAgentSummary`, `AiAgentsResponse`, `AiChatRequest`, `AiChatResponse`, `AiCompleteRequest`, `AiConversation`, `AiMessage`, `AiModelsResponse`, `AiPendingAction`, `AiPendingActionStatus`, `AiStreamChunk`, `ApproveAiPendingActionResponse`, `AutomationActionsResponse`, `AutomationTriggerRequest`, `AutomationTriggerResponse`, `BatchDataRequest`, `BatchDataResponse`, `CheckPermissionRequest`, `CheckPermissionResponse`, `CreateAiConversationRequest`, `CreateDataRequest`, `CreateDataResponse`, `CreateManyDataRequest`, `CreateManyDataResponse`, `DeleteDataRequest`, `DeleteDataResponse`, `DeleteManyDataRequest`, `DeleteManyDataResponse`, `DeleteMetaItemRequest`, `DeleteMetaItemResponse`, `DisablePackageRequest`, `DisablePackageResponse`, `EnablePackageRequest`, `EnablePackageResponse`, `FindDataRequest`, `FindDataResponse`, `GetDataRequest`, `GetDataResponse`, `GetDiscoveryRequest`, `GetDiscoveryResponse`, `GetEffectivePermissionsRequest`, `GetEffectivePermissionsResponse`, `GetFieldLabelsRequest`, `GetFieldLabelsResponse`, `GetLocalesRequest`, `GetLocalesResponse`, `GetMetaItemCachedRequest`, `GetMetaItemCachedResponse`, `GetMetaItemRequest`, `GetMetaItemResponse`, `GetMetaItemsRequest`, `GetMetaItemsResponse`, `GetMetaTypesRequest`, `GetMetaTypesResponse`, `GetNotificationPreferencesRequest`, `GetNotificationPreferencesResponse`, `GetObjectPermissionsRequest`, `GetObjectPermissionsResponse`, `GetPackageRequest`, `GetPackageResponse`, `GetPresenceRequest`, `GetPresenceResponse`, `GetTranslationsRequest`, `GetTranslationsResponse`, `GetUiViewRequest`, `GetUiViewResponse`, `HttpFindQueryParams`, `InstallPackageRequest`, `InstallPackageResponse`, `ListAiConversationsRequest`, `ListAiConversationsResponse`, `ListAiPendingActionsRequest`, `ListAiPendingActionsResponse`, `ListNotificationsRequest`, `ListNotificationsResponse`, `ListPackagesRequest`, `ListPackagesResponse`, `MarkAllNotificationsReadRequest`, `MarkAllNotificationsReadResponse`, `MarkNotificationsReadRequest`, `MarkNotificationsReadResponse`, `Notification`, `NotificationPreferences`, `RealtimeConnectRequest`, `RealtimeConnectResponse`, `RealtimeDisconnectRequest`, `RealtimeDisconnectResponse`, `RealtimeSubscribeRequest`, `RealtimeSubscribeResponse`, `RealtimeUnsubscribeRequest`, `RealtimeUnsubscribeResponse`, `RegisterDeviceRequest`, `RegisterDeviceResponse`, `RejectAiPendingActionResponse`, `SaveMetaItemRequest`, `SaveMetaItemResponse`, `SetPresenceRequest`, `SetPresenceResponse`, `UninstallPackageRequest`, `UninstallPackageResponse`, `UnregisterDeviceRequest`, `UnregisterDeviceResponse`, `UpdateAiConversationRequest`, `UpdateDataRequest`, `UpdateDataResponse`, `UpdateManyDataRequest`, `UpdateManyDataResponse`, `UpdateNotificationPreferencesRequest`, `UpdateNotificationPreferencesResponse`, `ValidateDataIssue`, `ValidateDataRequest`, `ValidateDataResponse` | | [`query-adapter.zod.ts`](/docs/references/api/query-adapter) | `ODataQueryAdapter`, `OperatorMapping`, `QueryAdapterConfig`, `QueryAdapterTarget`, `RestQueryAdapter` | | [`realtime.zod.ts`](/docs/references/api/realtime) | `RealtimeConfig`, `RealtimeEvent`, `RealtimeEventType`, `RealtimePresence`, `Subscription`, `SubscriptionEvent`, `TransportProtocol` | | [`realtime-shared.zod.ts`](/docs/references/api/realtime-shared) | `BasePresence`, `PresenceStatus`, `RealtimeRecordAction` | @@ -100,7 +100,7 @@ REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. ## Automation Protocol -**Source:** `packages/spec/src/automation/` · **Import:** `@objectstack/spec/automation` · **14 pages, 77 schemas** +**Source:** `packages/spec/src/automation/` · **Import:** `@objectstack/spec/automation` · **13 pages, 68 schemas** Flows and their nodes, approvals, ETL pipelines, webhooks, state machines, execution records. @@ -110,7 +110,6 @@ Flows and their nodes, approvals, ETL pipelines, webhooks, state machines, execu | [`bpmn-interop.zod.ts`](/docs/references/automation/bpmn-interop) | `BpmnDiagnostic`, `BpmnElementMapping`, `BpmnExportOptions`, `BpmnImportOptions`, `BpmnInteropResult`, `BpmnUnmappedStrategy`, `BpmnVersion` | | [`builtin-node-config.zod.ts`](/docs/references/automation/builtin-node-config) | `CreateRecordConfig`, `DeleteRecordConfig`, `GetRecordConfig`, `MapConfig`, `ScreenConfig`, `ScreenFieldConfig`, `UpdateRecordConfig` | | [`control-flow.zod.ts`](/docs/references/automation/control-flow) | `FlowRegion`, `LoopConfig`, `ParallelBranch`, `ParallelConfig`, `RetryPolicy`, `TryCatchConfig` | -| [`etl.zod.ts`](/docs/references/automation/etl) | `ETLDestination`, `ETLEndpointType`, `ETLPipeline`, `ETLPipelineRun`, `ETLRunStatus`, `ETLSource`, `ETLSyncMode`, `ETLTransformation`, `ETLTransformationType` | | [`execution.zod.ts`](/docs/references/automation/execution) | `Checkpoint`, `ConcurrencyPolicy`, `ExecutionError`, `ExecutionErrorSeverity`, `ExecutionLog`, `ExecutionStatus`, `ExecutionStepLog`, `ExecutionStepMetrics`, `ExecutionStepSkipReason`, `FlowRunGateSummary`, `FlowRunNodeSummary`, `FlowRunSummary`, `ScheduleState` | | [`flow.zod.ts`](/docs/references/automation/flow) | `Flow`, `FlowEdge`, `FlowNode`, `FlowNodeAction`, `FlowVariable`, `FlowVersionHistory` | | [`flow-function.zod.ts`](/docs/references/automation/flow-function) | `FlowFunctionEffect` | @@ -318,7 +317,7 @@ Studio designer metadata — the authoring surfaces for the protocols above. ## System Protocol -**Source:** `packages/spec/src/system/` · **Import:** `@objectstack/spec/system` · **37 pages, 296 schemas** +**Source:** `packages/spec/src/system/` · **Import:** `@objectstack/spec/system` · **37 pages, 292 schemas** The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. @@ -338,7 +337,7 @@ The runtime environment — logging, jobs, cache, metrics, notifications, i18n a | [`email-template.zod.ts`](/docs/references/system/email-template) | `EmailTemplateDefinition`, `EmailTemplateDefinitionCategory`, `EmailTemplateDefinitionVariable` | | [`encryption.zod.ts`](/docs/references/system/encryption) | `EncryptionAlgorithm`, `EncryptionConfig`, `FieldEncryption`, `KeyManagementProvider`, `KeyRotationPolicy` | | [`environment-artifact.zod.ts`](/docs/references/system/environment-artifact) | `Sha256Digest` | -| [`http-server.zod.ts`](/docs/references/system/http-server) | `MiddlewareConfig`, `MiddlewareType`, `RouteHandlerMetadata`, `ServerCapabilities`, `ServerEvent`, `ServerEventType`, `ServerStatus` | +| [`http-server.zod.ts`](/docs/references/system/http-server) | `MiddlewareConfig`, `MiddlewareType`, `RouteHandlerMetadata` | | [`incident-response.zod.ts`](/docs/references/system/incident-response) | `Incident`, `IncidentCategory`, `IncidentNotificationMatrix`, `IncidentNotificationRule`, `IncidentResponsePhase`, `IncidentResponsePolicy`, `IncidentSeverity`, `IncidentStatus` | | [`job.zod.ts`](/docs/references/system/job) | `CronSchedule`, `IntervalSchedule`, `Job`, `JobExecution`, `JobExecutionStatus`, `OnceSchedule`, `RetryPolicy`, `Schedule` | | [`license.zod.ts`](/docs/references/system/license) | `Feature`, `License`, `LicenseMetricType`, `Plan` | diff --git a/content/docs/references/integration/connector.mdx b/content/docs/references/integration/connector.mdx index c109cc96fa..405501c354 100644 --- a/content/docs/references/integration/connector.mdx +++ b/content/docs/references/integration/connector.mdx @@ -11,10 +11,12 @@ Defines the standard connector specification for external system integration. Connectors enable ObjectStack to sync data with SaaS apps, databases, file storage, and message queues through a unified protocol. -**Positioning in the sync/integration layering** (L1 "Simple Sync" was -retired in #4738 — narrative-only, zero consumers; see -`packages/spec/docs/SYNC_ARCHITECTURE.md`): -- **ETL Pipeline** ([automation/etl.zod.ts](/docs/references/automation/etl)) - Data engineers - Aggregate 10 sources to warehouse +**Positioning in the sync/integration layering** — this file is now the ONLY +layer. Both layers above it were retired under ADR-0049 for the same measured +reason, that no engine ever executed them: L1 "Simple Sync" +(`automation/sync.zod.ts`) in #4738, and L2 "ETL Pipeline" +(`automation/etl.zod.ts`) in #6414. See +`packages/spec/docs/SYNC_ARCHITECTURE.md`: - **Enterprise Connector** (THIS FILE) - System integrators - Full SAP integration; connector-attached sync via `syncConfig` **SCOPE: Most comprehensive integration layer.** @@ -95,9 +97,11 @@ Authentication is now imported from the canonical `auth/config.zod.ts`. - Microsoft Dynamics 365 connector **When to downgrade:** -- Data transformation only → Use [ETL Pipeline](/docs/references/automation/etl) - -See also: [../automation/etl.zod.ts](/docs/references/automation/etl) for the ETL Pipeline layer (data engineering) +- Per-field value conversion on import only → the import mapping's own + `transform` (`data/mapping.zod.ts`), which the REST import path executes + row by row. (This used to point at `automation/etl.zod.ts`; L2 was retired + at #6414 for having no executor, so the pointer would have been a signpost + landing nowhere — the same defect class this header names below.) ## There is no "Trigger Registry" alternative @@ -110,9 +114,11 @@ platform's authority, at a dead end (#4499; removed alongside the #4480 per-provider template cluster). The same defect class as the `capabilities.readOnly` prescription #4487 corrected: a signpost must land somewhere enforced. Lightweight cases are served HERE — a connector instance -with simple `auth` — or by `automation/etl.zod.ts` for transformation -pipelines. (The automation-side L1 "Simple Sync" file was itself retired as -a dead end of the same class in #4738.) +with simple `auth`. (Both automation-side layers were themselves retired as +dead ends of the same class: L1 "Simple Sync" in #4738, L2 `etl.zod.ts` in +#6414. This paragraph named L2 as the transformation destination until the +second retirement; a signpost that must land somewhere enforced cannot make +an exception for itself.) **Source:** `packages/spec/src/integration/connector.zod.ts` diff --git a/content/docs/references/system/http-server.mdx b/content/docs/references/system/http-server.mdx index 839e78dcce..b69fc374e4 100644 --- a/content/docs/references/system/http-server.mdx +++ b/content/docs/references/system/http-server.mdx @@ -21,8 +21,8 @@ Architecture alignment: ## TypeScript Usage ```typescript -import { MiddlewareConfigSchema, MiddlewareType, RouteHandlerMetadataSchema, ServerCapabilitiesSchema, ServerEventSchema, ServerEventType, ServerStatusSchema } from '@objectstack/spec/system'; -import type { MiddlewareConfig, MiddlewareType, RouteHandlerMetadata, ServerCapabilities, ServerEvent, ServerEventType, ServerStatus } from '@objectstack/spec/system'; +import { MiddlewareConfigSchema, MiddlewareType, RouteHandlerMetadataSchema } from '@objectstack/spec/system'; +import type { MiddlewareConfig, MiddlewareType, RouteHandlerMetadata } from '@objectstack/spec/system'; // Validate data const result = MiddlewareConfigSchema.parse(data); @@ -76,64 +76,3 @@ const result = MiddlewareConfigSchema.parse(data); --- -## ServerCapabilities - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **httpVersions** | `Enum<'1.0' \| '1.1' \| '2.0' \| '3.0'>[]` | ✅ | Supported HTTP versions | -| **websocket** | `boolean` | ✅ | WebSocket support | -| **sse** | `boolean` | ✅ | Server-Sent Events support | -| **serverPush** | `boolean` | ✅ | HTTP/2 Server Push support | -| **streaming** | `boolean` | ✅ | Response streaming support | -| **middleware** | `boolean` | ✅ | Middleware chain support | -| **routeParams** | `boolean` | ✅ | URL parameter support (/users/:id) | -| **compression** | `boolean` | ✅ | Built-in compression support | - - ---- - -## ServerEvent - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **type** | `Enum<'starting' \| 'started' \| 'stopping' \| 'stopped' \| 'request' \| 'response' \| 'error'>` | ✅ | Event type | -| **timestamp** | `string` | ✅ | Event timestamp (ISO 8601) | -| **data** | `Record` | optional | Event-specific data | - - ---- - -## ServerEventType - -### Allowed Values - -* `starting` -* `started` -* `stopping` -* `stopped` -* `request` -* `response` -* `error` - - ---- - -## ServerStatus - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **state** | `Enum<'stopped' \| 'starting' \| 'running' \| 'stopping' \| 'error'>` | ✅ | Current server state | -| **uptime** | `integer` | optional | Server uptime in milliseconds | -| **server** | `{ port: integer; host: string; url?: string }` | optional | | -| **connections** | `{ active: integer; total: integer }` | optional | | -| **requests** | `{ total: integer; success: integer; errors: integer }` | optional | | - - ---- - diff --git a/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md b/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md index 68619d5e59..fb2fa60a11 100644 --- a/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md +++ b/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md @@ -21,9 +21,9 @@ regenerate. | Measure | Value | |---|---| | Triaged directories | 5 | -| Object sites in them | 444 | -| Still-open (strip) sites | 185 | -| Files carrying at least one | 29 | +| Object sites in them | 434 | +| Still-open (strip) sites | 182 | +| Files carrying at least one | 28 | Remaining strip sites by class: @@ -31,7 +31,7 @@ Remaining strip sites by class: |---|---| | authorable — the ruling's forced scope | 43 | | unresolved — needs a per-schema verdict | 33 | -| wire / open — out of forced scope | 107 | +| wire / open — out of forced scope | 104 | | no door — no carrier, ADR-0049 territory | 1 | | no gate — carrier live, no parse | 0 | | covered — no carrier, no parse, guarded at every consumer | 1 | @@ -46,10 +46,10 @@ The `strict` column is the one the campaign schedules against; it counts both th |---|---|---|---|---|---| | `ui/` | 160 | 116 | 5 | 0 | 39 | | `data/` | 162 | 54 | 1 | 0 | 107 | -| `automation/` | 75 | 49 | 0 | 0 | 26 | +| `automation/` | 65 | 42 | 0 | 0 | 23 | | `security/` | 20 | 7 | 0 | 0 | 13 | | `studio/` | 27 | 27 | 0 | 0 | 0 | -| **total** | **444** | **253** | **6** | **0** | **185** | +| **total** | **434** | **246** | **6** | **0** | **182** | ## File-level triage — site counts @@ -118,7 +118,6 @@ classify and is not listed (it becomes reportable the day it grows its first sit | `bpmn-interop.zod.ts` | 5 | | `builtin-node-config.zod.ts` | 8 | | `control-flow.zod.ts` | 5 | -| `etl.zod.ts` | 10 | | `execution.zod.ts` | 13 | | `flow-function.zod.ts` | 1 | | `flow.zod.ts` | 11 | @@ -128,7 +127,7 @@ classify and is not listed (it becomes reportable the day it grows its first sit | `state-machine.zod.ts` | 6 | | `time-relative-trigger.zod.ts` | 1 | | `webhook.zod.ts` | 1 | -| **total** | **75** | +| **total** | **65** | ### `security/` — sites @@ -213,22 +212,21 @@ over it is here. ### `automation/` — open -**26 strip of 75**, in 5 file(s). +**23 strip of 65**, in 4 file(s). | File | Strip | Sites | |---|---|---| | `bpmn-interop.zod.ts` | 5 | 5 | -| `etl.zod.ts` | 3 | 10 | | `execution.zod.ts` | 13 | 13 | | `flow.zod.ts` | 1 | 11 | | `node-executor.zod.ts` | 4 | 4 | -| **total** | **26** | **75** | +| **total** | **23** | **65** | | Bucket | Sites | |---|---| | authorable — the ruling's forced scope | 0 | | unresolved — needs a per-schema verdict | 0 | -| wire / open — out of forced scope | 26 | +| wire / open — out of forced scope | 23 | | no door — no carrier, ADR-0049 territory | 0 | | no gate — carrier live, no parse | 0 | | covered — no carrier, no parse, guarded at every consumer | 0 | @@ -266,11 +264,11 @@ directory rather than per file. | Dir | Sites | |---|---| | `ai/` | 77 | -| `api/` | 401 | +| `api/` | 391 | | `cloud/` | 83 | | `identity/` | 33 | | `integration/` | 10 | | `kernel/` | 319 | | `qa/` | 6 | | `shared/` | 20 | -| `system/` | 367 | +| `system/` | 361 | diff --git a/docs/audits/2026-07-unknown-key-strictness-ledger.md b/docs/audits/2026-07-unknown-key-strictness-ledger.md index c07b69ec6f..48c0b0738e 100644 --- a/docs/audits/2026-07-unknown-key-strictness-ledger.md +++ b/docs/audits/2026-07-unknown-key-strictness-ledger.md @@ -711,7 +711,6 @@ column does not move and the `strip` column falls by the count of what left. | File | Class | Note | |---|---|---| | `flow.zod.ts` | authorable | **strict as of #4001** — the four outer authoring shapes at step 1, and **the six nested blocks at batch 11** (`FlowNode.connectorConfig` / `.position` / `.inputSchema` / `.waitEventConfig` / `.boundaryConfig`, `Flow.errorHandling`). The gap between those two dates is this campaign's own finding 17 inside its own file: closing the shells left the gate rejecting `nodee:` at node level while `connectorConfig: { connectorId, actionId, params: {…} }` parsed clean and the executor dispatched `input ?? {}` — a successful connector call carrying nothing. Worth recording precisely, because the obvious example is the wrong one: a slip on a REQUIRED key was always loud (it then reads as missing). What `.strip` swallowed here is the OPTIONAL half — the input map, the retry budget, `interrupting: false`, `required: true` — i.e. exactly the keys an author adds to CONSTRAIN behaviour, replaced by a permissive default without a word. `Flow.errorHandling` gained a second chapter at **#4964**: closing it in 批 11 revealed (rather than caused) that its retry keys were a THIRD encoding of the policy #4661 had converged — it spelled the base delay `retryDelayMs` where the shared declaration spells it `backoffMs` and tombstones the old word, so the strictness this row records was, for one release, rejecting an author for having read the newer file. The block now builds from `retryPolicyShape()`. Site count unchanged; only the vocabulary. Two things stay open and are now pinned in code with the reason, so a later sweep stops rather than "finishes" the file: the node `config` slot (ADR-0018 plugin namespace) and `FlowVersionHistorySchema` (the file's only WIRE shape — emitted on publish, never authored; its `definition` is `FlowSchema`, so the authored half inside a history record is gated anyway) | -| `etl.zod.ts` | mixed | **7 strict as of #4001 批 12** — the authoring half (`ETLSource` + `.incremental`, `ETLDestination`, `ETLTransformation`, `ETLPipeline` + `.retry` + `.notifications`). The other 3 — `ETLPipelineRun` + `.stats` + `.error` — are **deliberately left open**: engine-emitted run state (an id it minted, a status it reached, counters it accumulated), same disposition and same reason as `FlowVersionHistorySchema` above and all of `execution.zod.ts`. The exemption is recorded on the schema itself, not only here, because a note only this file carries is a note the next sweep does not read. The old blanket `authorable (p)` was too wide; verification split it. ⚠️ **Read the classification caveat before reusing this verdict**: `etl.zod.ts` has NO parse site in objectstack / objectui / cloud, so neither half could be settled by pointing at a live call. The 7 are authorable because the exported schema and type ARE the door (`SYNC_ARCHITECTURE.md` and the module's `@example` both hand-write `const p: ETLPipeline = { … }`) — the `webhook.zod.ts` posture. The 3 are wire on the shape's semantics plus settled precedent, NOT on an emit site anyone can point at today; if an ETL engine ever lands and a run result turns out to be operator-authored, that verdict is the one to revisit. Two out-of-scope findings were filed rather than fixed here; **the first is now closed**: the `retry` block was a third retry-policy vocabulary #4661's convergence never reached (#4962 — converged onto `shared/RetryPolicySchema` in the v17 window, together with `flow.errorHandling` (#4964), the fourth. Both were anonymous inline blocks, so the dual-source instrument that drove #4661 could not see them: it asks how many declarations share an exported NAME, and neither has one. 批 12's five curated `retry` entries described that divergence and dissolved with it — the block's site count is unchanged, only its vocabulary). **The second is now closed too**: all nine type aliases exported the PARSED shape under the bare name with no `*Parsed` counterpart, so the authoring door this row's whole classification rests on — `const p: ETLPipeline = { … }` — did not actually compile, and the SYNC_ARCHITECTURE.md examples proving it were the evidence (#4963 — bare names flipped to `z.input`, nine `*Parsed` added, house convention per `shared/retry-policy.zod.ts`; three-repo zero importers made the migration surface empty). Worth carrying forward for the next classification: **"authorable because the exported type IS the door" is a claim about a type that must be checked, not assumed** — 批 12 read the door correctly and nobody compiled it. **−12 at #4738**: `sync.zod.ts` (the L1 "Simple Sync" file — `DataSyncConfig`, its `ConflictResolution` enum and satellites, formerly this row's co-candidate) was deleted whole rather than hardened: three-repo zero importers, no parse site, defs unreachable from the metadata-type roots (#4650 gate), so there was no author for strictness to protect (#4535 C13+C15). The integration-side `ConflictResolution` → `ConnectorConflictResolution` rename in the same change is name-only and moves no sites | | `execution.zod.ts` | wire | run-state envelopes — never strict. +5 at #4354 (the run-summary family: step metrics / skip reason / per-node / per-gate / the summary itself) — engine-emitted telemetry read by the Console and by operator queries, nobody authors them, so the `wire` verdict covers them unchanged | | `state-machine.zod.ts` | authorable | **strict as of #4001 批 10** — all six sites (`ActionRef` / `GuardRef` / `Transition` / `StateNode` + `.meta` / `StateMachine`). **The `(p)` was NOT a formality here.** ADR-0020 retired this XState shape as a *record-lifecycle* declaration — the top-level `workflow` metadata type and `object.stateMachines` are both gone, and a record's transitions live on the `state_machine` VALIDATION RULE instead — so had those been the only doors this file would be DEAD surface, and the correct action would have been to fix its class, not close it. One authoring door survives: `ai/agent.zod.ts`'s `lifecycle` is `StateMachineSchema`, and `agent` is a registered type, so `defineStack({ agents })` / meta REST / the Studio agent form all reach here through `AgentSchema.parse()`. Verified by parse: an agent whose lifecycle carried `stats`, a state with `onn` (one keystroke from `on`) and a `meta` with two unknown keys **parsed clean**, returning a machine with NO transitions at all — the declaration whose whole job is to deny undeclared transitions, silently emptied and reported valid. `.meta` was checked for the #4909 open-slot case and is CLOSED: the hand-written `StateNodeConfig` type declares exactly its four keys (passthrough would open the Zod while `tsc` stayed shut), nothing in the repo reads any `meta` key, and the prior behaviour was strip — an author's `meta` arrived as `{}` — so there was no openness to preserve. ⚠️ `ActionRef` / `GuardRef` are UNIONS: a strict branch's message does not reach the top (zod raises one `invalid_union` whose message is the literal `"Invalid input"`, with the real prescription nested in `issue.errors[]`), which `formatZodError` then flattens away — filed, not fixed here. **−1 at #4658**: the orphan `EventSchema` (`{ type, schema }`, an XState-style signal declaration nothing referenced — `StateMachineSchema` names event types as `on:` record keys) was deleted rather than converged with `kernel/events/core.zod.ts`'s envelope `EventSchema`, whose key set it did not intersect (#4535 C6). The remaining 6 sites and their verdict are unchanged | | `control-flow.zod.ts` | authorable | **strict as of #4001 批 10** — all five sites (`FlowRegion` / `Loop` / `ParallelBranch` / `Parallel` / `TryCatch`). The `(p)` resolves to authorable on the executors' own parse seam (`parseNodeConfig`, #4277) plus `validateControlFlow`'s region parse. **`validateControlFlow` is a sibling guard, not a key gate, and the two do not fight**: it answers single-entry / single-exit / acyclic, which no key check can decide, and the schema answers key membership, which no structural check can decide. They meet at exactly one seam — the guard `safeParse`s each region slot before analyzing it, so an undeclared region key now surfaces there as `: invalid region — `, the guard's framing wrapping the schema's prescription. Nothing was duplicated and nothing removed; the guard simply stopped silently repairing its own input before judging it. Two curation entries had to be MEASURED rather than reasoned: the bare edit-distance fallback answers `itemVariable` with **`indexVariable`** — binding the loop INDEX where the author wanted the ITEM — so the alias exists to overrule a confidently wrong suggestion from this campaign's own helper (the `pii` → `min` shape, third instance); and `join`/`joinGateway` needed two DISTINCT prescriptions because `guidance` emits one bullet per key verbatim, so a shared string printed the same paragraph twice. Its test instrument also had to be rebuilt: `region-slots.test.ts` probed every construct with every candidate key at once and depended on `.strip` to discard the mismatches, so it returned "no schema accepts any region" the moment the shapes closed — it failed loudly, which is the only reason this is a footnote and not a fourth finding-3. Structural validation by `validateControlFlow` remains. **−1 at #4661**: `RetryPolicySchema` moved out to `shared/retry-policy.zod.ts` — `./automation` and `./system` published the same name for two different declarations (#4411), so the retry policy converged onto one. The site still exists and is still non-strict and authorable; it is simply no longer in a directory this ledger sections. ⚠️ That is a coverage gap worth knowing about: this audit sections `ui/` / `data/` / `automation/` / `security/` / `studio/` only, so a `shared/` shape is unaudited by construction. The tolerance is deliberate here — the `retryDelayMs` → `backoffMs` rename is tombstoned via `retiredKey()` precisely because a non-strict parent would otherwise swallow the old spelling. **#4964 widened that rename to `flow.errorHandling`**, which spelled the base delay the pre-17 way while the shared policy tombstoned it — so the two automation retry surfaces now teach the same word, and the tombstone's prescription names all four surfaces instead of the two #4661 could see | @@ -804,14 +803,14 @@ them from scratch next batch. | File | Class | Batch | |---|---|---| | `execution.zod.ts` | wire | **out of scope** — engine-emitted run state; the ledger row already says "never strict" | -| `etl.zod.ts` | wire | **Authorable half closed at 批 12** (7 sites: `ETLSource` + `.incremental`, `ETLDestination`, `ETLTransformation`, `ETLPipeline` + `.retry` + `.notifications`). `.retry` was re-pointed at the shared `RetryPolicySchema` at #4962 (`maxAttempts` → `maxRetries`, default 3 → 0, three knobs gained) — a vocabulary change inside an already-closed site, so this row's numbers do not move. What is left is `ETLPipelineRun` + `.stats` + `.error` — engine-emitted run state, exempt for the `FlowVersionHistorySchema` reason and pinned as such in `etl.test.ts`, so closing it means deleting a test that says not to. **This row shrinks without disappearing** — the second in `automation/` to do so, after `flow.zod.ts` reached its own wire floor of 1 at 批 11 (the two batches were in flight together and arrived at the same shape independently, which is the better evidence that it is the right one). Worth naming because the reverse pin cannot see it: the pin fires on zero, so a row that stops at its wire floor looks exactly like a row nobody finished. The Class column is the only thing separating them — read it before treating this as unfinished work | | `flow.zod.ts` | wire | **batch 11 closed the 6 authorable** (`FlowNode.connectorConfig` / `.position` / `.inputSchema` / `.waitEventConfig` / `.boundaryConfig`, `Flow.errorHandling`). (`Flow.errorHandling`'s retry keys were re-pointed at the shared `RetryPolicySchema` at #4964 — a vocabulary change inside an already-closed site, so this row's numbers do not move.) The 1 left is `FlowVersionHistorySchema`, which this table has exempted since it was written — **do not close it**: it is emitted on publish, not authored, so closing it makes a future emitter-side field a parse failure for whoever reads history. The exemption now also lives beside the schema and in `flow.test.ts`, because a row in a table is not where the next person to open that file will look | | `bpmn-interop.zod.ts` | wire (p) | **out of scope** — third-party BPMN import/export shapes; strictness turns an upstream addition into our parse crash | | `node-executor.zod.ts` | wire | **out of scope** — executor registration contract, code-to-code | -Eight rows have left this table across three waves of the ruling's `automation/` -main body, each one on reverse-pin evidence — the row was deleted because the -gate went red on it still being there, not because someone remembered: +Nine rows have left this table — eight across three waves of the ruling's +`automation/` main body, each one on reverse-pin evidence (the row was deleted +because the gate went red on it still being there, not because someone +remembered), and one because its file was retired outright: | wave | rows removed | other change | |---|---|---| @@ -819,6 +818,29 @@ gate went red on it still being there, not because someone remembered: | **批 10** (#4973) | `control-flow` (5) · `state-machine` (6) | — | | **批 11** (#4974) | `flow-function` (1) · `time-relative-trigger` (1) · `webhook` (1) | `flow.zod.ts` 7 → 1 | | **批 12** (#4979) | — | `etl.zod.ts` 10 → 3 | +| **#6414** (ADR-0049) | `etl.zod.ts` (3) | the file was DELETED, not hardened | + +**The ninth row left for a reason none of the four waves above shares, and it is +worth separating.** 批 9–12 removed rows because the sites were CLOSED — the +schema went `strictObject` and the reverse pin went red on a row with nothing +left to classify. `etl.zod.ts` left because the whole L2 ETL layer was retired +under ADR-0049 enforce-or-remove (#6414): no engine ever parsed, scheduled or +executed an `ETLPipeline`, so there was no author for strictness to protect. That +is the `sync.zod.ts` disposition (#4738), which this ledger recorded from the +inside — the old `etl.zod.ts` triage row carried the `−12 at #4738` clause +describing L1's deletion, and its own classification caveat said the quiet part +out loud: *"`etl.zod.ts` has NO parse site in objectstack / objectui / cloud, so +neither half could be settled by pointing at a live call"*, and the 7 authorable +sites were authorable *"because the exported schema and type ARE the door"*. +A door nobody walked through. The row that recorded L1's retirement has now been +removed by the same reading applied one layer up, which is the ledger working: +the caveat it insisted on writing down is what made the second verdict cheap. + +⚠️ **Do not read this as "hardening was wasted work".** 批 12's measurement is +exactly what made #6414 decidable — it is the reason the file's authoring door +was known to be a type annotation rather than a parse, and #4963 (the nine +`*Parsed` aliases) is the reason anyone had checked that the door compiled at +all. Retirement and hardening answer different questions in that order. **How those waves met is worth recording, because it is the failure mode this table is most exposed to.** Each PR deleted its own rows and decremented this diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index 24f5ebf800..a1e531a88a 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -282,9 +282,6 @@ Finally it narrows the aggregation vocabulary: `array_agg` and `string_agg` leav - **`job-retry-policy-constraints-tightened`** — `job.retryPolicy.maxRetries (> 10) / job.retryPolicy.backoffMultiplier (< 1)` → maxRetries <= 10, and backoffMultiplier >= 1 - Why not automatic: The converged RetryPolicy (#4661) keeps the automation side's bounds, which the job side never had: `maxRetries` is capped at 10 and `backoffMultiplier` floored at 1. Neither has a lossless rewrite. Clamping `maxRetries: 20` to 10 would halve a retry budget its author chose, and a `backoffMultiplier` below 1 describes a delay that SHRINKS on each attempt — retrying a failing dependency ever faster, which is the opposite of backoff and was never a shape the engine meant to offer. Both now fail at parse time with the bound named, rather than being silently reinterpreted. Choosing the replacement count (or accepting the cap) is the author's call. - Done when: Every job declaring `retryPolicy` parses: no `maxRetries` above 10 and no `backoffMultiplier` below 1 remain, and each adjusted value was re-chosen knowing a retry re-runs the handler with its writes and callouts. No job fails to register with the retry-policy bound prescription. -- **`etl-retry-converged-onto-retry-policy`** — `etlPipeline.retry.maxAttempts (and any count above 10)` → maxRetries, same number — plus an explicit count if you relied on the old default of 3 - - Why not automatic: An ETL pipeline's `retry` was a THIRD retry vocabulary that #4661's convergence never reached, because that pass was driven by duplicated exported NAMES and this block is an anonymous inline object (#4962). It now carries the shared `RetryPolicySchema` contract, which changes three things with no single lossless rewrite between them. The rename `maxAttempts` → `maxRetries` IS lossless and the tombstone performs it — both keys counted the retries AFTER the initial attempt, so the number does not change, and subtracting one (correct for `integration/connector.zod.ts`'s identically-spelled `RetryConfig.maxAttempts`, which includes the first attempt) would silently run one attempt fewer than asked. What needs a human: the count now DEFAULTS TO 0 instead of 3, so a pipeline that wrote `retry: {}` or omitted the count bought three silent re-runs and now buys none. That is deliberate and the business case is the destination — an ETL destination is a foreign system by definition, and an implicit retry against a non-idempotent one is a duplicate write (a second invoice, a second export, a second webhook). Retrying is now something an author states and thereby claims idempotency for. The shared contract also caps `maxRetries` at 10, which this block never did; clamping a larger budget would silently halve a number its author chose, so it fails at parse with the bound named instead. - - Done when: No ETL pipeline declares `retry.maxAttempts`; every one that wants retries declares `maxRetries` >= 1 explicitly (the number carried over unchanged from `maxAttempts`), and every pipeline that was relying on the old implicit 3 has either written `maxRetries: 3` or been re-decided against the duplicate-write risk at its destination. No count exceeds 10. Pipelines that want the old flat 60s backoff state `backoffMs: 60000` explicitly, since the shared default is 1000. - **`flow-retry-max-retries-required`** — `flow.errorHandling.maxRetries (under strategy: 'retry')` → an explicit count >= 1 (e.g. maxRetries: 3), or strategy: 'fail' - Why not automatic: maxRetries had two defaults — FlowSchema `.default(0)` and the engine's `maxRetries ?? 3` — so an unstated count retried 0 times through the schema and 3 times through a hand-built definition (#4247). With the engine's copy removed the unstated count is unambiguously 0, and retrying zero times is exactly `strategy: 'fail'`, so the schema now refuses the combination instead of it silently doing nothing. There is no lossless rewrite: 0 preserves the behaviour a parsed flow got but contradicts what its author wrote, and any positive count is a NEW decision about re-running the whole flow with its side effects. That choice is the author's. - Done when: Every flow declaring `errorHandling.strategy: 'retry'` also declares `maxRetries` >= 1, and each count was chosen knowing a retry replays the flow FROM THE START (records re-created, callouts re-fired); flows that never actually wanted retries say `strategy: 'fail'`. No flow fails to register with the maxRetries prescription. @@ -378,6 +375,15 @@ Finally it narrows the aggregation vocabulary: `array_agg` and `string_agg` leav - **`driver-sql-distinct-bare-filter-typed`** — `SqlDriver.distinct() third argument — any value` → a bare FilterCondition (@objectstack/spec/data) — the same value find() carries under query.where, never a query envelope - Why not automatic: This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning "which products among completed orders" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320. - Done when: No caller passes a non-object to `distinct()`'s third argument. A scalar there is now a compile error (`TS2345: Argument of type 'string' is not assignable to parameter of type 'FilterCondition'`); rewrite it as the bare filter it was always meant to be — `'completed'` becomes `{ status: 'completed' }`. ⚠️ That is NOT an equivalent rewrite: the old spelling returned the UNFILTERED set, so the answer changes once fixed, and the changed answer is the one the call always meant. An untyped JS caller gets no compile error and no behaviour change — for them this entry is the only notice that the spelling never filtered anything. A query envelope or a FilterArray in that slot still compiles and is rejected at run time with INVALID_FILTER / 400. +- **`http-server-runtime-vocabulary-retired`** — `system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)` → (removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected) + - Why not automatic: The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so "zero consumers in this repo" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295. + - Done when: No source imports `ServerEvent`, `ServerEventType`, `ServerEventSchema`, `ServerCapabilities`, `ServerCapabilitiesSchema`, `ServerCapabilitiesParsed`, `ServerStatus` or `ServerStatusSchema` from `@objectstack/spec/system` — a grep over consumer code resolves none of them, and `tsc` reports TS2724/TS2305 on any that survives. The route-registration half of the same module still resolves (`RouteHandlerMetadataSchema`, `MiddlewareType`, `MiddlewareConfigSchema`, `MiddlewareConfig`), and `StackServerConfigSchema` — the one authorable server surface — is untouched: a stack declaring `server: { trustProxy, security }` parses exactly as it did in 16.x. +- **`view-management-protocol-retired`** — `api.listViews / api.getView / api.createView / api.updateView / api.deleteView (the ViewProtocol interface and its ten Request/Response schemas in api/protocol.zod.ts — 10 defs, 25 exported names)` → the two view surfaces that are actually routed. For a view's STORED definition, the generic metadata methods with `type: 'view'` — `getMetaItem` / `getMetaItems` / `saveMetaItem` / `deleteMetaItem`, served at `/api/v1/meta/view/:name`. For the RESOLVED render-time view, `getUiView` (`GetUiViewRequest` / `GetUiViewResponse`), served at `/api/v1/ui/view/:object/:type`. Neither is addressed by a `viewId`, which is the one thing the retired surface offered and the one thing nothing implemented + - Why not automatic: A complete viewId-addressed CRUD surface — list (with a list/form filter), read, create, patch, delete — with none of the three things a protocol method needs. Measured on origin/main immediately before the removal: no implementation (`packages/metadata-protocol/src/protocol.ts` declares no `listViews` / `getView` / `createView` / `updateView` / `deleteView`; its only view resolver is `getUiView`), no route (`packages/rest/src/rest-server.ts` never mentions `viewId`, so nothing viewId-addressed is reachable over HTTP at all), and no caller (the only `ViewProtocol` mention outside its own file was the services checklist, which already recorded the five as declared-and-unrouted). The look-alike hits a bare-name grep turns up are all different contracts: `metadata-manager.ts`'s `getView(name: string)` is another class, and objectui's `getView(objectName, viewId)` resolves through `client.meta.getItem('view', …)`, i.e. the metadata route. What makes this worth a removal rather than a note is that the cost is already measured. A declared surface that is name-identical and semantics-adjacent to a real one is an attractive nuisance in every grep, and it mis-directed a decision once: #5948's issue body AND its 2026-08-07 maintainer ruling both read `GetViewResponseSchema` (zero implementations) as the contract of `GET /ui/view/:object/:type`, whose declared response is `GetUiViewResponseSchema` — one word apart, 250 lines up. That ruling's reasoning happened to survive the mix-up ("nobody can consume `{object, view}` successfully today" was true, though not for the stated reason), which is the luck this removal stops relying on. Route 3: none of the ten was a key on an authorable shape, nothing parsed them, so there is no tombstone and no D2 conversion — RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. If reading and writing ONE view by id becomes a real requirement it returns implementation-first. ADR-0049, ADR-0087, maintainer ruling 2026-08-07, #6239. + - Done when: No source imports `ListViewsRequest(Schema)`, `ListViewsResponse(Schema)`, `GetViewRequest(Schema)`, `GetViewResponse(Schema)`, `CreateViewRequest(Schema)`, `CreateViewResponse(Schema)`, `UpdateViewRequest(Schema)`, `UpdateViewResponse(Schema)`, `DeleteViewRequest(Schema)` or `DeleteViewResponse(Schema)` from `@objectstack/spec/api`, and no host declares a `ViewProtocol` member. Reading and writing views still works end to end through the surfaces that were always the live ones: `GET /api/v1/meta/view/:name` returns the stored definition and `GET /api/v1/ui/view/:object/:type` returns the resolved view, both unchanged by this removal. `GetUiViewRequestSchema` / `GetUiViewResponseSchema` still resolve — they are the shapes #5948 meant. +- **`etl-pipeline-layer-retired`** — `automation.etlPipeline / automation.etlPipelineRun / automation.etlSource / automation.etlDestination / automation.etlTransformation (the whole L2 layer of automation/etl.zod.ts, its four enums and the `ETL` factory — 9 defs, 27 exported names)` → (removed — no protocol surface replaces it, deliberately. Layer by layer: connector-attached synchronisation is `ConnectorSchema.syncConfig` (`integration/connector.zod.ts`), which IS parsed and executed; per-field value transformation on import is `shared/mapping.zod.ts`, whose `transform` is applied row by row by the REST import path and recorded key by key in `packages/spec/liveness/mapping.json`; scheduling is `system/job.zod.ts`. What has NO replacement is multi-source, multi-stage movement with joins and aggregations — because it never had an implementation either. It returns through the ENFORCE route: the engine first, the vocabulary second) + - Why not automatic: The reading #4738 used to retire L1 `DataSyncConfig`, re-measured one layer up and identical: narrative-only. No engine ever parsed, scheduled or executed an `ETLPipeline`. Measured on origin/main immediately before the removal: the only non-spec references in this repo are two fumadocs-generated documentation sources (`apps/docs/.source/*.ts`), not executors; objectui has no reference at all; there is no `liveness/etl.json` or `pipeline.json`, so no ADR-0049 gate ever had a reading on it — while the same file family's EXECUTED half does have one (`liveness/mapping.json`), which is the contrast that makes the absence meaningful rather than an oversight. The `etl` string in this registry was the one untested link the finding named, and it is not a loader path: it was the id of the #4962 retry-vocabulary entry, absorbed here. The layer was ADR-0078's asymmetry in its purest form — an author could write a complete ten-stage pipeline, get no error, and get no execution. It was also advertised: `packages/spec/docs/SYNC_ARCHITECTURE.md` named `ETLPipeline` as the recommended destination for authors displaced by the L1 retirement (#4738) and listed ten transformation types with copyable examples down to `script | Custom JavaScript/Python`. That document is rewritten in the same change; a retirement whose own doc still recommends the retired layer is self-contradictory, and forwarding L1's authors to a second layer with no executor was the defect compounding rather than closing. ⚠️ `etl-retry-converged-onto-retry-policy` (#4962) is SUBSUMED here, the #4657/#4834/#5055 way: both land in the unreleased protocol 17, so composed, a rename of `retry.maxAttempts` on a shape that does not survive the major has no observable effect — and keeping both would tell an upgrader to rewrite a key on a schema the same upgrade deletes. The `maxAttempts` `retiredKey()` tombstone goes with the shape that carried it, which is strictly stronger than the tombstone: there is no longer a `retry` block to author the key into. Route 3 — no carrier key, no parse site, so no D2 conversion and no tombstone; RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. ADR-0049, ADR-0078, #6414. + - Done when: No source imports `ETLPipeline`, `ETLPipelineParsed`, `ETLPipelineSchema`, `ETLPipelineRun(Schema)`, `ETLSource(Schema)`, `ETLDestination(Schema)`, `ETLTransformation(Schema)`, `ETLEndpointType(Schema)`, `ETLTransformationType(Schema)`, `ETLSyncMode(Schema)`, `ETLRunStatus(Schema)` or the `ETL` factory from `@objectstack/spec/automation`; `tsc` reports TS2724/TS2305 on any that survives. Every author who was pointed at L2 has been re-pointed by name: SYNC_ARCHITECTURE.md no longer lists an L2 row, no longer recommends `ETLPipeline` as L1's destination and no longer advertises a transformation-type table. The surviving layers still parse unchanged — a connector declaring `syncConfig` and an import declaring `mapping.transform` both behave exactly as they did in 16.x. --- diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index 083256a677..90dde58b30 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -5207,11 +5207,12 @@ export type { GetPresenceResponse, // Workflow re-exports removed (#4451, v17): the types were deleted from // @objectstack/spec/api with the retired workflow slot. - ListViewsResponse, - GetViewResponse, - CreateViewResponse, - UpdateViewResponse, - DeleteViewResponse, + // View-management re-exports removed (#6239, v17): the five viewId-addressed + // methods and their ten schemas were deleted from @objectstack/spec/api with + // the retired `ViewProtocol` — no host implemented them and no route reached + // them. A view's stored definition travels on the metadata types + // (`GetMetaItemResponse` / `SaveMetaItemResponse` with `type: 'view'`), and + // the resolved render-time view on `getUiView`. RegisterDeviceRequest, RegisterDeviceResponse, ListNotificationsResponse, diff --git a/packages/spec/PROTOCOL_MAP.md b/packages/spec/PROTOCOL_MAP.md index d7d34d6e60..77033104a9 100644 --- a/packages/spec/PROTOCOL_MAP.md +++ b/packages/spec/PROTOCOL_MAP.md @@ -70,7 +70,6 @@ This document serves as the **Grand Map** of the ObjectStack specification. It l | [`flow.zod.ts`](src/automation/flow.zod.ts) | ⭐ | **Visual Flow**. Complex orchestration logic (decisions, loops, CRUD). | | [`approval.zod.ts`](src/automation/approval.zod.ts) | ⭐ | **Approval Node**. Flow node config for human approval pauses. | | [`webhook.zod.ts`](src/automation/webhook.zod.ts) | ⭐ | **Webhooks**. Outbound HTTP notification configuration. | -| [`etl.zod.ts`](src/automation/etl.zod.ts) | | **ETL Jobs**. Extract-Transform-Load definitions. | --- diff --git a/packages/spec/api-surface/api.json b/packages/spec/api-surface/api.json index 761ac5507c..ee12fb522e 100644 --- a/packages/spec/api-surface/api.json +++ b/packages/spec/api-surface/api.json @@ -200,12 +200,6 @@ "CreateManyDataResponseSchema (const)", "CreateRequest (type)", "CreateRequestSchema (const)", - "CreateViewRequest (type)", - "CreateViewRequestParsed (type)", - "CreateViewRequestSchema (const)", - "CreateViewResponse (type)", - "CreateViewResponseParsed (type)", - "CreateViewResponseSchema (const)", "CrossObjectBatchDroppedFields (type)", "CrossObjectBatchDroppedFieldsSchema (const)", "CrossObjectBatchOperation (type)", @@ -268,10 +262,6 @@ "DeleteResponse (type)", "DeleteResponseParsed (type)", "DeleteResponseSchema (const)", - "DeleteViewRequest (type)", - "DeleteViewRequestSchema (const)", - "DeleteViewResponse (type)", - "DeleteViewResponseSchema (const)", "DeviceRequestResponse (type)", "DeviceRequestResponseParsed (type)", "DeviceRequestResponseSchema (const)", @@ -469,11 +459,6 @@ "GetUiViewResponse (type)", "GetUiViewResponseParsed (type)", "GetUiViewResponseSchema (const)", - "GetViewRequest (type)", - "GetViewRequestSchema (const)", - "GetViewResponse (type)", - "GetViewResponseParsed (type)", - "GetViewResponseSchema (const)", "HandlerStatus (type)", "HandlerStatusSchema (const)", "HttpFindQueryParamsSchema (const)", @@ -570,11 +555,6 @@ "ListRunsResponse (type)", "ListRunsResponseParsed (type)", "ListRunsResponseSchema (const)", - "ListViewsRequest (type)", - "ListViewsRequestSchema (const)", - "ListViewsResponse (type)", - "ListViewsResponseParsed (type)", - "ListViewsResponseSchema (const)", "LoginRequest (type)", "LoginRequestParsed (type)", "LoginRequestSchema (const)", @@ -950,12 +930,6 @@ "UpdateNotificationPreferencesResponseSchema (const)", "UpdateRequest (type)", "UpdateRequestSchema (const)", - "UpdateViewRequest (type)", - "UpdateViewRequestParsed (type)", - "UpdateViewRequestSchema (const)", - "UpdateViewResponse (type)", - "UpdateViewResponseParsed (type)", - "UpdateViewResponseSchema (const)", "UploadArtifactRequest (type)", "UploadArtifactRequestParsed (type)", "UploadArtifactRequestSchema (const)", @@ -989,7 +963,6 @@ "VersioningConfigParsed (type)", "VersioningConfigSchema (const)", "VersioningStrategy (type)", - "ViewProtocol (interface)", "WELL_KNOWN_CAPABILITY_KEYS (const)", "WebSocketConfig (type)", "WebSocketConfigParsed (type)", diff --git a/packages/spec/api-surface/automation.json b/packages/spec/api-surface/automation.json index 31d4703e7d..97767d9b31 100644 --- a/packages/spec/api-surface/automation.json +++ b/packages/spec/api-surface/automation.json @@ -76,34 +76,6 @@ "DeleteRecordConfig (type)", "DeleteRecordConfigParsed (type)", "DeleteRecordConfigSchema (const)", - "ETL (const)", - "ETLDestination (type)", - "ETLDestinationParsed (type)", - "ETLDestinationSchema (const)", - "ETLEndpointType (type)", - "ETLEndpointTypeParsed (type)", - "ETLEndpointTypeSchema (const)", - "ETLPipeline (type)", - "ETLPipelineParsed (type)", - "ETLPipelineRun (type)", - "ETLPipelineRunParsed (type)", - "ETLPipelineRunSchema (const)", - "ETLPipelineSchema (const)", - "ETLRunStatus (type)", - "ETLRunStatusParsed (type)", - "ETLRunStatusSchema (const)", - "ETLSource (type)", - "ETLSourceParsed (type)", - "ETLSourceSchema (const)", - "ETLSyncMode (type)", - "ETLSyncModeParsed (type)", - "ETLSyncModeSchema (const)", - "ETLTransformation (type)", - "ETLTransformationParsed (type)", - "ETLTransformationSchema (const)", - "ETLTransformationType (type)", - "ETLTransformationTypeParsed (type)", - "ETLTransformationTypeSchema (const)", "ExecutionError (type)", "ExecutionErrorParsed (type)", "ExecutionErrorSchema (const)", diff --git a/packages/spec/api-surface/system.json b/packages/spec/api-surface/system.json index 3594ed81bc..64c57c287b 100644 --- a/packages/spec/api-surface/system.json +++ b/packages/spec/api-surface/system.json @@ -555,17 +555,9 @@ "SecurityEventCorrelation (type)", "SecurityEventCorrelationParsed (type)", "SecurityEventCorrelationSchema (const)", - "ServerCapabilities (type)", - "ServerCapabilitiesParsed (type)", - "ServerCapabilitiesSchema (const)", - "ServerEvent (type)", - "ServerEventSchema (const)", - "ServerEventType (type)", "ServerRateLimitConfig (type)", "ServerRateLimitConfigParsed (type)", "ServerRateLimitConfigSchema (const)", - "ServerStatus (type)", - "ServerStatusSchema (const)", "ServiceConfigSchema (const)", "ServiceCriticalitySchema (const)", "ServiceLevelIndicator (type)", diff --git a/packages/spec/authorable-defaults/automation.json b/packages/spec/authorable-defaults/automation.json index 24cb8103a1..7b62499363 100644 --- a/packages/spec/authorable-defaults/automation.json +++ b/packages/spec/authorable-defaults/automation.json @@ -36,10 +36,6 @@ "automation/ConcurrencyPolicy:lockScope = \"global\"", "automation/ConcurrencyPolicy:maxConcurrent = 1", "automation/ConcurrencyPolicy:onConflict = \"queue\"", - "automation/ETLDestination:writeMode = \"append\"", - "automation/ETLPipeline:enabled = true", - "automation/ETLPipeline:syncMode = \"full\"", - "automation/ETLTransformation:continueOnError = false", "automation/ExecutionError:retryable = false", "automation/Flow:runAs = \"user\"", "automation/Flow:status = \"draft\"", diff --git a/packages/spec/authorable-defaults/system.json b/packages/spec/authorable-defaults/system.json index 48c313222b..06e9380b13 100644 --- a/packages/spec/authorable-defaults/system.json +++ b/packages/spec/authorable-defaults/system.json @@ -214,14 +214,6 @@ "system/SecurityEventCorrelation:linkAuthToAudit = true", "system/SecurityEventCorrelation:linkEncryptionToAudit = true", "system/SecurityEventCorrelation:linkMaskingToAudit = true", - "system/ServerCapabilities:compression = true", - "system/ServerCapabilities:httpVersions = [\"1.1\"]", - "system/ServerCapabilities:middleware = true", - "system/ServerCapabilities:routeParams = true", - "system/ServerCapabilities:serverPush = false", - "system/ServerCapabilities:sse = false", - "system/ServerCapabilities:streaming = true", - "system/ServerCapabilities:websocket = false", "system/ServerRateLimitConfig:enabled = false", "system/ServerRateLimitConfig:maxRequests = 100", "system/ServerRateLimitConfig:windowMs = 60000", diff --git a/packages/spec/authorable-surface/api.json b/packages/spec/authorable-surface/api.json index 78e361db81..0d5c357a9e 100644 --- a/packages/spec/authorable-surface/api.json +++ b/packages/spec/authorable-surface/api.json @@ -391,11 +391,6 @@ "api/CreateManyDataResponse:object", "api/CreateManyDataResponse:records", "api/CreateRequest:data", - "api/CreateViewRequest:data", - "api/CreateViewRequest:object", - "api/CreateViewResponse:object", - "api/CreateViewResponse:view", - "api/CreateViewResponse:viewId", "api/CrossObjectBatchDroppedFields:fields", "api/CrossObjectBatchDroppedFields:index", "api/CrossObjectBatchDroppedFields:object", @@ -477,11 +472,6 @@ "api/DeleteResponse:id", "api/DeleteResponse:meta", "api/DeleteResponse:success", - "api/DeleteViewRequest:object", - "api/DeleteViewRequest:viewId", - "api/DeleteViewResponse:object", - "api/DeleteViewResponse:success", - "api/DeleteViewResponse:viewId", "api/DeviceRequestResponse:code", "api/DeviceRequestResponse:expiresAt", "api/DeviceRequestResponse:interval", @@ -774,10 +764,6 @@ "api/GetUiViewResponse:name", "api/GetUiViewResponse:object", "api/GetUiViewResponse:protection", - "api/GetViewRequest:object", - "api/GetViewRequest:viewId", - "api/GetViewResponse:object", - "api/GetViewResponse:view", "api/HttpFindQueryParams:count", "api/HttpFindQueryParams:distinct [RETIRED]", "api/HttpFindQueryParams:expand", @@ -964,10 +950,6 @@ "api/ListRunsResponse:error", "api/ListRunsResponse:meta", "api/ListRunsResponse:success", - "api/ListViewsRequest:object", - "api/ListViewsRequest:type", - "api/ListViewsResponse:object", - "api/ListViewsResponse:views", "api/LoginRequest:email", "api/LoginRequest:password", "api/LoginRequest:provider", @@ -1595,12 +1577,6 @@ "api/UpdateNotificationPreferencesRequest:preferences", "api/UpdateNotificationPreferencesResponse:preferences", "api/UpdateRequest:data", - "api/UpdateViewRequest:data", - "api/UpdateViewRequest:object", - "api/UpdateViewRequest:viewId", - "api/UpdateViewResponse:object", - "api/UpdateViewResponse:view", - "api/UpdateViewResponse:viewId", "api/UploadArtifactRequest:artifact", "api/UploadArtifactRequest:releaseNotes", "api/UploadArtifactRequest:sha256", diff --git a/packages/spec/authorable-surface/automation.json b/packages/spec/authorable-surface/automation.json index de6ef76b4c..961510ed77 100644 --- a/packages/spec/authorable-surface/automation.json +++ b/packages/spec/authorable-surface/automation.json @@ -91,41 +91,6 @@ "automation/DeleteRecordConfig:filter", "automation/DeleteRecordConfig:multi", "automation/DeleteRecordConfig:objectName", - "automation/ETLDestination:config", - "automation/ETLDestination:connector", - "automation/ETLDestination:primaryKey", - "automation/ETLDestination:type", - "automation/ETLDestination:writeMode", - "automation/ETLPipeline:description", - "automation/ETLPipeline:destination", - "automation/ETLPipeline:enabled", - "automation/ETLPipeline:label", - "automation/ETLPipeline:metadata", - "automation/ETLPipeline:name", - "automation/ETLPipeline:notifications", - "automation/ETLPipeline:retry", - "automation/ETLPipeline:schedule", - "automation/ETLPipeline:source", - "automation/ETLPipeline:syncMode", - "automation/ETLPipeline:tags", - "automation/ETLPipeline:transformations", - "automation/ETLPipelineRun:completedAt", - "automation/ETLPipelineRun:durationMs", - "automation/ETLPipelineRun:error", - "automation/ETLPipelineRun:id", - "automation/ETLPipelineRun:logs", - "automation/ETLPipelineRun:pipelineName", - "automation/ETLPipelineRun:startedAt", - "automation/ETLPipelineRun:stats", - "automation/ETLPipelineRun:status", - "automation/ETLSource:config", - "automation/ETLSource:connector", - "automation/ETLSource:incremental", - "automation/ETLSource:type", - "automation/ETLTransformation:config", - "automation/ETLTransformation:continueOnError", - "automation/ETLTransformation:name", - "automation/ETLTransformation:type", "automation/ExecutionError:code", "automation/ExecutionError:context", "automation/ExecutionError:executionId", diff --git a/packages/spec/authorable-surface/system.json b/packages/spec/authorable-surface/system.json index 27ad36886f..cf6ac808c1 100644 --- a/packages/spec/authorable-surface/system.json +++ b/packages/spec/authorable-surface/system.json @@ -1027,25 +1027,9 @@ "system/SecurityEventCorrelation:linkAuthToAudit", "system/SecurityEventCorrelation:linkEncryptionToAudit", "system/SecurityEventCorrelation:linkMaskingToAudit", - "system/ServerCapabilities:compression", - "system/ServerCapabilities:httpVersions", - "system/ServerCapabilities:middleware", - "system/ServerCapabilities:routeParams", - "system/ServerCapabilities:serverPush", - "system/ServerCapabilities:sse", - "system/ServerCapabilities:streaming", - "system/ServerCapabilities:websocket", - "system/ServerEvent:data", - "system/ServerEvent:timestamp", - "system/ServerEvent:type", "system/ServerRateLimitConfig:enabled", "system/ServerRateLimitConfig:maxRequests", "system/ServerRateLimitConfig:windowMs", - "system/ServerStatus:connections", - "system/ServerStatus:requests", - "system/ServerStatus:server", - "system/ServerStatus:state", - "system/ServerStatus:uptime", "system/ServiceConfig:id", "system/ServiceConfig:name", "system/ServiceConfig:options", diff --git a/packages/spec/docs/SYNC_ARCHITECTURE.md b/packages/spec/docs/SYNC_ARCHITECTURE.md index bcc0c0bd72..81b7113473 100644 --- a/packages/spec/docs/SYNC_ARCHITECTURE.md +++ b/packages/spec/docs/SYNC_ARCHITECTURE.md @@ -1,18 +1,31 @@ # Data Synchronization Architecture -ObjectStack implements a **2-layer architecture** for data synchronization and integration, designed to serve different audiences and use cases. - -> **History note (v17):** this document used to describe a 3-layer architecture whose -> first layer — **"L1: Simple Sync"** (`DataSyncConfig` in `automation/sync.zod.ts`) — -> was retired in #4738. See [Retired: L1 Simple Sync](#retired-l1-simple-sync-v17) -> for what happened and what to use instead. The historical L2/L3 numbering is kept -> in the level headings so older references stay legible. +ObjectStack has **one** protocol layer for data synchronization and integration: the +Enterprise Connector. This document describes it, and records the two layers that +were removed above it. + +> **History note (v17):** this document used to describe a 3-layer architecture. Both +> of the layers above L3 have since been retired under ADR-0049 enforce-or-remove, for +> the same measured reason — **no engine ever executed either of them**: +> **"L1: Simple Sync"** (`DataSyncConfig`, `automation/sync.zod.ts`) in #4738, and +> **"L2: ETL Pipeline"** (`ETLPipeline`, `automation/etl.zod.ts`) in #6414. See +> [Retired: L1 Simple Sync](#retired-l1-simple-sync-v17) and +> [Retired: L2 ETL Pipeline](#retired-l2-etl-pipeline-v17) for what each declared and +> what to use instead. The historical L3 numbering is kept in the level heading so +> older references stay legible. +> +> ⚠️ **This document was itself part of the L2 defect.** When L1 was retired it sent +> L1's authors on to L2 — a layer with no executor either — and it advertised ten ETL +> transformation types in a table concrete enough to copy from. A document that +> recommends a layer nothing runs is how a `declared ≠ enforced` gap propagates +> instead of closing. It is corrected here, in the same change as the retirement, +> which is why the L2 section below tells you what is gone rather than how to author +> it. ## Overview | Level | Protocol | File | Audience | Use Case | Complexity | |-------|----------|------|----------|----------|------------| -| **L2: ETL Pipeline** | `ETLPipeline` | `automation/etl.zod.ts` | Data engineers | Aggregate 10 sources to data warehouse | ⭐⭐ Moderate | | **L3: Enterprise Connector** | `Connector` | `integration/connector.zod.ts` | System integrators | Full SAP integration with advanced features | ⭐⭐⭐ Advanced | --- @@ -36,8 +49,14 @@ live declarations in `integration/connector.zod.ts` and `ui/offline.zod.ts` (the - **Connector-attached sync** — `ConnectorSchema.syncConfig` (`integration/connector.zod.ts`): the live, parsed sync-strategy surface (strategy, direction, schedule, `conflictResolution`, batching, delete mode). -- **Transformation pipelines** — `ETLPipeline` (`automation/etl.zod.ts`) for - multi-source, multi-stage data movement. +- **Transformation pipelines** — ~~`ETLPipeline` (`automation/etl.zod.ts`) for + multi-source, multi-stage data movement~~ **also retired, at #6414** (ADR-0049), on + the same reading this section applies to L1: zero execution-side consumers, no + `liveness/` ledger row, no engine that ever parsed a pipeline. This bullet is the + reason the L2 retirement had to correct this document rather than only the schema — + it was actively forwarding displaced L1 authors to a second inert layer. There is + no third layer to forward to; see + [Retired: L2 ETL Pipeline](#retired-l2-etl-pipeline-v17). - **Client offline sync** — ~~`SyncConfigSchema` / `ConflictResolution` (`ui/offline.zod.ts`)~~ **also retired, at #4988** (ADR-0049). That vocabulary had no carrier key either: no schema in the protocol declared an `offline:` @@ -49,134 +68,54 @@ live declarations in `integration/connector.zod.ts` and `ui/offline.zod.ts` (the --- -## Level 2: ETL Pipeline - -**File:** `packages/spec/src/automation/etl.zod.ts` -**Audience:** Data engineers, analytics teams -**Complexity:** ⭐⭐ Moderate - -### Purpose - -Advanced data pipelines for complex transformations, multi-source aggregation, and data warehouse population. - -### Key Features - -- ✅ Multi-source, multi-stage pipelines -- ✅ Complex transformations (join, aggregate, filter, custom SQL) -- ✅ Data normalization and deduplication -- ✅ Split/merge operations -- ✅ Incremental extraction with change data capture (CDC) -- ✅ Data quality validation - -### Use Cases - -1. **Data Warehouse Population** - Aggregate data from 10+ sources into Snowflake -2. **Business Intelligence** - Transform operational data for analytics -3. **Data Migration** - Move data from legacy systems to modern platforms -4. **Master Data Management** - Consolidate customer data from multiple systems - -### Example - -> **`ETLPipeline` is the AUTHOR shape.** It is `z.input` of `ETLPipelineSchema` -> (#4963, the house `X` / `XParsed` convention), so every key carrying a -> `.default()` — `syncMode`, `enabled`, `destination.writeMode`, a -> transformation's `continueOnError`, `source.incremental.enabled` — is optional -> when you write a pipeline, and `schedule` takes the bare cron string the -> schema wraps for you. Annotate the **result** of -> `ETLPipelineSchema.parse(…)` with **`ETLPipelineParsed`**, where those same -> keys are all present. The example below states them anyway, because it is a -> tour of the surface; the Migration Guide's examples omit them, because that is -> what ordinary authoring looks like. - -```typescript -import type { ETLPipeline } from '@objectstack/spec/automation'; - -const dataWarehousePipeline: ETLPipeline = { - name: 'customer_360_pipeline', - label: 'Customer 360 Data Warehouse Pipeline', - - // Extract from Salesforce - source: { - type: 'api', - connector: 'salesforce', - config: { - object: 'Account' - }, - incremental: { - enabled: true, - cursorField: 'LastModifiedDate' - } - }, - - // Transform: Join with support tickets, aggregate metrics - transformations: [ - { - type: 'join', - config: { - source: 'zendesk', - joinKey: 'email', - joinType: 'left' - } - }, - { - type: 'aggregate', - config: { - groupBy: ['customer_id'], - metrics: { - total_tickets: 'COUNT(ticket_id)', - avg_satisfaction: 'AVG(satisfaction_score)' - } - } - }, - { - type: 'filter', - config: { - condition: 'annual_revenue > 100000' - } - } - ], - - // Load to Snowflake - destination: { - type: 'warehouse', - connector: 'snowflake', - config: { - database: 'analytics', - schema: 'customer_360', - table: 'customers' - }, - writeMode: 'upsert', - primaryKey: ['customer_id'] - }, - - syncMode: 'incremental', - schedule: '0 2 * * *', // Daily at 2 AM - enabled: true -}; -``` - -### Transformation Types - -| Type | Description | Example | -|------|-------------|---------| -| `map` | Field mapping/renaming | `{ 'old_name': 'new_name' }` | -| `filter` | Row filtering | `status == "active"` | -| `aggregate` | Aggregation/grouping | `SUM(revenue) BY customer_id` | -| `join` | Join with other data | `LEFT JOIN orders ON customer_id` | -| `script` | Custom JavaScript/Python | `return row.price * 1.1` | -| `lookup` | Enrich with reference data | Lookup country from zip code | -| `split` | Split one record into many | Split line items from order | -| `merge` | Merge multiple records | Deduplicate customers | -| `normalize` | Data normalization | Phone number formatting | -| `deduplicate` | Remove duplicates | Based on email | - -### Best Practices - -- Use **incremental sync** with cursor fields for large datasets -- Add **data quality checks** in transformation pipeline -- Monitor **pipeline performance** and optimize slow transformations -- Use **staging tables** for complex multi-stage pipelines -- Configure **alerting** for pipeline failures +## Retired: L2 ETL Pipeline (v17) + +**Removed in:** #6414 (ADR-0049 enforce-or-remove; ADR-0078 no-silently-inert-metadata) +**Was:** `ETLPipeline`, `ETLPipelineRun`, `ETLSource`, `ETLDestination`, +`ETLTransformation`, the `ETLEndpointType` / `ETLTransformationType` / `ETLSyncMode` / +`ETLRunStatus` enums and the `ETL` factory, in +`packages/spec/src/automation/etl.zod.ts` + +L2 was **narrative-only**, on exactly the reading that retired L1 one layer up. No +engine ever parsed, scheduled or executed an `ETLPipeline`. Measured on `origin/main` +immediately before the removal: + +- the only non-spec references in this repo were two fumadocs-generated documentation + sources (`apps/docs/.source/*.ts`) — not executors; +- objectui had no reference at all; +- there was no `packages/spec/liveness/etl.json`, so no ADR-0049 gate ever had a + reading on the surface. The contrast that makes that absence meaningful rather than + an oversight is in the same file family: import mapping's `transform` **is** applied + row by row by the REST import path, and it **does** have a ledger + (`packages/spec/liveness/mapping.json`). + +What an author got was ADR-0078's asymmetry in its purest form: write a complete +ten-stage pipeline, get no error, and get no execution. + +**What to use instead — layer by layer, and one honest gap:** + +- **Scheduled, connector-attached synchronisation** — `ConnectorSchema.syncConfig` + (`integration/connector.zod.ts`), the live, parsed surface described under L3 below: + strategy, direction, cron schedule, `conflictResolution`, batching, delete mode. +- **Per-field value conversion on import** — `mapping.fieldMapping[].transform` + (`data/mapping.zod.ts`): a string enum (`none` / `constant` / `map` / `split` / + `join` / `lookup`) with its settings in `params`, applied row by row by the REST + import path and recorded key by key in `packages/spec/liveness/mapping.json`. +- **Recurring execution** — `system/job.zod.ts`. +- **Multi-source aggregation, joins, custom-SQL stages: nothing.** This is the gap, + stated plainly rather than papered over with a redirect — the mistake this section + replaces. There is no replacement surface because there was never an implementation; + the ten transformation types this document used to tabulate (`map`, `filter`, + `aggregate`, `join`, `script`, `lookup`, `split`, `merge`, `normalize`, + `deduplicate`) named capabilities no runtime had. If multi-stage movement becomes a + real requirement it returns through ADR-0049's **enforce** route — the engine first, + the vocabulary second — not by re-publishing the shape. + +**Already authored a pipeline?** Nothing was deployed under it (that is the finding), +so there is no data migration. `tsc` reports TS2724/TS2305 at every import of a +retired name, and the D3 record is the `etl-pipeline-layer-retired` entry in +`packages/spec/src/migrations/registry.ts`, which `os migrate meta` and the generated +upgrade guide project. --- @@ -227,13 +166,15 @@ Complete, production-grade integration with external systems. Includes authentic > executed any of the five**, and the `javascript` member advertised > `dialect: "js"`, a dialect retired in #3278. An L3 connector mapping moves a > value from `source` to `target`; it does not compute one. **Value conversion -> belongs on a surface that runs it:** the L2 import mapping's own `transform` +> belongs on a surface that runs it:** the import mapping's own `transform` > (`mapping.fieldMapping[].transform` in `data/mapping.zod.ts` — a string enum, > `none`/`constant`/`map`/`split`/`join`/`lookup`, with its settings in `params`), > applied row by row by the REST import path, which rejects its own `javascript` -> value with a 400 rather than pretending to run it — or an ETL transformation -> step (L2 above). Already authored the retired key? `os migrate meta --from 16` -> rewrites it. +> value with a 400 rather than pretending to run it. That is now the ONLY such +> surface: this note used to offer "or an ETL transformation step (L2 above)" as a +> second option, and L2 was retired at #6414 for having no executor — the second +> option was the same defect this note is about, one layer up. Already authored the +> retired key? `os migrate meta --from 16` rewrites it. ### Use Cases @@ -254,9 +195,8 @@ Complete, production-grade integration with external systems. Includes authentic > the schema wraps for you. Annotate the **result** of > `ConnectorSchema.parse(…)` with **`ConnectorParsed`**, which is `z.infer`: > there those keys are all present and `schedule` is already the -> `{ dialect: 'cron', source }` envelope. This matches L2 above, where the bare -> `ETLPipeline` is the author shape and the parse result is `ETLPipelineParsed` -> — the two halves of the spec agree now, and +> `{ dialect: 'cron', source }` envelope. The same convention held on L2's +> `ETLPipeline` / `ETLPipelineParsed` before that layer was retired (#6414), and > **[ADR-0122](../../../docs/adr/0122-schema-type-alias-naming-convention.md) > is why**: the bare name is the author state and `XParsed` is the parsed state, > repo-wide. Earlier revisions of this note called L2's spelling "the house @@ -327,8 +267,10 @@ const sapConnector: Connector = { // nothing anywhere read the transform. The `javascript` member is what // made the gap visible — it recommended the retired `js` dialect // (#3278), so the only spelling that parsed was a bare string, which - // means CEL. Value conversion belongs on a surface that runs it: the L2 - // import mapping's own `transform`, or an ETL transformation step.) + // means CEL. Value conversion belongs on a surface that runs it: the + // import mapping's own `transform` (`data/mapping.zod.ts`). The "or an ETL + // transformation step" this used to add is gone — L2 was retired at #6414 + // for having no executor, which is the very defect this comment is about.) syncMode: 'bidirectional' } ], @@ -403,29 +345,26 @@ const sapConnector: Connector = { ### Decision Matrix -| Question | Answer → Level | -|----------|----------------| -| Do you need to transform values at all — joins and aggregations, or just a per-field convert? | **Yes** → L2 (ETL) for joins/aggregations, or the import mapping's `fieldMapping[].transform` for per-field conversion. **Not** L3: a connector's `fieldMappings` declares `dataType` and `syncMode` and performs no value transformation (#5552) | -| Do you need multi-source aggregation? | **Yes** → L2 (ETL) | +With L1 and L2 both retired there is only one level left to choose, so this matrix now +mostly answers "which surface", and — for the two questions that used to route to L2 — +"none, and here is why". + +| Question | Answer → Surface | +|----------|------------------| +| Do you need to convert a value per field on import? | **Yes** → the import mapping's `fieldMapping[].transform` (`data/mapping.zod.ts`), applied row by row by the REST import path. **Not** L3: a connector's `fieldMappings` declares `dataType` and `syncMode` and performs no value transformation (#5552) | +| Do you need joins, aggregations or custom-SQL stages? | **No surface provides this.** It was L2's headline claim and L2 had no executor (#6414). Do it in the destination system, or in a `flow` / job you write. Do not author a shape hoping it runs | +| Do you need multi-source aggregation? | **Same answer**, and for the same reason — see [Retired: L2 ETL Pipeline](#retired-l2-etl-pipeline-v17) | | Do you need real-time webhooks? | **Yes** → L3 (Connector) | | Do you need advanced authentication (OAuth2, SAML)? | **Yes** → L3 (Connector) | | Do you need retry policies and circuit breaking? | **Yes** → L3 (Connector) — `retryConfig`, `health.circuitBreaker`. Outbound **rate limiting** is not a reason to pick any level: no level provides it (#4911); throttle at the provider or gateway | | Is it a simple point-to-point sync with an external system? | **Yes** → L3 (Connector) with `syncConfig` | -| Are you building a data warehouse pipeline? | **Yes** → L2 (ETL) | +| Are you building a data warehouse pipeline? | The extraction half is L3 (`syncConfig`); the warehouse-side transformation is the warehouse's own tooling. There is no ObjectStack pipeline protocol (#6414) | | Are you integrating with an enterprise system? | **Yes** → L3 (Connector) | -| Do you need client-side offline sync? | **Yes** → `ui/offline.zod.ts` (a separate protocol, not this layering) | +| Do you need client-side offline sync? | Not this layering — and note `ui/offline.zod.ts` was itself retired at #4988 for having no carrier key | ### Common Patterns -#### Pattern 1: Analytics Pipeline (L2) -``` -Salesforce → ETL → Transform → Snowflake -HubSpot ↗ ↘ Analytics Dashboard -Stripe ↗ -``` -Use **L2 ETL Pipeline** for multi-source data warehousing. - -#### Pattern 2: Enterprise Integration (L3) +#### Pattern 1: Enterprise Integration (L3) ``` ObjectStack ↔ Enterprise Connector ↔ SAP ↓ @@ -435,34 +374,28 @@ Use **L3 Enterprise Connector** for production-grade integrations — including straightforward point-to-point sync, via a connector instance with simple `auth` and a `syncConfig`. -#### Pattern 3: Hybrid Approach +#### Pattern 2: Ingest, then transform where it runs ``` -External API → L3 Connector → ObjectStack -ObjectStack → L2 ETL → Data Warehouse +External API → L3 Connector → ObjectStack → (warehouse's own ELT) ``` -Combine levels for complex scenarios. +The second arrow used to read `ObjectStack → L2 ETL → Data Warehouse`, and that hop +never executed. Land the data with a connector, then transform it with a tool that +actually runs — the warehouse's own ELT, a `flow`, or a scheduled job. --- ## Migration Guide -### From L3 (`syncConfig`) to L2 +### From L2 (`ETLPipeline`) to what exists -When a connector's declarative sync needs to transform values — joins and -aggregations, or a per-field convert that `fieldMappings` cannot do (#5552): +L2 was retired at #6414. Nothing was ever deployed under it — that is the finding, not +a consolation — so this is a source edit, not a data migration. -**Before (L3 `syncConfig`):** -```typescript -const connector: Connector = { - name: 'orders', - type: 'saas', - authentication: { type: 'api-key', ... }, - syncConfig: { strategy: 'incremental', direction: 'import' } -}; -``` +**Before** — the retired L2 shape. Shown as plain text, not a `typescript` fence, on +purpose: `ETLPipeline` no longer exists, so this snippet does not compile and must not +be picked up by the documentation compile gate as if it should. -**After (L2):** -```typescript +``` import type { ETLPipeline } from '@objectstack/spec/automation'; const pipeline: ETLPipeline = { @@ -475,46 +408,38 @@ const pipeline: ETLPipeline = { }; ``` -Every endpoint carries a `config` bag — it is the one required key besides -`type`, and it is where endpoint-specific settings (`table`, `endpoint`, `path`, -`format`) live. `syncMode`, `enabled`, `destination.writeMode` and the -transformation's `continueOnError` are omitted on purpose: they have defaults, -and `ETLPipeline` is the author shape. - -### From L2 to L3 +**After** — split it by which half had a runtime. The extraction half does: -When your ETL pipeline needs webhooks, advanced auth, or retry / circuit-breaker -policies: - -**Before (L2):** ```typescript -import type { ETLPipeline } from '@objectstack/spec/automation'; +import type { Connector } from '@objectstack/spec/integration'; -const pipeline: ETLPipeline = { - name: 'external_api_ingest', - source: { type: 'api', connector: 'external_api', config: { endpoint: '/events' } }, - destination: { type: 'database', config: { table: 'external_events' } } +const orders: Connector = { + name: 'orders', + label: 'Orders API', + type: 'saas', + syncConfig: { strategy: 'incremental', direction: 'import' } }; ``` -**After (L3):** -```typescript -const connector: Connector = { - authentication: { type: 'oauth2', ... }, - webhooks: [...], - retryConfig: { ... } -}; -``` +The `transformations` half has no runtime, and never did. Aggregations, joins and +custom-SQL stages belong to whatever actually computes: the destination warehouse's +ELT, a `flow`, or a scheduled job you write. + +Per-field value conversion on import — a cast, a constant, a lookup — is the import +mapping's `fieldMapping[].transform` (`data/mapping.zod.ts`), which is executed. + +### From L3 (`syncConfig`) to a pipeline + +There is no pipeline layer to move up to. This section used to describe exactly that +move — "when a connector's declarative sync needs to transform values … **After (L2)**" +— and the destination did not run. If `syncConfig` plus `fieldMapping[].transform` does +not cover the case, the work belongs outside the sync protocol until an engine exists +to receive it (ADR-0049: enforce, then declare). --- ## API Reference -### Level 2: ETL Pipeline -- [ETLPipeline Schema](../src/automation/etl.zod.ts) -- [ETL Transformations](../src/automation/etl.zod.ts#L151) -- [ETL Run Result](../src/automation/etl.zod.ts#L316) - ### Level 3: Enterprise Connector - [Connector Schema](../src/integration/connector.zod.ts) - [Authentication](../src/auth/config.zod.ts) diff --git a/packages/spec/json-schema.manifest/api.json b/packages/spec/json-schema.manifest/api.json index 800ccac6e7..0dda14515f 100644 --- a/packages/spec/json-schema.manifest/api.json +++ b/packages/spec/json-schema.manifest/api.json @@ -86,8 +86,6 @@ "api/CreateManyDataRequest", "api/CreateManyDataResponse", "api/CreateRequest", - "api/CreateViewRequest", - "api/CreateViewResponse", "api/CrossObjectBatchDroppedFields", "api/CrossObjectBatchOperation", "api/CrossObjectBatchRequest", @@ -111,8 +109,6 @@ "api/DeleteMetaItemRequest", "api/DeleteMetaItemResponse", "api/DeleteResponse", - "api/DeleteViewRequest", - "api/DeleteViewResponse", "api/DeviceRequestResponse", "api/DeviceTokenResponse", "api/DisablePackageRequest", @@ -199,8 +195,6 @@ "api/GetTranslationsResponse", "api/GetUiViewRequest", "api/GetUiViewResponse", - "api/GetViewRequest", - "api/GetViewResponse", "api/HandlerStatus", "api/HttpFindQueryParams", "api/HttpMethod", @@ -240,8 +234,6 @@ "api/ListRecordResponse", "api/ListRunsRequest", "api/ListRunsResponse", - "api/ListViewsRequest", - "api/ListViewsResponse", "api/LoginRequest", "api/LoginType", "api/MarkAllNotificationsReadRequest", @@ -397,8 +389,6 @@ "api/UpdateNotificationPreferencesRequest", "api/UpdateNotificationPreferencesResponse", "api/UpdateRequest", - "api/UpdateViewRequest", - "api/UpdateViewResponse", "api/UploadArtifactRequest", "api/UploadArtifactResponse", "api/UploadChunkRequest", diff --git a/packages/spec/json-schema.manifest/automation.json b/packages/spec/json-schema.manifest/automation.json index 2ac89e9685..d71a3b8ea7 100644 --- a/packages/spec/json-schema.manifest/automation.json +++ b/packages/spec/json-schema.manifest/automation.json @@ -25,15 +25,6 @@ "automation/DecisionConfig", "automation/DecisionOutputDef", "automation/DeleteRecordConfig", - "automation/ETLDestination", - "automation/ETLEndpointType", - "automation/ETLPipeline", - "automation/ETLPipelineRun", - "automation/ETLRunStatus", - "automation/ETLSource", - "automation/ETLSyncMode", - "automation/ETLTransformation", - "automation/ETLTransformationType", "automation/ExecutionError", "automation/ExecutionErrorSeverity", "automation/ExecutionLog", diff --git a/packages/spec/json-schema.manifest/system.json b/packages/spec/json-schema.manifest/system.json index e2834118fc..40c12cf943 100644 --- a/packages/spec/json-schema.manifest/system.json +++ b/packages/spec/json-schema.manifest/system.json @@ -217,11 +217,7 @@ "system/SearchProvider", "system/SecurityContextConfig", "system/SecurityEventCorrelation", - "system/ServerCapabilities", - "system/ServerEvent", - "system/ServerEventType", "system/ServerRateLimitConfig", - "system/ServerStatus", "system/ServiceConfig", "system/ServiceCriticality", "system/ServiceLevelIndicator", diff --git a/packages/spec/scripts/build-docs.ts b/packages/spec/scripts/build-docs.ts index a6a8f1d3e3..97596413c8 100644 --- a/packages/spec/scripts/build-docs.ts +++ b/packages/spec/scripts/build-docs.ts @@ -507,7 +507,11 @@ const SECTION_GROUPS: Record ], automation: [ { section: 'Flow & Execution', pages: ['flow', 'control-flow', 'execution', 'node-executor', 'state-machine', 'time-relative-trigger'] }, - { section: 'Integration & Data', pages: ['sync', 'etl', 'connector', 'webhook', 'bpmn-interop', 'offline'] }, + // `sync` (#4738, L1) and `etl` (#6414, L2) are both retired; `offline` went + // with `ui/offline.zod.ts` (#4988). `buildCategoryPages` filters by what was + // emitted, so leaving a dead name here is silently harmless — which is why + // each is removed deliberately instead. + { section: 'Integration & Data', pages: ['connector', 'webhook', 'bpmn-interop'] }, { section: 'Approvals & Jobs', pages: ['approval', 'job'] }, ], cloud: [ diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index 8a394f9ad0..8af6543845 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -448,13 +448,6 @@ "toMajor": 17, "rationale": "The converged RetryPolicy (#4661) keeps the automation side's bounds, which the job side never had: `maxRetries` is capped at 10 and `backoffMultiplier` floored at 1. Neither has a lossless rewrite. Clamping `maxRetries: 20` to 10 would halve a retry budget its author chose, and a `backoffMultiplier` below 1 describes a delay that SHRINKS on each attempt — retrying a failing dependency ever faster, which is the opposite of backoff and was never a shape the engine meant to offer. Both now fail at parse time with the bound named, rather than being silently reinterpreted. Choosing the replacement count (or accepting the cap) is the author's call." }, - { - "surface": "etlPipeline.retry.maxAttempts (and any count above 10)", - "replacement": "maxRetries, same number — plus an explicit count if you relied on the old default of 3", - "migrationId": "etl-retry-converged-onto-retry-policy", - "toMajor": 17, - "rationale": "An ETL pipeline's `retry` was a THIRD retry vocabulary that #4661's convergence never reached, because that pass was driven by duplicated exported NAMES and this block is an anonymous inline object (#4962). It now carries the shared `RetryPolicySchema` contract, which changes three things with no single lossless rewrite between them. The rename `maxAttempts` → `maxRetries` IS lossless and the tombstone performs it — both keys counted the retries AFTER the initial attempt, so the number does not change, and subtracting one (correct for `integration/connector.zod.ts`'s identically-spelled `RetryConfig.maxAttempts`, which includes the first attempt) would silently run one attempt fewer than asked. What needs a human: the count now DEFAULTS TO 0 instead of 3, so a pipeline that wrote `retry: {}` or omitted the count bought three silent re-runs and now buys none. That is deliberate and the business case is the destination — an ETL destination is a foreign system by definition, and an implicit retry against a non-idempotent one is a duplicate write (a second invoice, a second export, a second webhook). Retrying is now something an author states and thereby claims idempotency for. The shared contract also caps `maxRetries` at 10, which this block never did; clamping a larger budget would silently halve a number its author chose, so it fails at parse with the bound named instead." - }, { "surface": "flow.errorHandling.maxRetries (under strategy: 'retry')", "replacement": "an explicit count >= 1 (e.g. maxRetries: 3), or strategy: 'fail'", @@ -671,6 +664,27 @@ "migrationId": "driver-sql-distinct-bare-filter-typed", "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." + }, + { + "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", + "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", + "migrationId": "http-server-runtime-vocabulary-retired", + "toMajor": 17, + "rationale": "The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so \"zero consumers in this repo\" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295." + }, + { + "surface": "api.listViews / api.getView / api.createView / api.updateView / api.deleteView (the ViewProtocol interface and its ten Request/Response schemas in api/protocol.zod.ts — 10 defs, 25 exported names)", + "replacement": "the two view surfaces that are actually routed. For a view's STORED definition, the generic metadata methods with `type: 'view'` — `getMetaItem` / `getMetaItems` / `saveMetaItem` / `deleteMetaItem`, served at `/api/v1/meta/view/:name`. For the RESOLVED render-time view, `getUiView` (`GetUiViewRequest` / `GetUiViewResponse`), served at `/api/v1/ui/view/:object/:type`. Neither is addressed by a `viewId`, which is the one thing the retired surface offered and the one thing nothing implemented", + "migrationId": "view-management-protocol-retired", + "toMajor": 17, + "rationale": "A complete viewId-addressed CRUD surface — list (with a list/form filter), read, create, patch, delete — with none of the three things a protocol method needs. Measured on origin/main immediately before the removal: no implementation (`packages/metadata-protocol/src/protocol.ts` declares no `listViews` / `getView` / `createView` / `updateView` / `deleteView`; its only view resolver is `getUiView`), no route (`packages/rest/src/rest-server.ts` never mentions `viewId`, so nothing viewId-addressed is reachable over HTTP at all), and no caller (the only `ViewProtocol` mention outside its own file was the services checklist, which already recorded the five as declared-and-unrouted). The look-alike hits a bare-name grep turns up are all different contracts: `metadata-manager.ts`'s `getView(name: string)` is another class, and objectui's `getView(objectName, viewId)` resolves through `client.meta.getItem('view', …)`, i.e. the metadata route. What makes this worth a removal rather than a note is that the cost is already measured. A declared surface that is name-identical and semantics-adjacent to a real one is an attractive nuisance in every grep, and it mis-directed a decision once: #5948's issue body AND its 2026-08-07 maintainer ruling both read `GetViewResponseSchema` (zero implementations) as the contract of `GET /ui/view/:object/:type`, whose declared response is `GetUiViewResponseSchema` — one word apart, 250 lines up. That ruling's reasoning happened to survive the mix-up (\"nobody can consume `{object, view}` successfully today\" was true, though not for the stated reason), which is the luck this removal stops relying on. Route 3: none of the ten was a key on an authorable shape, nothing parsed them, so there is no tombstone and no D2 conversion — RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. If reading and writing ONE view by id becomes a real requirement it returns implementation-first. ADR-0049, ADR-0087, maintainer ruling 2026-08-07, #6239." + }, + { + "surface": "automation.etlPipeline / automation.etlPipelineRun / automation.etlSource / automation.etlDestination / automation.etlTransformation (the whole L2 layer of automation/etl.zod.ts, its four enums and the `ETL` factory — 9 defs, 27 exported names)", + "replacement": "(removed — no protocol surface replaces it, deliberately. Layer by layer: connector-attached synchronisation is `ConnectorSchema.syncConfig` (`integration/connector.zod.ts`), which IS parsed and executed; per-field value transformation on import is `shared/mapping.zod.ts`, whose `transform` is applied row by row by the REST import path and recorded key by key in `packages/spec/liveness/mapping.json`; scheduling is `system/job.zod.ts`. What has NO replacement is multi-source, multi-stage movement with joins and aggregations — because it never had an implementation either. It returns through the ENFORCE route: the engine first, the vocabulary second)", + "migrationId": "etl-pipeline-layer-retired", + "toMajor": 17, + "rationale": "The reading #4738 used to retire L1 `DataSyncConfig`, re-measured one layer up and identical: narrative-only. No engine ever parsed, scheduled or executed an `ETLPipeline`. Measured on origin/main immediately before the removal: the only non-spec references in this repo are two fumadocs-generated documentation sources (`apps/docs/.source/*.ts`), not executors; objectui has no reference at all; there is no `liveness/etl.json` or `pipeline.json`, so no ADR-0049 gate ever had a reading on it — while the same file family's EXECUTED half does have one (`liveness/mapping.json`), which is the contrast that makes the absence meaningful rather than an oversight. The `etl` string in this registry was the one untested link the finding named, and it is not a loader path: it was the id of the #4962 retry-vocabulary entry, absorbed here. The layer was ADR-0078's asymmetry in its purest form — an author could write a complete ten-stage pipeline, get no error, and get no execution. It was also advertised: `packages/spec/docs/SYNC_ARCHITECTURE.md` named `ETLPipeline` as the recommended destination for authors displaced by the L1 retirement (#4738) and listed ten transformation types with copyable examples down to `script | Custom JavaScript/Python`. That document is rewritten in the same change; a retirement whose own doc still recommends the retired layer is self-contradictory, and forwarding L1's authors to a second layer with no executor was the defect compounding rather than closing. ⚠️ `etl-retry-converged-onto-retry-policy` (#4962) is SUBSUMED here, the #4657/#4834/#5055 way: both land in the unreleased protocol 17, so composed, a rename of `retry.maxAttempts` on a shape that does not survive the major has no observable effect — and keeping both would tell an upgrader to rewrite a key on a schema the same upgrade deletes. The `maxAttempts` `retiredKey()` tombstone goes with the shape that carried it, which is strictly stronger than the tombstone: there is no longer a `retry` block to author the key into. Route 3 — no carrier key, no parse site, so no D2 conversion and no tombstone; RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. ADR-0049, ADR-0078, #6414." } ], "removed": [] @@ -1178,13 +1192,6 @@ "toMajor": 17, "rationale": "The converged RetryPolicy (#4661) keeps the automation side's bounds, which the job side never had: `maxRetries` is capped at 10 and `backoffMultiplier` floored at 1. Neither has a lossless rewrite. Clamping `maxRetries: 20` to 10 would halve a retry budget its author chose, and a `backoffMultiplier` below 1 describes a delay that SHRINKS on each attempt — retrying a failing dependency ever faster, which is the opposite of backoff and was never a shape the engine meant to offer. Both now fail at parse time with the bound named, rather than being silently reinterpreted. Choosing the replacement count (or accepting the cap) is the author's call." }, - { - "surface": "etlPipeline.retry.maxAttempts (and any count above 10)", - "replacement": "maxRetries, same number — plus an explicit count if you relied on the old default of 3", - "migrationId": "etl-retry-converged-onto-retry-policy", - "toMajor": 17, - "rationale": "An ETL pipeline's `retry` was a THIRD retry vocabulary that #4661's convergence never reached, because that pass was driven by duplicated exported NAMES and this block is an anonymous inline object (#4962). It now carries the shared `RetryPolicySchema` contract, which changes three things with no single lossless rewrite between them. The rename `maxAttempts` → `maxRetries` IS lossless and the tombstone performs it — both keys counted the retries AFTER the initial attempt, so the number does not change, and subtracting one (correct for `integration/connector.zod.ts`'s identically-spelled `RetryConfig.maxAttempts`, which includes the first attempt) would silently run one attempt fewer than asked. What needs a human: the count now DEFAULTS TO 0 instead of 3, so a pipeline that wrote `retry: {}` or omitted the count bought three silent re-runs and now buys none. That is deliberate and the business case is the destination — an ETL destination is a foreign system by definition, and an implicit retry against a non-idempotent one is a duplicate write (a second invoice, a second export, a second webhook). Retrying is now something an author states and thereby claims idempotency for. The shared contract also caps `maxRetries` at 10, which this block never did; clamping a larger budget would silently halve a number its author chose, so it fails at parse with the bound named instead." - }, { "surface": "flow.errorHandling.maxRetries (under strategy: 'retry')", "replacement": "an explicit count >= 1 (e.g. maxRetries: 3), or strategy: 'fail'", @@ -1401,6 +1408,27 @@ "migrationId": "driver-sql-distinct-bare-filter-typed", "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." + }, + { + "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", + "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", + "migrationId": "http-server-runtime-vocabulary-retired", + "toMajor": 17, + "rationale": "The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so \"zero consumers in this repo\" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295." + }, + { + "surface": "api.listViews / api.getView / api.createView / api.updateView / api.deleteView (the ViewProtocol interface and its ten Request/Response schemas in api/protocol.zod.ts — 10 defs, 25 exported names)", + "replacement": "the two view surfaces that are actually routed. For a view's STORED definition, the generic metadata methods with `type: 'view'` — `getMetaItem` / `getMetaItems` / `saveMetaItem` / `deleteMetaItem`, served at `/api/v1/meta/view/:name`. For the RESOLVED render-time view, `getUiView` (`GetUiViewRequest` / `GetUiViewResponse`), served at `/api/v1/ui/view/:object/:type`. Neither is addressed by a `viewId`, which is the one thing the retired surface offered and the one thing nothing implemented", + "migrationId": "view-management-protocol-retired", + "toMajor": 17, + "rationale": "A complete viewId-addressed CRUD surface — list (with a list/form filter), read, create, patch, delete — with none of the three things a protocol method needs. Measured on origin/main immediately before the removal: no implementation (`packages/metadata-protocol/src/protocol.ts` declares no `listViews` / `getView` / `createView` / `updateView` / `deleteView`; its only view resolver is `getUiView`), no route (`packages/rest/src/rest-server.ts` never mentions `viewId`, so nothing viewId-addressed is reachable over HTTP at all), and no caller (the only `ViewProtocol` mention outside its own file was the services checklist, which already recorded the five as declared-and-unrouted). The look-alike hits a bare-name grep turns up are all different contracts: `metadata-manager.ts`'s `getView(name: string)` is another class, and objectui's `getView(objectName, viewId)` resolves through `client.meta.getItem('view', …)`, i.e. the metadata route. What makes this worth a removal rather than a note is that the cost is already measured. A declared surface that is name-identical and semantics-adjacent to a real one is an attractive nuisance in every grep, and it mis-directed a decision once: #5948's issue body AND its 2026-08-07 maintainer ruling both read `GetViewResponseSchema` (zero implementations) as the contract of `GET /ui/view/:object/:type`, whose declared response is `GetUiViewResponseSchema` — one word apart, 250 lines up. That ruling's reasoning happened to survive the mix-up (\"nobody can consume `{object, view}` successfully today\" was true, though not for the stated reason), which is the luck this removal stops relying on. Route 3: none of the ten was a key on an authorable shape, nothing parsed them, so there is no tombstone and no D2 conversion — RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. If reading and writing ONE view by id becomes a real requirement it returns implementation-first. ADR-0049, ADR-0087, maintainer ruling 2026-08-07, #6239." + }, + { + "surface": "automation.etlPipeline / automation.etlPipelineRun / automation.etlSource / automation.etlDestination / automation.etlTransformation (the whole L2 layer of automation/etl.zod.ts, its four enums and the `ETL` factory — 9 defs, 27 exported names)", + "replacement": "(removed — no protocol surface replaces it, deliberately. Layer by layer: connector-attached synchronisation is `ConnectorSchema.syncConfig` (`integration/connector.zod.ts`), which IS parsed and executed; per-field value transformation on import is `shared/mapping.zod.ts`, whose `transform` is applied row by row by the REST import path and recorded key by key in `packages/spec/liveness/mapping.json`; scheduling is `system/job.zod.ts`. What has NO replacement is multi-source, multi-stage movement with joins and aggregations — because it never had an implementation either. It returns through the ENFORCE route: the engine first, the vocabulary second)", + "migrationId": "etl-pipeline-layer-retired", + "toMajor": 17, + "rationale": "The reading #4738 used to retire L1 `DataSyncConfig`, re-measured one layer up and identical: narrative-only. No engine ever parsed, scheduled or executed an `ETLPipeline`. Measured on origin/main immediately before the removal: the only non-spec references in this repo are two fumadocs-generated documentation sources (`apps/docs/.source/*.ts`), not executors; objectui has no reference at all; there is no `liveness/etl.json` or `pipeline.json`, so no ADR-0049 gate ever had a reading on it — while the same file family's EXECUTED half does have one (`liveness/mapping.json`), which is the contrast that makes the absence meaningful rather than an oversight. The `etl` string in this registry was the one untested link the finding named, and it is not a loader path: it was the id of the #4962 retry-vocabulary entry, absorbed here. The layer was ADR-0078's asymmetry in its purest form — an author could write a complete ten-stage pipeline, get no error, and get no execution. It was also advertised: `packages/spec/docs/SYNC_ARCHITECTURE.md` named `ETLPipeline` as the recommended destination for authors displaced by the L1 retirement (#4738) and listed ten transformation types with copyable examples down to `script | Custom JavaScript/Python`. That document is rewritten in the same change; a retirement whose own doc still recommends the retired layer is self-contradictory, and forwarding L1's authors to a second layer with no executor was the defect compounding rather than closing. ⚠️ `etl-retry-converged-onto-retry-policy` (#4962) is SUBSUMED here, the #4657/#4834/#5055 way: both land in the unreleased protocol 17, so composed, a rename of `retry.maxAttempts` on a shape that does not survive the major has no observable effect — and keeping both would tell an upgrader to rewrite a key on a schema the same upgrade deletes. The `maxAttempts` `retiredKey()` tombstone goes with the shape that carried it, which is strictly stronger than the tombstone: there is no longer a `retry` block to author the key into. Route 3 — no carrier key, no parse site, so no D2 conversion and no tombstone; RETIRED_DEFS_BY_MAJOR plus this entry are the declaration. ADR-0049, ADR-0078, #6414." } ], "removed": [] diff --git a/packages/spec/src/api/protocol.test.ts b/packages/spec/src/api/protocol.test.ts index 774aae5689..921dd1d5ce 100644 --- a/packages/spec/src/api/protocol.test.ts +++ b/packages/spec/src/api/protocol.test.ts @@ -14,14 +14,7 @@ import { CreateManyDataResponseSchema, UpdateManyDataRequestSchema, DeleteManyDataRequestSchema, - // Views - ListViewsRequestSchema, - ListViewsResponseSchema, - GetViewRequestSchema, - CreateViewRequestSchema, - UpdateViewRequestSchema, - DeleteViewRequestSchema, - DeleteViewResponseSchema, + // View-management schemas removed with the retired ViewProtocol (#6239, v17) // Permissions CheckPermissionRequestSchema, CheckPermissionResponseSchema, @@ -166,24 +159,41 @@ describe('ObjectStack Protocol', () => { expect(DeleteManyDataRequestSchema.safeParse(deleteManyReq).success).toBe(true); }); - it('validates Views operations', () => { - expect(ListViewsRequestSchema.safeParse({ object: 'project', type: 'list' }).success).toBe(true); - expect(ListViewsResponseSchema.safeParse({ - object: 'project', - views: [{ list: { columns: [] } }], - }).success).toBe(true); - expect(GetViewRequestSchema.safeParse({ object: 'project', viewId: 'v1' }).success).toBe(true); - expect(CreateViewRequestSchema.safeParse({ - object: 'project', - data: { list: { columns: [] } }, - }).success).toBe(true); - expect(UpdateViewRequestSchema.safeParse({ - object: 'project', - viewId: 'v1', - data: { list: { columns: [] } }, - }).success).toBe(true); - expect(DeleteViewRequestSchema.safeParse({ object: 'project', viewId: 'v1' }).success).toBe(true); - expect(DeleteViewResponseSchema.safeParse({ object: 'project', viewId: 'v1', success: true }).success).toBe(true); + it('no longer publishes a viewId-addressed view CRUD surface (#6239)', async () => { + // Retired at protocol 17: five methods, ten schemas, zero implementations + // and zero routes — and already mis-read once as the contract of + // `GET /ui/view/:object/:type` (#5948). The unit test that stood here + // asserted the ten shapes parse; the shapes are what was removed, so it is + // replaced wholesale rather than re-spelled (the retirement playbook's + // third fixture disposition — an assertion that keeps passing because + // nothing is produced is not coverage). + // + // Reverse verification, direction predicted first: these are + // `false`-expecting existence checks over a runtime namespace, so restoring + // any removed limb turns exactly this test red. Plain red, not one of the + // inverted directions — nothing downstream counts these names. + const protocol = await import('./protocol.zod'); + for (const name of [ + 'ListViewsRequestSchema', 'ListViewsResponseSchema', + 'GetViewRequestSchema', 'GetViewResponseSchema', + 'CreateViewRequestSchema', 'CreateViewResponseSchema', + 'UpdateViewRequestSchema', 'UpdateViewResponseSchema', + 'DeleteViewRequestSchema', 'DeleteViewResponseSchema', + ]) { + expect(name in protocol, `${name} must not be exported`).toBe(false); + } + }); + + it('keeps the two view surfaces that ARE routed', async () => { + // The point of the removal, asserted positively so a later reader cannot + // mistake it for "views left the protocol". `getUiView` serves the resolved + // render-time view; the stored definition travels on the generic metadata + // methods with `type: 'view'`. + const protocol = await import('./protocol.zod'); + expect('GetUiViewRequestSchema' in protocol).toBe(true); + expect('GetUiViewResponseSchema' in protocol).toBe(true); + expect('GetMetaItemRequestSchema' in protocol).toBe(true); + expect('SaveMetaItemRequestSchema' in protocol).toBe(true); }); it('validates Permissions operations', () => { diff --git a/packages/spec/src/api/protocol.zod.ts b/packages/spec/src/api/protocol.zod.ts index d29a0179ac..805fbeafc3 100644 --- a/packages/spec/src/api/protocol.zod.ts +++ b/packages/spec/src/api/protocol.zod.ts @@ -746,62 +746,53 @@ export { }; // ========================================== -// View Management Operations +// View Management Operations — RETIRED // ========================================== -export const ListViewsRequestSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name (snake_case)'), - type: z.enum(['list', 'form']).optional().describe('Filter by view type'), -})); - -export const ListViewsResponseSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name'), - views: z.array(ViewSchema).describe('Array of view definitions'), -})); - -export const GetViewRequestSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name (snake_case)'), - viewId: z.string().describe('View identifier'), -})); - -export const GetViewResponseSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name'), - view: ViewSchema.describe('View definition'), -})); - -export const CreateViewRequestSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name (snake_case)'), - data: ViewSchema.describe('View definition to create'), -})); - -export const CreateViewResponseSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name'), - viewId: z.string().describe('Created view identifier'), - view: ViewSchema.describe('Created view definition'), -})); - -export const UpdateViewRequestSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name (snake_case)'), - viewId: z.string().describe('View identifier'), - data: ViewSchema.partial().describe('Partial view data to update'), -})); - -export const UpdateViewResponseSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name'), - viewId: z.string().describe('Updated view identifier'), - view: ViewSchema.describe('Updated view definition'), -})); - -export const DeleteViewRequestSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name (snake_case)'), - viewId: z.string().describe('View identifier to delete'), -})); - -export const DeleteViewResponseSchema = lazySchema(() => z.object({ - object: z.string().describe('Object name'), - viewId: z.string().describe('Deleted view identifier'), - success: z.boolean().describe('Whether deletion succeeded'), -})); +// `ListViews` / `GetView` / `CreateView` / `UpdateView` / `DeleteView` — +// five methods and their ten Request/Response schemas — were REMOVED per +// ADR-0049 enforce-or-remove (#6239, protocol 17, maintainer ruling +// 2026-08-07). Route 3 of the retirement playbook: not one of the ten was a +// KEY on an authorable shape, nothing parsed them, and no route could reach the +// methods, so there is no tombstone to write and no source or `sys_metadata` +// row for a D2 conversion to rewrite. `RETIRED_DEFS_BY_MAJOR[17]` plus the D3 +// `SemanticMigration` `view-management-protocol-retired` ARE the declaration. +// +// ## What it declared, and what serves it +// +// A viewId-addressed CRUD surface over views: list by object (+ optional +// list/form filter), read one, create, patch, delete. Measured on `origin/main` +// immediately before the removal, it had ZERO of the three things a protocol +// method needs: +// +// - no implementation — `packages/metadata-protocol/src/protocol.ts` declares +// no `listViews`/`getView`/`createView`/`updateView`/`deleteView`; its only +// view resolver is `getUiView`; +// - no route — `packages/rest/src/rest-server.ts` never mentions `viewId`, so +// nothing viewId-addressed is reachable over HTTP at all; +// - no caller — the only `ViewProtocol` mention outside this file was +// `content/docs/kernel/services-checklist.mdx`, which already recorded the +// five as declared-and-unrouted. +// +// Views are read and written today through surfaces that DO exist: the +// metadata routes (`/api/v1/meta/view/:name`, i.e. `getMetaItem` / +// `saveMetaItem` / `deleteMetaItem` with `type: 'view'`) for the stored +// definition, and `getUiView` (`GET /api/v1/ui/view/:object/:type`) for the +// resolved render-time view. +// +// ## Why this one was worth the removal rather than a note +// +// A declared surface that is name-identical and semantics-adjacent to a real +// one is not merely dead weight; it is an attractive nuisance in every grep. +// It had already cost once: #5948's issue body AND its 2026-08-07 maintainer +// ruling both read `GetViewResponseSchema` (this block, zero implementations) +// as the contract of `GET /ui/view/:object/:type`, whose declared response is +// `GetUiViewResponseSchema` — 250 lines up, one word different. The ruling's +// reasoning happened to survive the mix-up, which is the luck that makes this +// class of defect worth removing rather than annotating. +// +// If "read/write ONE view by id" becomes a real requirement, it returns through +// the ENFORCE route — implementation first, vocabulary second (ADR-0049). // ========================================== // Permission Operations @@ -1423,30 +1414,6 @@ export type DeleteManyDataResponse = z.input; -// View Management Types -export type ListViewsRequest = z.input; -export type ListViewsResponse = z.input; -/** Post-parse shape of {@link ListViewsResponse} — defaults applied, transforms run (ADR-0122). */ -export type ListViewsResponseParsed = z.infer; -export type GetViewRequest = z.input; -export type GetViewResponse = z.input; -/** Post-parse shape of {@link GetViewResponse} — defaults applied, transforms run (ADR-0122). */ -export type GetViewResponseParsed = z.infer; -export type CreateViewRequest = z.input; -/** Post-parse shape of {@link CreateViewRequest} — defaults applied, transforms run (ADR-0122). */ -export type CreateViewRequestParsed = z.infer; -export type CreateViewResponse = z.input; -/** Post-parse shape of {@link CreateViewResponse} — defaults applied, transforms run (ADR-0122). */ -export type CreateViewResponseParsed = z.infer; -export type UpdateViewRequest = z.input; -/** Post-parse shape of {@link UpdateViewRequest} — defaults applied, transforms run (ADR-0122). */ -export type UpdateViewRequestParsed = z.infer; -export type UpdateViewResponse = z.input; -/** Post-parse shape of {@link UpdateViewResponse} — defaults applied, transforms run (ADR-0122). */ -export type UpdateViewResponseParsed = z.infer; -export type DeleteViewRequest = z.input; -export type DeleteViewResponse = z.input; - // Permission Types export type CheckPermissionRequest = z.input; export type CheckPermissionResponse = z.input; @@ -1652,14 +1619,11 @@ export interface PackageProtocol { disablePackage?(request: DisablePackageRequest): Promise; } -/** View management (optional). */ -export interface ViewProtocol { - listViews?(request: ListViewsRequest): Promise; - getView?(request: GetViewRequest): Promise; - createView?(request: CreateViewRequest): Promise; - updateView?(request: UpdateViewRequest): Promise; - deleteView?(request: DeleteViewRequest): Promise; -} +// `ViewProtocol` (`listViews` / `getView` / `createView` / `updateView` / +// `deleteView`) was REMOVED at protocol 17 — see the "View Management +// Operations — RETIRED" note above (#6239). No host implemented it and no route +// reached it; view read/write is `MetadataProtocol` (`getMetaItem` / +// `saveMetaItem` / `deleteMetaItem` with `type: 'view'`) plus `getUiView`. /** Permissions (optional). */ export interface PermissionProtocol { diff --git a/packages/spec/src/automation/etl-author-shape.test.ts b/packages/spec/src/automation/etl-author-shape.test.ts deleted file mode 100644 index 0bfd75e64c..0000000000 --- a/packages/spec/src/automation/etl-author-shape.test.ts +++ /dev/null @@ -1,381 +0,0 @@ -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. - -import { describe, it, expect } from 'vitest'; -import ts from 'typescript'; -import { readFileSync } from 'node:fs'; -import { dirname, resolve } from 'node:path'; -import { fileURLToPath } from 'node:url'; - -import { ETL, ETLPipelineSchema } from './etl.zod'; - -// ─── [#4963] `ETLPipeline` is the AUTHOR shape — proved with the compiler ─── -// -// Until 17 all nine `etl.zod.ts` type aliases were `z.infer` under the bare -// name with no `*Parsed` counterpart, against the house convention (bare name = -// `z.input` = what an author writes; `XParsed` = `z.infer` = what a parse -// returns — written up on `shared/retry-policy.zod.ts`, followed by every -// sibling automation config). On this file that was not cosmetic: six defaulted -// keys and a transform-typed `schedule` were all REQUIRED under `z.infer`, so -// `const p: ETLPipeline = { … }` — the file's only authoring door, there being -// no parse site in objectstack / objectui / cloud — did not compile. -// `SYNC_ARCHITECTURE.md` carried three examples that proved it. -// -// ## Why this file uses the compiler API instead of type-level pins -// -// #4642 established that a conditional-type pin in a `packages/spec` test is a -// NO-OP: `tsconfig.json` excludes `**/*.test.ts` (the package's measured -// `TEST_DEBT` entry in `scripts/check-type-check-coverage.mjs`) and vitest never -// enables `typecheck`. A `@ts-expect-error` or `expectTypeOf` written here would -// be read by nothing. So the pins below drive `ts.createProgram` themselves and -// assert on real diagnostics, the way `sync-retirement.test.ts` does — and, like -// it, they carry anti-vacuity guards, because a harness that resolves nothing -// reports zero errors and looks exactly like success. - -const SPEC_DIR = resolve(dirname(fileURLToPath(import.meta.url)), '../..'); -const SYNC_ARCHITECTURE = resolve(SPEC_DIR, 'docs/SYNC_ARCHITECTURE.md'); - -/** - * Compile a set of probe files against this package's real source and return - * each one's diagnostics, keyed by probe name. - * - * `@objectstack/spec/` is mapped through `paths` to the entry barrel in - * `src/`, which is what lets a documentation snippet be compiled VERBATIM — - * import line included — rather than rewritten into a relative import that no - * reader of the docs would ever type. - * - * `noUnusedLocals` is deliberately off: a documentation snippet declares a - * `const` and stops, and TS6133 is a lint opinion about the snippet's framing, - * not a statement about whether the pipeline literal is well-typed. Everything - * else runs at the repo's real strictness (`strict: true`). - */ -function compileProbes(probes: Readonly>): Map { - const dir = resolve(SPEC_DIR, 'src/__etl_author_shape_probes__'); - const paths = new Map(); - for (const [name, text] of Object.entries(probes)) paths.set(resolve(dir, `${name}.ts`), text); - - const options: ts.CompilerOptions = { - target: ts.ScriptTarget.ES2020, - module: ts.ModuleKind.ESNext, - moduleResolution: ts.ModuleResolutionKind.Bundler, - strict: true, - skipLibCheck: true, - noEmit: true, - noUnusedLocals: false, - noUnusedParameters: false, - baseUrl: SPEC_DIR, - paths: { '@objectstack/spec/*': [resolve(SPEC_DIR, 'src/*/index.ts')] }, - }; - - const host = ts.createCompilerHost(options, true); - const realGetSourceFile = host.getSourceFile.bind(host); - const realFileExists = host.fileExists.bind(host); - const realReadFile = host.readFile.bind(host); - host.getSourceFile = (fileName, languageVersion, onError, shouldCreate) => { - const overlay = paths.get(resolve(fileName)); - return overlay === undefined - ? realGetSourceFile(fileName, languageVersion, onError, shouldCreate) - : ts.createSourceFile(fileName, overlay, languageVersion, true); - }; - host.fileExists = (fileName) => paths.has(resolve(fileName)) || realFileExists(fileName); - host.readFile = (fileName) => paths.get(resolve(fileName)) ?? realReadFile(fileName); - - const program = ts.createProgram([...paths.keys()], options, host); - const out = new Map(); - for (const name of Object.keys(probes)) out.set(name, []); - for (const d of ts.getPreEmitDiagnostics(program)) { - const file = d.file?.fileName ? resolve(d.file.fileName) : undefined; - for (const [name] of Object.entries(probes)) { - if (file === resolve(dir, `${name}.ts`)) out.get(name)!.push(d); - } - } - return out; -} - -/** One diagnostic per line, `TS: `, for readable assertions. */ -function render(diagnostics: readonly ts.Diagnostic[]): string { - return diagnostics - .map((d) => `TS${d.code}: ${ts.flattenDiagnosticMessageText(d.messageText, ' ')}`) - .join('\n'); -} - -/** Every ```typescript fence in a markdown file. */ -function typescriptBlocks(markdown: string): string[] { - return [...markdown.matchAll(/```typescript\r?\n([\s\S]*?)```/g)].map((m) => m[1]); -} - -describe('[#4963] SYNC_ARCHITECTURE.md pipeline examples compile', () => { - const markdown = readFileSync(SYNC_ARCHITECTURE, 'utf8'); - const blocks = typescriptBlocks(markdown); - const pipelineBlocks = blocks.filter((b) => b.includes('ETLPipeline')); - - it('finds the examples this gate exists for, and counts the ones it skips', () => { - // Anti-vacuity: a selector that matched nothing would make the compile - // assertion below pass over an empty program — the way a gate goes dormant. - expect(pipelineBlocks.length, 'ETLPipeline examples in SYNC_ARCHITECTURE.md').toBe(3); - // The other three are the L3 `Connector` examples, out of this gate's scope - // because they belong to `integration/connector.zod.ts` — and covered, since - // #5515, by that file's own gate: `integration/connector-author-shape.test.ts` - // classifies the same three (two Migration-Guide sketches that elide with a - // bare `...`, which is not TypeScript; one full `sapConnector` example) and - // compiles the third. When this pin was written that example reported four - // diagnostics, three of them keys or values the schema REJECTS (`sourceField` - // / `targetField` for `source` / `target`, `transform.type: 'custom'` — - // which #5552 then made moot by retiring `transform` outright — - // `webhooks[].retryPolicy`); those are fixed in the document. The fourth was - // `Connector` being `z.infer` — this issue's twin on a file whose migration - // surface is NOT empty — and it is solved there by annotating the example - // with `ConnectorInput`; the alias flip itself is still open as #5551. - // The total is pinned rather than left open so that ADDING a block to this - // document is a decision someone has to make on purpose: a new ETL example - // is picked up automatically by the selector above, and anything else - // turns this red until it is classified here. - expect(blocks.length, 'total ```typescript blocks — classify any new one').toBe(6); - }); - - it('compiles all three verbatim, import line included, with zero diagnostics', () => { - const probes: Record = {}; - pipelineBlocks.forEach((block, i) => { probes[`doc-example-${i}`] = block; }); - // The harness's own control: a probe that MUST fail. Without it, a - // resolution failure (paths mapping wrong, host overlay not applied) would - // report zero diagnostics for every block and read as three green examples. - probes['harness-self-test'] = [ - "import type { ETLPipeline } from '@objectstack/spec/automation';", - "const broken: ETLPipeline = { name: 'no_source_no_destination' };", - ].join('\n'); - - const results = compileProbes(probes); - expect(render(results.get('harness-self-test')!), 'the harness must be able to report an error') - .toContain('TS2739'); - - for (const [name, diagnostics] of results) { - if (name === 'harness-self-test') continue; - expect(render(diagnostics), `${name} must compile clean`).toBe(''); - } - }); -}); - -describe('[#4963] the bare name is the author shape; `*Parsed` is the parse result', () => { - /** - * The load-bearing pin, written as ONE program so the positive and the - * negative share a document: the same literal, the same keys, only the - * annotation differs. A red `parsed-*` probe therefore cannot be a literal - * that was wrong for an unrelated reason — its `author-*` twin just compiled. - */ - const probes = { - // ── author shape: every defaulted key omitted, cron written bare ── - 'author-pipeline': ` - import type { ETLPipeline } from '@objectstack/spec/automation'; - const p: ETLPipeline = { - name: 'customer_360', - source: { type: 'api', connector: 'salesforce', config: { object: 'Account' } }, - destination: { type: 'warehouse', config: { table: 'customers' } }, - transformations: [{ type: 'filter', config: { condition: 'active' } }], - schedule: '0 2 * * *', - }; - `, - 'author-source': ` - import type { ETLSource } from '@objectstack/spec/automation'; - const s: ETLSource = { type: 'api', config: {}, incremental: { cursorField: 'updated_at' } }; - `, - 'author-destination': ` - import type { ETLDestination } from '@objectstack/spec/automation'; - const d: ETLDestination = { type: 'database', config: { table: 't' } }; - `, - 'author-transformation': ` - import type { ETLTransformation } from '@objectstack/spec/automation'; - const t: ETLTransformation = { type: 'map', config: {} }; - `, - 'author-run': ` - import type { ETLPipelineRun } from '@objectstack/spec/automation'; - const r: ETLPipelineRun = { - id: 'run-1', pipelineName: 'customer_360', status: 'succeeded', - startedAt: '2024-01-01T02:00:00Z', stats: {}, - }; - `, - - // ── parsed shape: the SAME literals, which must now be rejected ── - 'parsed-pipeline': ` - import type { ETLPipelineParsed } from '@objectstack/spec/automation'; - const p: ETLPipelineParsed = { - name: 'customer_360', - source: { type: 'api', connector: 'salesforce', config: { object: 'Account' } }, - destination: { type: 'warehouse', config: { table: 'customers' } }, - transformations: [{ type: 'filter', config: { condition: 'active' } }], - schedule: '0 2 * * *', - }; - `, - // Everything a parse WOULD have filled in is present except `syncMode` and - // `enabled`, so the top-level omission is the only thing left to report. - // The probe above cannot say this on its own: TypeScript reports the - // deepest mismatch it finds and stops, so with `writeMode` / - // `continueOnError` / the cron envelope also missing, the two top-level - // keys never appear in its message at all. - 'parsed-pipeline-top-level-only': ` - import type { ETLPipelineParsed } from '@objectstack/spec/automation'; - const p: ETLPipelineParsed = { - name: 'customer_360', - source: { - type: 'api', connector: 'salesforce', config: { object: 'Account' }, - incremental: { enabled: true, cursorField: 'updated_at' }, - }, - destination: { type: 'warehouse', config: { table: 'customers' }, writeMode: 'upsert' }, - transformations: [{ type: 'filter', config: { condition: 'active' }, continueOnError: false }], - schedule: { dialect: 'cron', source: '0 2 * * *' }, - }; - `, - 'parsed-destination': ` - import type { ETLDestinationParsed } from '@objectstack/spec/automation'; - const d: ETLDestinationParsed = { type: 'database', config: { table: 't' } }; - `, - 'parsed-transformation': ` - import type { ETLTransformationParsed } from '@objectstack/spec/automation'; - const t: ETLTransformationParsed = { type: 'map', config: {} }; - `, - 'parsed-run': ` - import type { ETLPipelineRunParsed } from '@objectstack/spec/automation'; - const r: ETLPipelineRunParsed = { - id: 'run-1', pipelineName: 'customer_360', status: 'succeeded', - startedAt: '2024-01-01T02:00:00Z', stats: {}, - }; - `, - } as const; - - const results = compileProbes(probes); - - it.each(['author-pipeline', 'author-source', 'author-destination', 'author-transformation', 'author-run'])( - '%s compiles with the defaulted keys left out', - (name) => { - expect(render(results.get(name)!)).toBe(''); - }, - ); - - it('rejects the same pipeline literal under `ETLPipelineParsed`', () => { - // The direction stated before it was run: under the PARSED alias the - // defaulted keys are facts about a document that has already been through - // `.parse()`, so omitting them is an error. Each missing key is named - // rather than asserting "some diagnostic", because a bare non-empty check - // would still pass if the alias were quietly pointed back at `z.input` and - // the literal broke for an unrelated reason. - const message = render(results.get('parsed-pipeline')!); - expect(message).toContain('writeMode'); - expect(message).toContain('continueOnError'); - // The transform half: a bare cron string is the input, never the output. - expect(message).toContain("Type 'string' is not assignable"); - }); - - it('requires `syncMode` and `enabled` on `ETLPipelineParsed` — the top-level defaults', () => { - const message = render(results.get('parsed-pipeline-top-level-only')!); - expect(message).toContain('TS2739'); - expect(message).toContain('syncMode'); - expect(message).toContain('enabled'); - // …and the author-shape twin of this exact literal is clean, which is what - // makes the red above a statement about the ANNOTATION and nothing else. - expect(render(results.get('author-pipeline')!)).toBe(''); - }); - - it('rejects the same destination / transformation / run literals under `*Parsed`', () => { - expect(render(results.get('parsed-destination')!)).toContain('writeMode'); - expect(render(results.get('parsed-transformation')!)).toContain('continueOnError'); - // `stats: {}` is complete under the input shape and missing four counters - // under the parsed one — the wire half follows the same rule as the seven - // authoring shapes, deliberately (see the alias note in `etl.zod.ts`). - expect(render(results.get('parsed-run')!)).toContain('recordsRead'); - }); - - it('accepts a bare cron string only on the author side', () => { - // `schedule` is the second half of why the old aliases did not compile: - // `CronExpressionInputSchema` is a transform, so `z.infer` is the - // `{ dialect, source }` envelope and the string every doc example writes - // was rejected. This is asserted through the pipeline probes above rather - // than a fourth pair, because `schedule` has no standalone ETL alias. - expect(render(results.get('author-pipeline')!)).toBe(''); - expect(render(results.get('parsed-pipeline')!)).toContain("Type 'string' is not assignable"); - }); -}); - -describe('[#4963] all nine aliases carry the pair', () => { - const NAMES = [ - 'ETLEndpointType', 'ETLSource', 'ETLDestination', 'ETLTransformationType', - 'ETLTransformation', 'ETLSyncMode', 'ETLPipeline', 'ETLRunStatus', 'ETLPipelineRun', - ] as const; - - it('exports `X` and `XParsed` from `@objectstack/spec/automation` for every one of them', () => { - const program = ts.createProgram([resolve(SPEC_DIR, 'src/automation/index.ts')], { - module: ts.ModuleKind.ESNext, - moduleResolution: ts.ModuleResolutionKind.Bundler, - skipLibCheck: true, - noEmit: true, - }); - const checker = program.getTypeChecker(); - const sf = program.getSourceFile(resolve(SPEC_DIR, 'src/automation/index.ts')); - const moduleSym = sf && checker.getSymbolAtLocation(sf); - // Without this the `toContain`s below would assert over an empty list. - expect(moduleSym, 'the ./automation barrel must resolve').toBeTruthy(); - const exported = checker.getExportsOfModule(moduleSym!).map((s) => s.getName()); - expect(exported.length).toBeGreaterThan(50); - - for (const name of NAMES) { - expect(exported, `${name} must still be exported`).toContain(name); - expect(exported, `${name}Parsed must exist — the pair is the convention`).toContain(`${name}Parsed`); - } - }); - - it('leaves the four enum aliases mutually assignable — the pair is a deliberate synonym', () => { - // Stated as a compile probe rather than a comment so that an enum which - // later gains a `.transform()` or `.catch()` turns this red instead of - // silently making `X` and `XParsed` disagree behind four identical-looking - // declarations. - const enums = ['ETLEndpointType', 'ETLTransformationType', 'ETLSyncMode', 'ETLRunStatus']; - const probes: Record = {}; - for (const name of enums) { - probes[`enum-${name}`] = [ - `import type { ${name}, ${name}Parsed } from '@objectstack/spec/automation';`, - `declare const a: ${name}; declare const b: ${name}Parsed;`, - `const toParsed: ${name}Parsed = a; const toInput: ${name} = b;`, - 'void toParsed; void toInput;', - ].join('\n'); - } - const results = compileProbes(probes); - for (const [name, diagnostics] of results) { - expect(render(diagnostics), `${name} must be assignable in both directions`).toBe(''); - } - }); -}); - -describe('[#4963] the ETL factories stopped working around their own return type', () => { - it('passes a bare cron string straight through instead of pre-wrapping it', () => { - // Pre-17 the helpers normalized `'0 * * * *'` into `{ dialect: 'cron', - // source }` because `z.infer` of `CronExpressionInputSchema` is the - // post-transform envelope and would not accept the string. Returning the - // AUTHOR shape moves that normalization back to where it belongs: the parse. - const pipeline = ETL.databaseSync({ - name: 'users_sync', sourceTable: 'src', destTable: 'dst', schedule: '0 * * * *', - }); - expect(pipeline.schedule).toBe('0 * * * *'); - - const parsed = ETLPipelineSchema.parse(pipeline); - expect(parsed.schedule).toEqual({ dialect: 'cron', source: '0 * * * *' }); - }); - - it('no longer spells out `enabled`, and the parse still supplies it', () => { - // `enabled: true` restated the schema's own default and existed only - // because `z.infer` made the key required. Dropping it must not change what - // a parsed pipeline says — that is the whole claim of the flip. - const pipeline = ETL.apiToDatabase({ name: 'api_ingest', apiConnector: 'stripe', destTable: 'payments' }); - expect(pipeline).not.toHaveProperty('enabled'); - expect(ETLPipelineSchema.parse(pipeline).enabled).toBe(true); - }); - - it('still states what each helper DECIDES, not what the schema defaults to', () => { - // The keys that survived are the ones carrying intent: the two helpers are - // a contrast (incremental+upsert vs full+append) and a reader must see it - // without looking up two defaults. - const sync = ETL.databaseSync({ name: 'users_sync', sourceTable: 'src', destTable: 'dst' }); - expect(sync.syncMode).toBe('incremental'); - expect(sync.destination.writeMode).toBe('upsert'); - const ingest = ETL.apiToDatabase({ name: 'api_ingest', apiConnector: 'stripe', destTable: 'payments' }); - expect(ingest.syncMode).toBe('full'); - expect(ingest.destination.writeMode).toBe('append'); - for (const p of [sync, ingest]) expect(ETLPipelineSchema.safeParse(p).success).toBe(true); - }); -}); diff --git a/packages/spec/src/automation/etl.test.ts b/packages/spec/src/automation/etl.test.ts deleted file mode 100644 index 8567bbb5ac..0000000000 --- a/packages/spec/src/automation/etl.test.ts +++ /dev/null @@ -1,651 +0,0 @@ -import { describe, it, expect } from 'vitest'; -import { - ETLEndpointTypeSchema, - ETLSourceSchema, - ETLDestinationSchema, - ETLTransformationTypeSchema, - ETLTransformationSchema, - ETLSyncModeSchema, - ETLPipelineSchema, - ETLRunStatusSchema, - ETLPipelineRunSchema, - ETL, -} from './etl.zod'; - -describe('ETLEndpointTypeSchema', () => { - it('should accept all valid endpoint types', () => { - const types = [ - 'database', 'api', 'file', 'stream', 'object', - 'warehouse', 'storage', 'spreadsheet', - ]; - types.forEach(t => { - expect(() => ETLEndpointTypeSchema.parse(t)).not.toThrow(); - }); - }); - - it('should reject invalid endpoint type', () => { - expect(() => ETLEndpointTypeSchema.parse('ftp')).toThrow(); - }); -}); - -describe('ETLSourceSchema', () => { - it('should accept minimal source', () => { - expect(() => ETLSourceSchema.parse({ - type: 'database', - config: { table: 'users' }, - })).not.toThrow(); - }); - - it('should accept full source with incremental config', () => { - expect(() => ETLSourceSchema.parse({ - type: 'api', - connector: 'salesforce', - config: { object: 'Account' }, - incremental: { - enabled: true, - cursorField: 'updated_at', - cursorValue: '2024-01-01T00:00:00Z', - }, - })).not.toThrow(); - }); - - it('should reject missing config', () => { - expect(() => ETLSourceSchema.parse({ - type: 'database', - })).toThrow(); - }); - - it('should reject invalid type', () => { - expect(() => ETLSourceSchema.parse({ - type: 'invalid', - config: {}, - })).toThrow(); - }); -}); - -describe('ETLDestinationSchema', () => { - it('should accept minimal destination with defaults', () => { - const result = ETLDestinationSchema.parse({ - type: 'database', - config: { table: 'accounts' }, - }); - expect(result.writeMode).toBe('append'); - }); - - it('should accept full destination', () => { - expect(() => ETLDestinationSchema.parse({ - type: 'warehouse', - connector: 'snowflake', - config: { schema: 'public', table: 'dim_accounts' }, - writeMode: 'upsert', - primaryKey: ['account_id'], - })).not.toThrow(); - }); - - it('should reject invalid writeMode', () => { - expect(() => ETLDestinationSchema.parse({ - type: 'database', - config: {}, - writeMode: 'truncate', - })).toThrow(); - }); -}); - -describe('ETLTransformationTypeSchema', () => { - it('should accept all valid types', () => { - const types = [ - 'map', 'filter', 'aggregate', 'join', 'script', - 'lookup', 'split', 'merge', 'normalize', 'deduplicate', - ]; - types.forEach(t => { - expect(() => ETLTransformationTypeSchema.parse(t)).not.toThrow(); - }); - }); - - it('should reject invalid type', () => { - expect(() => ETLTransformationTypeSchema.parse('pivot')).toThrow(); - }); -}); - -describe('ETLTransformationSchema', () => { - it('should accept minimal transformation with defaults', () => { - const result = ETLTransformationSchema.parse({ - type: 'map', - config: { Name: 'account_name' }, - }); - expect(result.continueOnError).toBe(false); - }); - - it('should accept full transformation', () => { - expect(() => ETLTransformationSchema.parse({ - name: 'filter_active', - type: 'filter', - config: { condition: 'status == "active"' }, - continueOnError: true, - })).not.toThrow(); - }); - - it('should reject missing config', () => { - expect(() => ETLTransformationSchema.parse({ - type: 'script', - })).toThrow(); - }); -}); - -describe('ETLSyncModeSchema', () => { - it('should accept all valid sync modes', () => { - ['full', 'incremental', 'cdc'].forEach(m => { - expect(() => ETLSyncModeSchema.parse(m)).not.toThrow(); - }); - }); - - it('should reject invalid mode', () => { - expect(() => ETLSyncModeSchema.parse('realtime')).toThrow(); - }); -}); - -describe('ETLPipelineSchema', () => { - const minimalPipeline = { - name: 'sf_to_postgres', - source: { type: 'api', config: { object: 'Account' } }, - destination: { type: 'database', config: { table: 'accounts' } }, - }; - - it('should accept minimal pipeline with defaults', () => { - const result = ETLPipelineSchema.parse(minimalPipeline); - expect(result.syncMode).toBe('full'); - expect(result.enabled).toBe(true); - }); - - it('should accept full pipeline', () => { - expect(() => ETLPipelineSchema.parse({ - name: 'multi_source_pipeline', - label: 'Multi-Source Pipeline', - description: 'Aggregates data from multiple sources', - source: { - type: 'api', - connector: 'salesforce', - config: { object: 'Account' }, - incremental: { enabled: true, cursorField: 'updated_at' }, - }, - destination: { - type: 'warehouse', - connector: 'snowflake', - config: { table: 'dim_accounts' }, - writeMode: 'merge', - primaryKey: ['account_id'], - }, - transformations: [ - { type: 'map', config: { Name: 'account_name' } }, - { type: 'filter', config: { condition: 'status == "active"' } }, - { name: 'dedup', type: 'deduplicate', config: { key: 'account_id' }, continueOnError: true }, - ], - syncMode: 'incremental', - schedule: '0 2 * * *', - enabled: true, - retry: { maxRetries: 5, backoffMs: 120000 }, - notifications: { - onSuccess: ['data-team@example.com'], - onFailure: ['ops@example.com'], - }, - tags: ['salesforce', 'analytics'], - metadata: { owner: 'data-team' }, - })).not.toThrow(); - }); - - it('should reject invalid name (not snake_case)', () => { - expect(() => ETLPipelineSchema.parse({ - ...minimalPipeline, - name: 'SfToPostgres', - })).toThrow(); - }); - - it('should reject missing source', () => { - expect(() => ETLPipelineSchema.parse({ - name: 'bad_pipeline', - destination: { type: 'database', config: {} }, - })).toThrow(); - }); - - /** - * #4962 — retry is OPT-IN, and this is the assertion that says so. - * - * Until 17 this block defaulted `maxAttempts: 3` / `backoffMs: 60000`, so - * `retry: {}` bought three silent re-runs a minute apart. It now carries the - * converged `RetryPolicySchema` contract, whose count defaults to 0. The - * business ground is the destination: an ETL destination is a foreign system - * by definition, so an implicit retry against a non-idempotent one is a - * duplicate write. Nothing deployed moves — `etl.zod.ts` has no parse site - * and an ETL pipeline is not a `defineStack` collection — which is exactly - * why this was the cheapest moment to fix the direction. - */ - it('defaults the retry count to 0 — declaring the block does not buy retries (#4962)', () => { - const result = ETLPipelineSchema.parse({ - ...minimalPipeline, - retry: {}, - }); - expect(result.retry?.maxRetries).toBe(0); - expect(result.retry?.backoffMs).toBe(1000); - }); - - it('accepts the three knobs this block never had before the convergence (#4962)', () => { - // `backoffMultiplier` / `maxRetryDelayMs` / `jitter` were 批 12's - // "documented ABSENCE" guidance entries — a nightly warehouse pipeline - // could only retry flat, uncapped and unjittered, the textbook - // thundering herd. - const result = ETLPipelineSchema.parse({ - ...minimalPipeline, - retry: { maxRetries: 3, backoffMs: 60000, backoffMultiplier: 2, maxRetryDelayMs: 600000, jitter: true }, - }); - expect(result.retry).toMatchObject({ - maxRetries: 3, backoffMs: 60000, backoffMultiplier: 2, maxRetryDelayMs: 600000, jitter: true, - }); - }); - - it('caps the retry count at 10, the shared contract\'s bound (#4962)', () => { - // The old inline block had no upper bound. Clamping silently would halve a - // budget its author chose, so the bound is refused at parse instead. - expect(() => ETLPipelineSchema.parse({ ...minimalPipeline, retry: { maxRetries: 11 } })).toThrow(); - expect(() => ETLPipelineSchema.parse({ ...minimalPipeline, retry: { maxRetries: 10 } })).not.toThrow(); - }); -}); - -describe('ETLRunStatusSchema', () => { - it('should accept all valid statuses', () => { - ['pending', 'running', 'succeeded', 'failed', 'cancelled', 'timeout'].forEach(s => { - expect(() => ETLRunStatusSchema.parse(s)).not.toThrow(); - }); - }); - - it('should reject invalid status', () => { - expect(() => ETLRunStatusSchema.parse('completed')).toThrow(); - }); -}); - -describe('ETLPipelineRunSchema', () => { - it('should accept minimal run', () => { - expect(() => ETLPipelineRunSchema.parse({ - id: 'run-001', - pipelineName: 'sf_to_postgres', - status: 'succeeded', - startedAt: '2024-01-01T02:00:00Z', - })).not.toThrow(); - }); - - it('should accept full run result', () => { - expect(() => ETLPipelineRunSchema.parse({ - id: 'run-002', - pipelineName: 'sf_to_postgres', - status: 'failed', - startedAt: '2024-01-01T02:00:00Z', - completedAt: '2024-01-01T02:15:00Z', - durationMs: 900000, - stats: { - recordsRead: 5000, - recordsWritten: 4950, - recordsErrored: 50, - bytesProcessed: 1048576, - }, - error: { - message: 'Connection timeout', - code: 'TIMEOUT', - details: { host: 'db.example.com' }, - }, - logs: ['Starting pipeline', 'Extraction complete', 'Load failed'], - })).not.toThrow(); - }); - - it('should reject invalid datetime', () => { - expect(() => ETLPipelineRunSchema.parse({ - id: 'run-003', - pipelineName: 'test', - status: 'running', - startedAt: 'not-a-date', - })).toThrow(); - }); - - it('should reject missing required fields', () => { - expect(() => ETLPipelineRunSchema.parse({ - id: 'run-004', - })).toThrow(); - }); -}); - -describe('ETL factory', () => { - it('should create database-to-database pipeline', () => { - const pipeline = ETL.databaseSync({ - name: 'users_sync', - sourceTable: 'users_source', - destTable: 'users_dest', - schedule: '0 * * * *', - }); - expect(pipeline.source.type).toBe('database'); - expect(pipeline.destination.type).toBe('database'); - expect(pipeline.destination.writeMode).toBe('upsert'); - expect(pipeline.syncMode).toBe('incremental'); - expect(() => ETLPipelineSchema.parse(pipeline)).not.toThrow(); - }); - - it('should create API-to-database pipeline', () => { - const pipeline = ETL.apiToDatabase({ - name: 'api_ingest', - apiConnector: 'stripe', - destTable: 'payments', - }); - expect(pipeline.source.type).toBe('api'); - expect(pipeline.source.connector).toBe('stripe'); - expect(pipeline.destination.writeMode).toBe('append'); - expect(pipeline.syncMode).toBe('full'); - expect(() => ETLPipelineSchema.parse(pipeline)).not.toThrow(); - }); -}); - -// ─── #4001 批 12 — unknown-key strictness (ADR-0078) ────────────────── -// -// The file splits 7 authorable / 3 wire. Everything below is written to fail -// LOUDLY if either half moves: the seven must reject, the three must tolerate, -// and every rejection is paired with a positive control parsing the SAME -// document minus the offending key — so a red assertion can never be a document -// that was invalid for an unrelated reason (the campaign's "prove the -// instrument red before trusting green"). - -/** A pipeline that parses clean — the base every negative case is built from. */ -const VALID_PIPELINE = { - name: 'customer_360_pipeline', - label: 'Customer 360', - source: { - type: 'api', - connector: 'salesforce', - config: { object: 'Account' }, - incremental: { enabled: true, cursorField: 'updated_at' }, - }, - destination: { - type: 'warehouse', - connector: 'snowflake', - config: { table: 'customers' }, - writeMode: 'upsert', - primaryKey: ['customer_id'], - }, - transformations: [{ name: 'only_active', type: 'filter', config: { condition: 'active' } }], - syncMode: 'incremental', - schedule: '0 2 * * *', - enabled: true, - retry: { maxRetries: 5, backoffMs: 120000 }, - notifications: { onSuccess: ['data@example.com'], onFailure: ['ops@example.com'] }, - tags: ['analytics'], - metadata: { owner: 'data-team' }, -} as const; - -/** A run result that parses clean — the base for the wire-half tolerance pins. */ -const VALID_RUN = { - id: 'run-001', - pipelineName: 'customer_360_pipeline', - status: 'succeeded', - startedAt: '2024-01-01T02:00:00Z', - completedAt: '2024-01-01T02:15:00Z', - durationMs: 900000, - stats: { recordsRead: 10, recordsWritten: 10, recordsErrored: 0, bytesProcessed: 2048 }, - error: { message: 'none', code: 'OK' }, - logs: ['ok'], -} as const; - -/** Deep-clone the base and drop an unknown key into one nested block. */ -function pipelineWith(path: readonly string[], key: string, value: unknown) { - const doc = structuredClone(VALID_PIPELINE) as Record; - let cursor: Record = doc; - for (const step of path) cursor = cursor[step] as Record; - cursor[key] = value; - return doc; -} - -/** The unknown-key message for `key` written at `path`, or `null` if it parsed. */ -function rejectionFor(path: readonly string[], key: string): string | null { - const result = ETLPipelineSchema.safeParse(pipelineWith(path, key, 'x')); - return result.success ? null : result.error.issues.map((i) => i.message).join('\n'); -} - -describe('[#4001 批 12] the seven authorable shapes reject unknown keys', () => { - it('parses the base document — the positive control every negative below relies on', () => { - const result = ETLPipelineSchema.safeParse(structuredClone(VALID_PIPELINE)); - expect(result.success, result.success ? '' : JSON.stringify(result.error.issues)).toBe(true); - }); - - // Each row: where the key is written, and a distinctive fragment of the - // surface name that must appear in the rejection. The surface name is what - // tells an author WHICH of the seven nested shapes they got wrong, which is - // the difference between a fixable error and a puzzle. - const surfaces: ReadonlyArray = [ - ['pipeline', [], 'this ETL pipeline'], - ['source', ['source'], 'this ETL source'], - ['source.incremental', ['source', 'incremental'], 'this ETL incremental extraction config'], - ['destination', ['destination'], 'this ETL destination'], - ['pipeline.retry', ['retry'], "this ETL pipeline's retry configuration"], - ['pipeline.notifications', ['notifications'], "this ETL pipeline's notification settings"], - ]; - - it.each(surfaces)('rejects an unknown key on %s, naming the surface', (_label, path, surface) => { - const message = rejectionFor(path, 'totallyMadeUpKey'); - expect(message, 'the key must be REJECTED, not stripped').not.toBeNull(); - expect(message).toContain(surface); - expect(message).toContain('`totallyMadeUpKey`'); - // History: every message says what used to happen silently. - expect(message).toContain('Until #4001'); - }); - - it('rejects an unknown key on a transformation — the seventh shape, reached through the array', () => { - const doc = structuredClone(VALID_PIPELINE) as Record; - (doc.transformations as Array>)[0].continueOnFailure = true; - const result = ETLPipelineSchema.safeParse(doc); - expect(result.success).toBe(false); - const message = result.success ? '' : result.error.issues.map((i) => i.message).join('\n'); - expect(message).toContain('this ETL transformation'); - expect(message).toContain('`continueOnFailure`'); - }); - - it('points a misplaced endpoint setting at the open `config` bag', () => { - // The dominant failure on this file is not a typo: `table` IS a real - // setting, one level down. `.strip` deleted it where it stood and the - // pipeline loaded into whatever `config.table` said instead. - const message = rejectionFor(['destination'], 'table'); - expect(message).toContain('inside the open `config` record'); - }); - - it('rejects each of the seven when parsed standalone, not only through the pipeline', () => { - // The nested shapes are also exported/reachable on their own; strictness - // must not depend on arriving through ETLPipelineSchema. - expect(ETLSourceSchema.safeParse({ type: 'database', config: {}, nope: 1 }).success).toBe(false); - expect(ETLDestinationSchema.safeParse({ type: 'database', config: {}, nope: 1 }).success).toBe(false); - expect(ETLTransformationSchema.safeParse({ type: 'map', config: {}, nope: 1 }).success).toBe(false); - // …and the same three documents WITHOUT the key still parse. - expect(ETLSourceSchema.safeParse({ type: 'database', config: {} }).success).toBe(true); - expect(ETLDestinationSchema.safeParse({ type: 'database', config: {} }).success).toBe(true); - expect(ETLTransformationSchema.safeParse({ type: 'map', config: {} }).success).toBe(true); - }); -}); - -describe('[#4001 批 12] curated prescriptions — each anchored to a sibling contract', () => { - it('renames the connector layer’s `timestampField` to `cursorField`', () => { - // Anchor: integration/connector.zod.ts DataSyncConfig.timestampField. - expect(rejectionFor(['source', 'incremental'], 'timestampField')) - .toContain('`timestampField` → `cursorField`'); - }); - - it('resolves a borrowed `strategy` to a DIFFERENT key per surface', () => { - // One connector enum (`full | incremental | upsert | append_only`) splits - // across two keys here. A single global alias would confidently misdirect - // one of the two surfaces, so this pair is the load-bearing assertion. - expect(rejectionFor(['destination'], 'strategy')).toContain('`strategy` → `writeMode`'); - expect(rejectionFor([], 'strategy')).toContain('`strategy` → `syncMode`'); - }); - - it('renames `onError` to `onFailure` on the notification block', () => { - expect(rejectionFor(['notifications'], 'onError')).toContain('`onError` → `onFailure`'); - }); - - /** - * The four 批 12 curation entries that #4962 DISSOLVED, asserted from the - * other side so a regression reads as a failure rather than as silence. - * - * 批 12 could only make this divergence audible: `maxRetries` was aliased - * *to* `maxAttempts` (pointing authors away from the canonical spelling), and - * `backoffMultiplier` / `maxRetryDelayMs` / `jitter` each carried a - * "documented ABSENCE" guidance entry. Convergence removes the divergence the - * entries described, so the entries had to go with it — a curated message - * outliving the shape it describes is worse than none, because it is - * confidently wrong. - */ - it('no longer points `maxRetries` at `maxAttempts` — the alias inverted (#4962)', () => { - // `maxRetries` is now a DECLARED key: writing it must parse, not suggest. - const result = ETLPipelineSchema.safeParse(pipelineWith(['retry'], 'maxRetries', 3)); - expect(result.success, result.success ? '' : JSON.stringify(result.error.issues)).toBe(true); - }); - - it('tombstones `maxAttempts` with the rename AND the off-by-one warning (#4962)', () => { - const retired = rejectionFor(['retry'], 'maxAttempts'); - expect(retired).toContain('was removed'); - expect(retired).toContain('maxRetries'); - expect(retired).toContain('#4962'); - // The number does NOT change — and the message must say so, because the - // identically-spelled connector key IS off by one. - expect(retired).toContain('NUMBER IS UNCHANGED'); - expect(retired).toContain('RetryConfig.maxAttempts'); - // The default flip has to travel with the rename, or an author does a - // lossless-looking rename and silently loses their three retries. - expect(retired).toContain('maxRetries: 3'); - }); - - it('tombstones `retryDelayMs` via the shared policy, naming this surface (#4661, #4964)', () => { - const retired = rejectionFor(['retry'], 'retryDelayMs'); - expect(retired).toContain('backoffMs'); - expect(retired).toContain('#4661'); - }); - - it('DECLARES the three keys 批 12 documented as absent (#4962)', () => { - for (const key of ['backoffMultiplier', 'maxRetryDelayMs', 'jitter']) { - const message = rejectionFor(['retry'], key); - // 'x' is the wrong TYPE for all three, so a rejection is expected — what - // must be gone is the absence prescription: the key is real now. - expect(message, `${key} must no longer be described as absent`).not.toContain('documented ABSENCE'); - } - const parsed = ETLPipelineSchema.safeParse( - pipelineWith(['retry'], 'jitter', true), - ); - expect(parsed.success).toBe(true); - }); - - it('explains that pipeline direction is structural, not a key', () => { - const message = rejectionFor([], 'direction'); - expect(message).toContain('no `direction` key'); - expect(message).toContain('swap the two endpoints'); - // A guidance entry SUPPRESSES the rename suggestion — the author is told - // the mechanism, not sent to a key that does not mean the same thing. - expect(message).not.toContain('Did you mean'); - }); - - it('never suggests a key the schema will not accept', () => { - // The helper's `acceptsNothing` rule, asserted from this file's side: every - // key named in a "Did you mean X" must itself parse when written. - const message = rejectionFor([], 'sourse') ?? ''; - const suggested = [...message.matchAll(/→ `([^`]+)`/g)].map((m) => m[1]); - expect(suggested).toContain('source'); - for (const key of suggested) { - expect(Object.keys(VALID_PIPELINE), `${key} must be a real, writable key`).toContain(key); - } - }); -}); - -describe('[#4001 批 12] the three wire shapes stay tolerant — deliberate, and pinned', () => { - // If a later sweep closes these, THESE tests are what must be consciously - // deleted. That is the point: the exemption is a decision with a receipt, not - // an omission (see the comment on ETLPipelineRunSchema). - it('parses the base run result — positive control', () => { - expect(ETLPipelineRunSchema.safeParse(structuredClone(VALID_RUN)).success).toBe(true); - }); - - it.each([ - ['ETLPipelineRunSchema', [] as string[]], - ['ETLPipelineRunSchema.stats', ['stats']], - ['ETLPipelineRunSchema.error', ['error']], - ])('%s forwards an engine-added key instead of crashing', (_label, path) => { - const doc = structuredClone(VALID_RUN) as Record; - let cursor: Record = doc; - for (const step of path) cursor = cursor[step] as Record; - // The realistic case: a future engine reports one more counter. On a strict - // shape that is a parse CRASH for every existing reader — the #3712 shape. - cursor.recordsSkippedByANewerEngine = 7; - expect(ETLPipelineRunSchema.safeParse(doc).success).toBe(true); - }); - - it('still validates what it does declare — tolerance is not absence of a contract', () => { - // Anti-vacuity: the pins above must not be passing because the schema - // validates nothing at all. - expect(ETLPipelineRunSchema.safeParse({ ...VALID_RUN, status: 'completed' }).success).toBe(false); - expect(ETLPipelineRunSchema.safeParse({ ...VALID_RUN, startedAt: 'not-a-date' }).success).toBe(false); - expect(ETLPipelineRunSchema.safeParse({ id: 'r' }).success).toBe(false); - }); -}); - -describe('[#4001 批 12] strictness does not disturb the published contract', () => { - it('converts to JSON Schema through the lazy proxy without throwing (#3746 hazard)', async () => { - const { z } = await import('zod'); - for (const schema of [ - ETLSourceSchema, ETLDestinationSchema, ETLTransformationSchema, - ETLPipelineSchema, ETLPipelineRunSchema, - ]) { - // `io: 'input'` is asserted rather than the default because - // `ETLPipelineSchema` does not convert in OUTPUT mode — `schedule` is - // `CronExpressionInputSchema`, a transform, and "Transforms cannot be - // represented in JSON Schema". That is PRE-EXISTING and unrelated to - // strictness (verified: the same throw on the pre-批-12 file), and - // `build-schemas.ts` already handles it by falling back to input mode. - // Asserting the default here would have pinned someone else's known - // limitation as if this batch owned it. - expect(() => z.toJSONSchema(schema as never, { io: 'input' })).not.toThrow(); - } - }); - - /** - * The campaign's standing claim is that strictness does not move the - * published JSON Schema: `build-schemas.ts` converts with `io: 'output'`, and - * OUTPUT mode already emits `additionalProperties: false` for a `.strip()` - * object (pinned in `shared/strict-object.test.ts`). - * - * That claim holds per-direction, and this file is the case where the - * direction is not the usual one. `ETLPipelineSchema` cannot convert in - * output mode at all — `schedule` is `CronExpressionInputSchema`, a transform - * — so `build-schemas.ts` falls back to `io: 'input'`, and INPUT mode - * distinguishes the two postures: strip emits nothing, strict emits `false`. - * - * So for this one schema the batch DOES narrow the published contract, from - * "unspecified" to "closed". That is the intended direction (the publication - * now matches the parse instead of being quieter than it) but it is a real - * artifact change, not a no-op, and pretending otherwise is how a generated - * baseline moves without anyone reading it. Pinned here in both directions so - * the distinction survives the next person who quotes the flat claim. - */ - it('narrows the published schema only where input-mode fallback applies', async () => { - const { z } = await import('zod'); - const json = (s: unknown, io: 'input' | 'output') => - z.toJSONSchema(s as never, { io }) as Record; - - // The seven, via the pipeline: closed in the direction that gets published. - expect(json(ETLPipelineSchema, 'input').additionalProperties).toBe(false); - - // The three: still `z.object`, so they follow strip's per-direction shape — - // open in input mode, `false` in output mode. This is the receipt that the - // wire exemption is real at the schema level and not only in the parse. - expect(json(ETLPipelineRunSchema, 'input').additionalProperties).toBeUndefined(); - expect(json(ETLPipelineRunSchema, 'output').additionalProperties).toBe(false); - }); - - it('leaves both ETL factories producing documents that parse', () => { - // The factories construct pipelines programmatically — exactly the caller a - // newly-strict schema would break if the campaign had guessed a key wrong. - expect(ETLPipelineSchema.safeParse(ETL.databaseSync({ - name: 'users_sync', sourceTable: 'src', destTable: 'dst', schedule: '0 * * * *', - })).success).toBe(true); - expect(ETLPipelineSchema.safeParse(ETL.apiToDatabase({ - name: 'api_ingest', apiConnector: 'stripe', destTable: 'payments', - })).success).toBe(true); - }); -}); diff --git a/packages/spec/src/automation/etl.zod.ts b/packages/spec/src/automation/etl.zod.ts deleted file mode 100644 index 2986611f3c..0000000000 --- a/packages/spec/src/automation/etl.zod.ts +++ /dev/null @@ -1,841 +0,0 @@ -// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. - -import { z } from 'zod'; -import { CronExpressionInputSchema } from '../shared/expression.zod'; -import { retiredKey } from '../shared/retired-key'; -import { retryPolicyShape } from '../shared/retry-policy.zod'; -import { strictObject } from '../shared/strict-object'; - -/** - * ETL (Extract, Transform, Load) Pipeline Protocol - LEVEL 2: Data Engineering - * - * Inspired by modern data integration platforms like Airbyte, Fivetran, and Apache NiFi. - * - * **Positioning in the sync/integration layering** (L1 "Simple Sync" was - * retired in #4738 — narrative-only, zero consumers; see - * `packages/spec/docs/SYNC_ARCHITECTURE.md`): - * - **ETL Pipeline** (THIS FILE) - Data engineers - Aggregate 10 sources to warehouse - * - **Enterprise Connector** (integration/connector.zod.ts) - System integrators - Full SAP integration; connector-attached sync via `syncConfig` - * - * ETL pipelines enable automated data synchronization between systems, transforming - * data as it moves from source to destination. - * - * **SCOPE: Advanced multi-source, multi-stage transformations.** - * Supports complex operations: joins, aggregations, filtering, custom SQL. - * - * ## When to Use This Layer - * - * **Use ETL Pipeline when:** - * - Combining data from multiple sources - * - Need aggregations, joins, transformations - * - Building data warehouses or analytics platforms - * - Complex data transformations required - * - * **Examples:** - * - Sales data from Salesforce + Marketing from HubSpot → Data Warehouse - * - Multi-region databases → Consolidated reporting - * - Legacy system migration with transformation - * - * **When to upgrade:** - * - Need full connector lifecycle (auth, webhooks, rate limits) → Use {@link file://../integration/connector.zod.ts | Enterprise Connector} - * - * @see {@link file://../integration/connector.zod.ts} for the Enterprise Connector layer - * - * ## Use Cases - * - * 1. **Data Warehouse Population** - * - Extract from multiple operational systems - * - Transform to analytical schema - * - Load into data warehouse - * - * 2. **System Integration** - * - Sync data between CRM and Marketing Automation - * - Keep product catalog synchronized across e-commerce platforms - * - Replicate data for backup/disaster recovery - * - * 3. **Data Migration** - * - Move data from legacy systems to modern platforms - * - Consolidate data from multiple sources - * - Split monolithic databases into microservices - * - * @see https://airbyte.com/ - * @see https://docs.fivetran.com/ - * @see https://nifi.apache.org/ - * - * @example - * ```typescript - * const salesforceToDB: ETLPipeline = { - * name: 'salesforce_to_postgres', - * label: 'Salesforce Accounts to PostgreSQL', - * source: { - * type: 'api', - * connector: 'salesforce', - * config: { object: 'Account' } - * }, - * destination: { - * type: 'database', - * connector: 'postgres', - * config: { table: 'accounts' } - * }, - * transformations: [ - * { type: 'map', config: { 'Name': 'account_name' } } - * ], - * schedule: '0 2 * * *' // Daily at 2 AM - * } - * ``` - */ - -/** - * ETL Source/Destination Type - */ -import { lazySchema } from '../shared/lazy-schema'; -export const ETLEndpointTypeSchema = lazySchema(() => z.enum([ - 'database', // SQL/NoSQL databases - 'api', // REST/GraphQL APIs - 'file', // CSV, JSON, XML, Excel files - 'stream', // Kafka, RabbitMQ, Kinesis - 'object', // ObjectStack object - 'warehouse', // Data warehouse (Snowflake, BigQuery, Redshift) - 'storage', // S3, Azure Blob, Google Cloud Storage - 'spreadsheet', // Google Sheets, Excel Online -])); - -// ─── `X` / `XParsed` — which shape the bare name means (#4963) ──────── -// -// House convention, stated once here because this file exports nine pairs of -// it: the **bare name is what an author writes** (`z.input` — defaults -// unapplied, every defaulted key optional), and `XParsed` is **what a parse -// returns** (`z.infer` — defaults applied, those same keys present). The -// clearest write-up is on `shared/retry-policy.zod.ts`; the sibling automation -// configs (`flow.zod.ts`, `io-node-config.zod.ts`, -// `builtin-node-config.zod.ts`, `control-flow.zod.ts`) all export the pair. -// -// Until 17 all nine aliases here were `z.infer` under the bare name with no -// `*Parsed` counterpart at all, and on this file that was not a style detail. -// Five named keys across four shapes carry `.default()` — -// `ETLDestination.writeMode`, `ETLTransformation.continueOnError`, -// `ETLPipeline.syncMode` / `.enabled`, `ETLSource.incremental.enabled` — plus -// all five of `ETLPipeline.retry`'s (via `retryPolicyShape()`) and all four of -// `ETLPipelineRun.stats`'. And `schedule` is a `CronExpressionInputSchema` -// transform whose *output* is the `{ dialect, source }` envelope. Under -// `z.infer` every one of them was REQUIRED and a bare-string cron was -// rejected, so the one use this file actually has — -// `const p: ETLPipeline = { … }`, hand-written, which is the whole authoring -// door given there is no parse site in objectstack / objectui / cloud — did not -// compile. `packages/spec/docs/SYNC_ARCHITECTURE.md` carried three examples -// that proved it, and both ETL factories below were forced to spell defaults -// out and pre-wrap their cron just to satisfy their own return type. -// -// Flipping the bare names is breaking and was done in one step (#4963): the -// three-repo importer count was zero, so the migration surface is empty. A -// consumer that reads a PARSE RESULT — `const p = ETLPipelineSchema.parse(raw)` -// annotated by hand — renames its annotation to `ETLPipelineParsed`. -// -// The four enum aliases (`ETLEndpointType`, `ETLTransformationType`, -// `ETLSyncMode`, `ETLRunStatus`) get the pair too, even though `z.input` and -// `z.infer` are the same type for an enum. That is deliberate, not -// cargo-culting: the convention's value is that a reader never has to know -// WHICH of the nine has defaults before choosing an annotation, and a pair that -// exists today keeps costing nothing while an enum that later gains a -// `.transform()` or `.catch()` would otherwise reopen exactly this issue. - -export type ETLEndpointType = z.input; -/** @see {@link ETLEndpointType} — the enum has no transform, so this is the same type. */ -export type ETLEndpointTypeParsed = z.infer; - -// ─── Unknown-key strictness (#4001 批 12, ADR-0078) ─────────────────── -// -// The seven AUTHORING shapes in this file are closed against undeclared keys; -// the three `ETLPipelineRun` shapes at the bottom stay tolerant. The split, and -// how it was verified rather than assumed, is recorded on `ETLPipelineRunSchema` -// — read it before closing anything else here. -// -// One thing to know about this file before reading the tables: `etl.zod.ts` has -// **no parse site anywhere** in objectstack / objectui / cloud. That does not -// make it unauthored — it makes the exported schema and the exported type the -// whole authoring door (`const p: ETLPipeline = { … }`, as -// `packages/spec/docs/SYNC_ARCHITECTURE.md` and this module's own `@example` -// write it), the same posture the ledger already carries for `webhook.zod.ts`. -// It does mean the curation below could not be measured from stored payloads -// the way 批 9's was, because there are none. So every alias and guidance entry -// here is instead anchored to a **sibling contract that exists in this repo** -// and spells the same intent differently — each one names its anchor. Nothing -// is written from imagination: the campaign's finding 7 is that a confidently -// wrong prescription costs more than no prescription at all. - -/** - * The shared second half of the endpoint/transformation histories: where a - * misplaced setting actually goes. - * - * `source`, `destination` and each `transformation` all pair a small closed key - * set with an open `config: z.record(…)` bag, and that pairing is what makes an - * unknown key on these three a **misplacement** far more often than a typo. - * `table`, `schema`, `endpoint`, `path`, `format`, `condition`, `groupBy` are - * all real, load-bearing settings — one nesting level down. `.strip` deleted - * them where they stood, so the pipeline parsed clean and then ran against a - * `config` missing exactly the setting the author had written. - * - * The pointer lives in `history` — appended to *every* unknown-key message on - * these surfaces — rather than in a per-key `guidance` table, because the - * misplaced key is drawn from the open bag's unbounded vocabulary. Enumerating - * it would be guesswork; naming the destination is not. - */ -const ETL_CONFIG_SLOT_POINTER = - 'If the key is a real setting (`table`, `schema`, `endpoint`, `path`, `format`, `condition`, `groupBy`, …) ' - + 'it belongs one level down, inside the open `config` record — that bag is deliberately unconstrained and ' - + 'is the only place this schema reads endpoint-specific settings from.'; - -const ETL_SOURCE_HISTORY = - 'Until #4001 an undeclared key on an ETL source was dropped silently — the pipeline parsed clean and ' - + 'extracted with the key ignored. ' + ETL_CONFIG_SLOT_POINTER; - -const ETL_INCREMENTAL_HISTORY = - 'Until #4001 an undeclared key here was dropped silently, and an incremental source whose cursor never ' - + 'took effect re-extracts the whole table (or nothing) on every run while still reporting success.'; - -/** - * Anchor: `DataSyncConfig` in `integration/connector.zod.ts` — the LIVE sibling - * (it is on the `ConnectorSchema.syncConfig` parse path) — calls this same thing - * `timestampField`, "Field to track last modification time". Identical intent, - * different word, and far outside the edit-distance window, which is exactly - * the category `aliases` exists for. - */ -const ETL_INCREMENTAL_ALIASES: Readonly> = { - timestampField: 'cursorField', -}; - -const ETL_DESTINATION_HISTORY = - 'Until #4001 an undeclared key on an ETL destination was dropped silently — the pipeline parsed clean and ' - + 'loaded with the key ignored. ' + ETL_CONFIG_SLOT_POINTER; - -/** - * Anchor: connector `syncConfig.strategy` (`SyncStrategySchema`, - * `integration/connector.zod.ts`) is ONE enum — `full | incremental | upsert | - * append_only` — whose four values split across TWO keys on this file: the - * write half (`upsert` / `append_only`) is the destination's `writeMode`, the - * extraction half (`full` / `incremental`) is the pipeline's `syncMode`. - * - * So the same borrowed word resolves to a different canonical key depending on - * which surface it was written on, and each surface names only its own half. - * Getting that right is the whole value of a hand-written alias here — a - * single global "strategy → syncMode" would send a destination author to the - * wrong key with full confidence. - */ -const ETL_DESTINATION_ALIASES: Readonly> = { - strategy: 'writeMode', -}; - -const ETL_TRANSFORMATION_HISTORY = - 'Until #4001 an undeclared key on an ETL transformation was dropped silently — the step ran with the key ' - + 'ignored and the pipeline reported success. ' + ETL_CONFIG_SLOT_POINTER; - -const ETL_PIPELINE_HISTORY = - 'Until #4001 an undeclared key on an ETL pipeline was dropped silently — the pipeline parsed clean and ' - + 'ran, minus whatever the key was meant to configure.'; - -/** See {@link ETL_DESTINATION_ALIASES} — this is that enum's extraction half. */ -const ETL_PIPELINE_ALIASES: Readonly> = { - strategy: 'syncMode', -}; - -/** - * Anchor: `DataSyncConfig.direction` (`import | export | bidirectional`) is a - * declared key on the connector layer and a **documented absence** here — which - * makes it the likeliest wrong key for anyone arriving from that layer, exactly - * the shape 批 9 recorded for `outputVariable` on `update_record`. - */ -const ETL_PIPELINE_GUIDANCE: Readonly> = { - direction: - 'An ETL pipeline has no `direction` key, by design: direction is stated STRUCTURALLY, by which endpoint ' - + 'is `source` and which is `destination`. `direction` (import/export/bidirectional) is the CONNECTOR ' - + "layer's spelling (`ConnectorSchema.syncConfig`); to reverse an ETL pipeline, swap the two endpoints.", -}; - -const ETL_RETRY_HISTORY = - 'Until #4001 an undeclared key here was dropped silently and the block fell back to its defaults ' - + '(3 attempts, 60s) while reporting the authored policy as accepted. Until #4962 this block was a ' - + 'SEPARATE retry vocabulary — it spelled the count `maxAttempts`, defaulted it to 3, and declared no ' - + 'backoff multiplier, ceiling or jitter; it now carries the converged `RetryPolicySchema` contract.'; - -/** - * Anchor: `RetryPolicySchema` (`shared/retry-policy.zod.ts`) — the retry policy - * #4661 converged onto ONE declaration for `job.retryPolicy` and a `try_catch` - * node's `retry` region. This block was a **third** encoding of the same - * concept that the convergence did not reach, because it is an anonymous - * inline object with no exported name and so never appeared in the #4411 / - * #4535 dual-source scan that drove that work. (`Flow.errorHandling` was the - * fourth — #4964, same construction, same blind spot.) - * - * 批 12 could only make the divergence *audible*: it closed the shape and spent - * five curated entries stating the diff between the two vocabularies where an - * author hits it. #4962 removed the diff instead, so all five entries are gone - * — four of them (`retryDelayMs` plus the "documented absence" of - * `backoffMultiplier` / `maxRetryDelayMs` / `jitter`) because those keys are - * now DECLARED here, and the fifth (`maxRetries` → `maxAttempts`) because it - * pointed the wrong way: `maxRetries` is the canonical spelling and - * `maxAttempts` is the tombstone. - * - * What survives is anchored the same way 批 12's entries were — to sibling - * contracts that exist in this repo and spell the same knob differently: - * `integration/connector.zod.ts`'s `RetryConfig` (`initialDelayMs`, - * `maxDelayMs`), and the plain-English count forms. This is deliberately the - * SAME table `Flow.errorHandling` carries, because after the convergence the - * two surfaces are the same contract and an author should not learn two - * different lessons from them. - */ -const ETL_RETRY_ALIASES: Readonly> = { - initialDelayMs: 'backoffMs', - maxDelayMs: 'maxRetryDelayMs', - retries: 'maxRetries', - attempts: 'maxRetries', -}; - -const ETL_NOTIFICATIONS_HISTORY = - 'Until #4001 an undeclared key here was dropped silently — nobody was notified and the run still reported ' - + 'success, which is the one outcome this block exists to prevent.'; - -/** - * Anchor: three in-repo surfaces spell the failure hook `onError` - * (`ui/widget.zod.ts`, `data/hook.zod.ts`'s declared key list, - * `kernel/plugin-loading.zod.ts`). This block spells it `onFailure`, and the - * two are six edits apart — unreachable by the distance fallback. - */ -const ETL_NOTIFICATIONS_ALIASES: Readonly> = { - onError: 'onFailure', -}; - -/** - * ETL Source Configuration - */ -export const ETLSourceSchema = lazySchema(() => strictObject({ - surface: 'this ETL source', - history: ETL_SOURCE_HISTORY, -}, { - /** - * Source type - */ - type: ETLEndpointTypeSchema.describe('Source type'), - - /** - * Connector identifier - * References a registered connector - * - * @example "salesforce", "postgres", "mysql", "s3" - */ - connector: z.string().optional().describe('Connector ID'), - - /** - * Source-specific configuration - * Structure varies by source type - * - * @example For database: { table: 'customers', schema: 'public' } - * @example For API: { endpoint: '/api/users', method: 'GET' } - * @example For file: { path: 's3://bucket/data.csv', format: 'csv' } - */ - config: z.record(z.string(), z.unknown()).describe('Source configuration'), - - /** - * Incremental sync configuration - * Allows extracting only changed data - */ - incremental: strictObject({ - surface: 'this ETL incremental extraction config', - history: ETL_INCREMENTAL_HISTORY, - aliases: ETL_INCREMENTAL_ALIASES, - }, { - enabled: z.boolean().default(false), - cursorField: z.string().describe('Field to track progress (e.g., updated_at)'), - cursorValue: z.unknown().optional().describe('Last processed value'), - }).optional().describe('Incremental extraction config'), -})); - -/** What an author writes — `incremental.enabled` optional. */ -export type ETLSource = z.input; -/** The post-parse shape — `incremental.enabled` present. */ -export type ETLSourceParsed = z.infer; - -/** - * ETL Destination Configuration - */ -export const ETLDestinationSchema = lazySchema(() => strictObject({ - surface: 'this ETL destination', - history: ETL_DESTINATION_HISTORY, - aliases: ETL_DESTINATION_ALIASES, -}, { - /** - * Destination type - */ - type: ETLEndpointTypeSchema.describe('Destination type'), - - /** - * Connector identifier - */ - connector: z.string().optional().describe('Connector ID'), - - /** - * Destination-specific configuration - */ - config: z.record(z.string(), z.unknown()).describe('Destination configuration'), - - /** - * Write mode - */ - writeMode: z.enum([ - 'append', // Add new records - 'overwrite', // Replace all data - 'upsert', // Insert or update based on key - 'merge', // Smart merge based on business rules - ]).default('append').describe('How to write data'), - - /** - * Primary key fields for upsert/merge - */ - primaryKey: z.array(z.string()).optional().describe('Primary key fields'), -})); - -/** What an author writes — `writeMode` optional (defaults to `append`). */ -export type ETLDestination = z.input; -/** The post-parse shape — `writeMode` present. */ -export type ETLDestinationParsed = z.infer; - -/** - * ETL Transformation Type - */ -export const ETLTransformationTypeSchema = lazySchema(() => z.enum([ - 'map', // Field mapping/renaming - 'filter', // Row filtering - 'aggregate', // Aggregation/grouping - 'join', // Joining with other data - 'script', // Custom JavaScript/Python script - 'lookup', // Enrich with lookup data - 'split', // Split one record into multiple - 'merge', // Merge multiple records into one - 'normalize', // Data normalization - 'deduplicate', // Remove duplicates -])); - -export type ETLTransformationType = z.input; -/** @see {@link ETLTransformationType} — the enum has no transform, so this is the same type. */ -export type ETLTransformationTypeParsed = z.infer; - -/** - * ETL Transformation Configuration - */ -export const ETLTransformationSchema = lazySchema(() => strictObject({ - surface: 'this ETL transformation', - history: ETL_TRANSFORMATION_HISTORY, - // No curated table. Every transformation-specific setting this file or - // SYNC_ARCHITECTURE.md ever writes (`condition`, `groupBy`, `joinKey`, - // `joinType`, `metrics`, `language`, `code`) lives inside the open `config` - // bag, so the pointer in `history` already answers them as a class; and this - // surface has no sibling contract spelling one of its four declared keys - // differently, so there is nothing an alias could honestly claim. Same - // reasoning as `HttpConfigSchema` in `io-node-config.zod.ts` (#4001 批 9). -}, { - /** - * Transformation name - */ - name: z.string().optional().describe('Transformation name'), - - /** - * Transformation type - */ - type: ETLTransformationTypeSchema.describe('Transformation type'), - - /** - * Transformation-specific configuration - * - * @example For map: { oldField: 'newField' } - * @example For filter: { condition: 'status == "active"' } - * @example For script: { language: 'javascript', code: '...' } - */ - config: z.record(z.string(), z.unknown()).describe('Transformation config'), - - /** - * Whether to continue on error - */ - continueOnError: z.boolean().default(false).describe('Continue on error'), -})); - -/** What an author writes — `continueOnError` optional (defaults to `false`). */ -export type ETLTransformation = z.input; -/** The post-parse shape — `continueOnError` present. */ -export type ETLTransformationParsed = z.infer; - -/** - * ETL Sync Mode - */ -export const ETLSyncModeSchema = lazySchema(() => z.enum([ - 'full', // Full refresh - extract all data every time - 'incremental', // Only extract changed data - 'cdc', // Change Data Capture - real-time streaming -])); - -export type ETLSyncMode = z.input; -/** @see {@link ETLSyncMode} — the enum has no transform, so this is the same type. */ -export type ETLSyncModeParsed = z.infer; - -/** - * ETL Pipeline Schema - * - * Complete definition of a data pipeline from source to destination with transformations. - */ -export const ETLPipelineSchema = lazySchema(() => strictObject({ - surface: 'this ETL pipeline', - history: ETL_PIPELINE_HISTORY, - aliases: ETL_PIPELINE_ALIASES, - guidance: ETL_PIPELINE_GUIDANCE, -}, { - /** - * Pipeline identifier (snake_case) - */ - name: z.string() - .regex(/^[a-z_][a-z0-9_]*$/) - .describe('Pipeline identifier (snake_case)'), - - /** - * Human-readable pipeline name - */ - label: z.string().optional().describe('Pipeline display name'), - - /** - * Pipeline description - */ - description: z.string().optional().describe('Pipeline description'), - - /** - * Data source configuration - */ - source: ETLSourceSchema.describe('Data source'), - - /** - * Data destination configuration - */ - destination: ETLDestinationSchema.describe('Data destination'), - - /** - * Transformation steps - * Applied in order from source to destination - */ - transformations: z.array(ETLTransformationSchema) - .optional() - .describe('Transformation pipeline'), - - /** - * Sync mode - */ - syncMode: ETLSyncModeSchema.default('full').describe('Sync mode'), - - /** - * Execution schedule (cron expression) - * - * @example "0 2 * * *" - Daily at 2 AM - * @example "0 *\/4 * * *" - Every 4 hours - * @example "0 0 * * 0" - Weekly on Sunday - */ - schedule: CronExpressionInputSchema.optional().describe('Cron schedule expression'), - - /** - * Whether pipeline is enabled - */ - enabled: z.boolean().default(true).describe('Pipeline enabled status'), - - /** - * Retry configuration for failed runs — the converged `RetryPolicySchema` - * contract (#4962), shared with `job.retryPolicy`, a `try_catch` node's - * `retry` and `flow.errorHandling`. - * - * Three things changed when this block stopped being its own vocabulary, and - * all three are breaking (17.0.0): - * - * 1. `maxAttempts` → `maxRetries`. Pure rename, value preserved: both count - * the retries AFTER the initial attempt. (Do NOT carry the off-by-one - * that `integration/connector.zod.ts`'s `RetryConfig.maxAttempts` needs — - * that key INCLUDES the first attempt and is a different number. The - * tombstone below says so, because the same word means two things one - * directory apart.) - * 2. The count now defaults to **0**, not 3. A pipeline that declared - * `retry: {}` used to buy three silent re-runs; it now buys none until - * the author states a count. An ETL destination is a foreign system by - * definition, so an implicit retry against a non-idempotent one is a - * duplicate write — a second invoice, a second export, a second webhook. - * That is the failure mode hardest to catch in tests and most expensive - * in production, and an unstated key is exactly where LLM-authored - * metadata hides it. - * 3. `backoffMultiplier` / `maxRetryDelayMs` / `jitter` are now declarable. - * They were the documented absence 批 12 spent three guidance entries on: - * a nightly warehouse pipeline retrying every 60s, flat and unjittered, - * is the textbook thundering herd. - * - * The base delay's default follows the shared contract (1000ms, not this - * block's old 60000ms). Nothing deployed moves: `etl.zod.ts` has no parse - * site in objectstack / objectui / cloud and an ETL pipeline is not a - * `defineStack` collection, so there is no stored pipeline for a default to - * change under. State `backoffMs` explicitly if you want the old minute. - */ - retry: strictObject({ - surface: "this ETL pipeline's retry configuration", - history: ETL_RETRY_HISTORY, - aliases: ETL_RETRY_ALIASES, - }, { - ...retryPolicyShape(), - - // ── Tombstone (ADR-0087) ────────────────────────────────────────── - // The count's pre-17 ETL spelling. Tombstoned rather than deleted even - // though this shape IS strict: an unknown-key rejection would carry the - // key, and what an upgrading author needs is the RENAME plus the warning - // that the identically-spelled connector key is a different number. - maxAttempts: retiredKey( - '`maxAttempts` was removed from an ETL pipeline\'s `retry` in @objectstack/spec 17.0.0 ' - + '(#4962) — the retry policy now has ONE vocabulary across `job.retryPolicy`, a ' - + '`try_catch` node\'s `retry`, `flow.errorHandling` and this block. Rename the key to ' - + '`maxRetries`; the NUMBER IS UNCHANGED, because this block\'s `maxAttempts` already ' - + 'counted the retries after the initial attempt. Do not subtract one — that adjustment ' - + 'belongs to `integration/connector.zod.ts`\'s `RetryConfig.maxAttempts`, which is a ' - + 'different key that INCLUDES the first attempt. Note the default also changed: an ' - + 'omitted count used to mean 3 retries and now means 0, so if you were relying on the ' - + 'old default, write `maxRetries: 3` explicitly.', - ), - }).optional().describe('Retry configuration'), - - /** - * Notification configuration - */ - notifications: strictObject({ - surface: "this ETL pipeline's notification settings", - history: ETL_NOTIFICATIONS_HISTORY, - aliases: ETL_NOTIFICATIONS_ALIASES, - }, { - onSuccess: z.array(z.string()).optional().describe('Email addresses for success notifications'), - onFailure: z.array(z.string()).optional().describe('Email addresses for failure notifications'), - }).optional().describe('Notification settings'), - - /** - * Pipeline tags for organization - */ - tags: z.array(z.string()).optional().describe('Pipeline tags'), - - /** - * Custom metadata - */ - metadata: z.record(z.string(), z.unknown()).optional().describe('Custom metadata'), -})); - -/** - * What an author writes — the annotation for a hand-written pipeline literal. - * - * `syncMode`, `enabled`, `destination.writeMode`, each transformation's - * `continueOnError`, `source.incremental.enabled` and every key of `retry` are - * optional here, and `schedule` accepts the bare cron string - * (`'0 2 * * *'`) that `CronExpressionInputSchema` wraps at parse. - */ -export type ETLPipeline = z.input; -/** - * The post-parse shape — every defaulted key present and `schedule` normalized - * to its `{ dialect: 'cron', source }` envelope. Annotate the RESULT of - * `ETLPipelineSchema.parse(…)` with this, never the literal you pass in. - */ -export type ETLPipelineParsed = z.infer; - -/** - * ETL Run Status - */ -export const ETLRunStatusSchema = lazySchema(() => z.enum([ - 'pending', // Queued for execution - 'running', // Currently executing - 'succeeded', // Completed successfully - 'failed', // Failed with errors - 'cancelled', // Manually cancelled - 'timeout', // Timed out -])); - -export type ETLRunStatus = z.input; -/** @see {@link ETLRunStatus} — the enum has no transform, so this is the same type. */ -export type ETLRunStatusParsed = z.infer; - -/** - * ETL Pipeline Run Result - * - * Result of a pipeline execution. - * - * ## Deliberately NOT strict — the wire half of this file (#4001 批 12) - * - * Everything above this line closed against unknown keys. These three shapes — - * `ETLPipelineRunSchema` and its `stats` / `error` blocks — stay tolerant, and - * this comment is the exemption record so the next sweep reads a decision - * rather than an omission. Same disposition, same reason, as - * `FlowVersionHistorySchema` in `flow.zod.ts` and the whole of - * `execution.zod.ts` ("run-state envelopes — never strict"). - * - * **Why.** Every key here is a fact the engine PRODUCES about a run that - * already happened: an id it minted, a status it reached, timestamps it - * observed, counters it accumulated, the error it caught. Nobody authors a run - * result — writing one by hand is not a use case, it is a lie about history. - * Strictness on a shape like this buys nothing (there is no author to protect - * from a silent strip) and costs the thing the campaign is most careful about: - * an engine that later reports one more counter would turn every existing - * reader's `.parse()` into a crash, which is how a tolerant wire shape becomes - * a breaking change by accident (#3712 did exactly this to `provenance` on - * `HookContextSchema`, and the ledger's `hook.zod.ts` row records the same - * split for the same reason). - * - * **How this was verified, and the limit of that verification.** The honest - * measurement is stated rather than dressed up: `etl.zod.ts` has NO parse site - * in objectstack, objectui or cloud, so neither half of this file could be - * classified by pointing at a live call. The seven above are authorable because - * the exported schema and type ARE the authoring door — `SYNC_ARCHITECTURE.md` - * and this module's `@example` both write `const p: ETLPipeline = { … }` by - * hand. These three are wire because their key set is engine-produced fact, and - * because no ETL engine exists yet the argument rests on the shape's semantics - * plus the campaign's settled precedent for exactly this pair — not on an emit - * site anyone can point at today. **The day an ETL engine lands, this comment - * is the thing to re-read**: if a run result turns out to be something an - * operator authors (a replay stub, a backfill marker), the verdict changes and - * the ledger row changes with it. - */ -export const ETLPipelineRunSchema = lazySchema(() => z.object({ - /** - * Run ID - */ - id: z.string().describe('Run identifier'), - - /** - * Pipeline name - */ - pipelineName: z.string().describe('Pipeline name'), - - /** - * Run status - */ - status: ETLRunStatusSchema.describe('Run status'), - - /** - * Start timestamp - */ - startedAt: z.string().datetime().describe('Start time'), - - /** - * End timestamp - */ - completedAt: z.string().datetime().optional().describe('Completion time'), - - /** - * Duration in milliseconds - */ - durationMs: z.number().optional().describe('Duration in ms'), - - /** - * Statistics - */ - stats: z.object({ - recordsRead: z.number().int().default(0).describe('Records extracted'), - recordsWritten: z.number().int().default(0).describe('Records loaded'), - recordsErrored: z.number().int().default(0).describe('Records with errors'), - bytesProcessed: z.number().int().default(0).describe('Bytes processed'), - }).optional().describe('Run statistics'), - - /** - * Error information - */ - error: z.object({ - message: z.string().describe('Error message'), - code: z.string().optional().describe('Error code'), - details: z.unknown().optional().describe('Error details'), - }).optional().describe('Error information'), - - /** - * Execution logs - */ - logs: z.array(z.string()).optional().describe('Execution logs'), -})); - -/** - * What a writer of a run result hands in — `stats`' four counters optional. - * - * A run result is engine-emitted, not authored (see the schema note above), so - * the pair here is about the READER's annotation, not an authoring door. It - * still follows the house convention rather than opting out, for the same - * reason `FlowVersionHistory` — the other wire shape in `automation/` — does: - * one rule for the whole namespace beats a per-shape exception nobody can - * predict from the outside. - */ -export type ETLPipelineRun = z.input; -/** The post-parse shape — `stats`' counters present, defaulted to 0. */ -export type ETLPipelineRunParsed = z.infer; - -/** - * Helper factory for creating ETL pipelines. - * - * Both helpers return {@link ETLPipeline} — the AUTHOR shape. They construct a - * literal by hand, which is the same act an author performs, so the input type - * is the correct return type; the caller passes the result to - * `ETLPipelineSchema.parse` (or hands it to an engine that will) exactly as - * they would their own literal. - * - * The annotation is textually unchanged from pre-17 but its MEANING flipped - * with the alias (#4963), and that removed two workarounds these helpers were - * carrying purely to satisfy their own return type: - * - * - **`enabled: true` is gone from both.** It stated the schema's own default - * and was written only because `z.infer` made the key required. What each - * helper still spells out is what it actually DECIDES: - * `databaseSync` is `incremental` + `upsert` (neither is the default) and - * `apiToDatabase` is `full` + `append`. The second pair does coincide with - * the defaults, and is kept deliberately — the two helpers exist as a - * contrast, and a reader comparing them must be able to see which extraction - * and write posture each one picked without going to look up two defaults. - * - **`schedule` is passed straight through.** It used to be pre-wrapped - * (`typeof s === 'string' ? { dialect: 'cron', source: s } : s`) because - * `z.infer` of `CronExpressionInputSchema` is the post-transform envelope - * and would not accept the bare cron string these helpers advertise. The - * union IS the input type, so the normalization belongs where it always - * did — in the schema, at parse. - */ -export const ETL = { - /** - * Create a simple database-to-database pipeline - */ - databaseSync: (params: { - name: string; - sourceTable: string; - destTable: string; - schedule?: import("../shared/expression.zod").CronExpressionInput; - }): ETLPipeline => ({ - name: params.name, - source: { - type: 'database', - config: { table: params.sourceTable }, - }, - destination: { - type: 'database', - config: { table: params.destTable }, - writeMode: 'upsert', - }, - syncMode: 'incremental', - schedule: params.schedule, - }), - - /** - * Create an API to database pipeline - */ - apiToDatabase: (params: { - name: string; - apiConnector: string; - destTable: string; - schedule?: import("../shared/expression.zod").CronExpressionInput; - }): ETLPipeline => ({ - name: params.name, - source: { - type: 'api', - connector: params.apiConnector, - config: {}, - }, - destination: { - type: 'database', - config: { table: params.destTable }, - writeMode: 'append', - }, - syncMode: 'full', - schedule: params.schedule, - }), -} as const; diff --git a/packages/spec/src/automation/index.ts b/packages/spec/src/automation/index.ts index d625af7836..f675c92f51 100644 --- a/packages/spec/src/automation/index.ts +++ b/packages/spec/src/automation/index.ts @@ -12,7 +12,25 @@ export { flowForm } from './flow.form'; export * from './execution.zod'; export * from './webhook.zod'; export * from './approval.zod'; -export * from './etl.zod'; +// `etl.zod.ts` (L2 "ETL Pipeline": ETLPipeline, ETLPipelineRun, their +// source/destination/transformation vocabulary, the ETLEndpointType / +// ETLTransformationType / ETLSyncMode / ETLRunStatus enums and the `ETL` +// factory) was removed here (#6414, ADR-0049 enforce-or-remove, protocol 17). +// The reading is the one #4738 used to retire L1 one layer up, re-measured on +// L2 and identical: narrative-only. No engine ever parsed, scheduled or ran an +// ETLPipeline; the schema had zero importers across objectstack, objectui and +// cloud outside spec's own tests, no `liveness/` ledger row (the neighbouring +// `mapping.json` exists precisely because import mapping's `transform` IS +// executed row by row), and no def reachable from a metadata-type root. +// Layer-by-layer, that leaves ONE surviving sync layer rather than a gap: +// connector-attached sync is `ConnectorSchema.syncConfig` +// (integration/connector.zod.ts), the live parse path. Per-field value +// transformation at import is `shared/mapping.zod.ts`. Multi-source, multi-stage +// movement has no protocol surface at all now — deliberately, because it had no +// executor: it returns through the ENFORCE route, engine first, vocabulary +// second. `packages/spec/docs/SYNC_ARCHITECTURE.md` was rewritten in the same +// change; a doc that still recommended L2 as L1's destination would have made +// the retirement self-contradictory. // `trigger-registry.zod` was removed here (#4499). Despite the filename it // contained no trigger registry — all 630 lines were a third declaration of // the connector vocabulary (ConnectorSchema, Authentication*, Operation*, @@ -28,7 +46,10 @@ export * from './time-relative-trigger.zod'; // objectui, no engine ever parsed or executed a DataSyncConfig, and the def was // unreachable from the metadata-type roots (#4650 gate). Connector-attached // sync config is `ConnectorSchema.syncConfig` (integration/connector.zod.ts, -// the live parse path); multi-step transformation is `etl.zod.ts`. The bare +// the live parse path). ⚠️ This note used to send readers on to `etl.zod.ts` +// for multi-step transformation; L2 was retired for the same narrative-only +// reason at #6414, so that pointer is gone rather than re-aimed — there is no +// third layer to forward to. The bare // `ConflictResolution` name went to `@objectstack/spec/ui` (offline sync) at // #4738 — and left the package entirely at #4988, which retired // `ui/offline.zod.ts` under ADR-0049. The connector vocabulary keeps its diff --git a/packages/spec/src/automation/sync-retirement.test.ts b/packages/spec/src/automation/sync-retirement.test.ts index bfb7111de6..05416ab6ca 100644 --- a/packages/spec/src/automation/sync-retirement.test.ts +++ b/packages/spec/src/automation/sync-retirement.test.ts @@ -133,8 +133,29 @@ describe('[#4738] sync/conflict dual-source retirement', () => { ]) { expect(automationNames, `./automation must not export ${retired}`).not.toContain(retired); } - expect(automationNames).toContain('ETLPipelineSchema'); + // ⚠️ The L2 anchor that stood here — `expect(automationNames).toContain( + // 'ETLPipelineSchema')` — was DELETED, not re-spelled, at #6414. It was + // written to prove the L1 retirement stopped at L1, and it did that job for + // four months; then L2 was retired on the same narrative-only reading, so + // keeping it would have asserted the survival of a layer this repo + // deliberately removed. Re-pointing it at another `automation/` export + // would have preserved the line and lost the meaning. What survives as the + // "did not over-reach" witness is `StateMachineSchema` plus the >50 export + // floor above — and, one layer out, the surviving sync surfaces are + // asserted by name in section 4 below. expect(automationNames).toContain('StateMachineSchema'); + for (const alsoRetired of [ + 'ETLPipeline', 'ETLPipelineSchema', 'ETLPipelineRun', 'ETLPipelineRunSchema', + 'ETLSource', 'ETLSourceSchema', 'ETLDestination', 'ETLDestinationSchema', + 'ETLTransformation', 'ETLTransformationSchema', 'ETLEndpointTypeSchema', + 'ETLTransformationTypeSchema', 'ETLSyncModeSchema', 'ETLRunStatusSchema', + 'ETL', + ]) { + expect( + automationNames, + `./automation must not export ${alsoRetired} (#6414, L2 retired on L1's reading)`, + ).not.toContain(alsoRetired); + } // 2. The renamed side: `ConnectorConflictResolution(Schema)` originates in // integration/connector.zod.ts and is exported by ./integration alone diff --git a/packages/spec/src/integration/connector-author-shape.test.ts b/packages/spec/src/integration/connector-author-shape.test.ts index d016eb988a..1aecde7815 100644 --- a/packages/spec/src/integration/connector-author-shape.test.ts +++ b/packages/spec/src/integration/connector-author-shape.test.ts @@ -21,11 +21,14 @@ import { // REJECTS. It was filed as #5515 rather than fixed there because the owner // schema is this file's, not `automation/etl.zod.ts`'s. This is that gate. // -// The two gates are deliberately SEPARATE — same document, different owning -// schema — and so is the ~40-line harness below, which is a near-copy of the -// sibling's. Sharing it would mean a third module imported by both; at two -// call sites the duplication is cheaper than the indirection, and each gate -// stays readable end to end. A third such gate is the point to extract. +// ⚠️ That sibling gate is GONE as of #6414: the L2 ETL layer it guarded was +// retired under ADR-0049 (no executor, ever), and a gate over a deleted schema +// has nothing left to assert. This is now the document's ONLY compile gate, so +// it absorbed the one assertion of the sibling's that was not about ETL — the +// TOTAL ```typescript block count, which is what makes adding a block to +// SYNC_ARCHITECTURE.md a deliberate act. The ~40-line harness below stays +// exactly where it was; it was a near-copy of the sibling's, and with the +// sibling gone it is simply the harness. // // ## Why the compiler API instead of type-level pins // @@ -142,32 +145,50 @@ function typescriptBlocks(markdown: string): string[] { */ const ELISION = /\.\.\.\s*[,}\]]/; -describe('[#5515] SYNC_ARCHITECTURE.md L3 connector example compiles', () => { +describe('[#5515] SYNC_ARCHITECTURE.md L3 connector examples compile', () => { const markdown = readFileSync(SYNC_ARCHITECTURE, 'utf8'); - const connectorBlocks = typescriptBlocks(markdown).filter((b) => b.includes('Connector')); + const allBlocks = typescriptBlocks(markdown); + const connectorBlocks = allBlocks.filter((b) => b.includes('Connector')); const sketches = connectorBlocks.filter((b) => ELISION.test(b)); const compilable = connectorBlocks.filter((b) => !ELISION.test(b)); - it('finds the example this gate exists for, and classifies the ones it skips', () => { + it('finds the examples this gate exists for, and classifies the ones it skips', () => { // Anti-vacuity, in both directions: a selector that matched nothing would // make the compile assertion below pass over an empty program (the way a // gate goes dormant), and a sketch counted as compilable would fail it for // a reason that is not about the schema. - expect(connectorBlocks.length, '`Connector` examples in SYNC_ARCHITECTURE.md').toBe(3); - // The two skipped ones are the Migration Guide's "Before (L3 `syncConfig`)" - // and "After (L3)" fragments, which elide with a bare `...`. They are - // exempt on their FORM (not TypeScript), never on their merits — the whole - // lesson of #5515 is that "it's only a doc snippet" is how four rejected - // spellings survived in a file authors copy from. - expect(sketches.length, 'Migration-Guide sketches that elide with `...`').toBe(2); - expect(compilable.length, 'the full `sapConnector` example').toBe(1); - // The document's TOTAL ```typescript count is pinned by the sibling gate - // (`automation/etl-author-shape.test.ts`), which is what makes ADDING a - // block a deliberate act; this gate pins the `Connector` slice of it. + // + // ⚠️ This block ABSORBED the total-count pin at #6414. It used to end with + // "the document's TOTAL ```typescript count is pinned by the sibling gate + // (`automation/etl-author-shape.test.ts`)" — and that sibling was deleted + // with the L2 layer it guarded. A deleted gate takes its assertions with it + // silently, which is the one way a documentation gate fails without anyone + // seeing red, so the total-count pin moves HERE rather than lapsing. That + // is what still makes adding a block to this document a deliberate act. + expect(allBlocks.length, 'total ```typescript blocks — classify any new one').toBe(2); + expect(connectorBlocks.length, '`Connector` examples in SYNC_ARCHITECTURE.md').toBe(2); + // Zero sketches TODAY, and the classifier is kept anyway. The two it used + // to exempt were the Migration Guide's "Before (L3 `syncConfig`)" / "After + // (L3)" fragments, which #6414 replaced when the guide's direction reversed + // (there is no longer an L2 to migrate to). They were exempt on their FORM + // (not TypeScript), never on their merits — the lesson of #5515 is that + // "it's only a doc snippet" is how four rejected spellings survived in a + // file authors copy from, so a re-introduced sketch must still be + // classified rather than silently compiled. + expect(sketches.length, 'Migration-Guide sketches that elide with `...`').toBe(0); + expect(compilable.length, 'full, compilable connector examples').toBe(2); + // Also pinned deliberately: the L2 "Before" snippet in the Migration Guide + // is fenced as PLAIN text, not `typescript`, because `ETLPipeline` no + // longer exists and the snippet is shown precisely as the thing that no + // longer compiles. If someone re-fences it as `typescript`, the total above + // goes to 3 and this gate says so. + expect(markdown).toContain("const pipeline: ETLPipeline = {"); + expect(markdown).not.toContain("```typescript\nimport type { ETLPipeline }"); }); - it('compiles the `sapConnector` example verbatim, import line included', () => { - const probes: Record = { 'doc-l3-connector': compilable[0]! }; + it('compiles every full connector example verbatim, import line included', () => { + const probes: Record = {}; + compilable.forEach((block, i) => { probes[`doc-l3-connector-${i}`] = block; }); // The harness's own control: a probe that MUST fail. Without it a // resolution failure (paths mapping wrong, host overlay not applied) would // report zero diagnostics and read as a green example. @@ -179,7 +200,12 @@ describe('[#5515] SYNC_ARCHITECTURE.md L3 connector example compiles', () => { const results = compileProbes(probes); expect(render(results.get('harness-self-test')!), 'the harness must be able to report an error') .toContain('TS2739'); - expect(render(results.get('doc-l3-connector')!), 'the L3 example must compile clean').toBe(''); + compilable.forEach((_, i) => { + expect( + render(results.get(`doc-l3-connector-${i}`)!), + `L3 example #${i} must compile clean`, + ).toBe(''); + }); }); }); diff --git a/packages/spec/src/integration/connector.zod.ts b/packages/spec/src/integration/connector.zod.ts index 9c3f7a49e6..77835cb883 100644 --- a/packages/spec/src/integration/connector.zod.ts +++ b/packages/spec/src/integration/connector.zod.ts @@ -14,10 +14,12 @@ import { retiredKey } from '../shared/retired-key'; * Connectors enable ObjectStack to sync data with SaaS apps, databases, file storage, * and message queues through a unified protocol. * - * **Positioning in the sync/integration layering** (L1 "Simple Sync" was - * retired in #4738 — narrative-only, zero consumers; see - * `packages/spec/docs/SYNC_ARCHITECTURE.md`): - * - **ETL Pipeline** (automation/etl.zod.ts) - Data engineers - Aggregate 10 sources to warehouse + * **Positioning in the sync/integration layering** — this file is now the ONLY + * layer. Both layers above it were retired under ADR-0049 for the same measured + * reason, that no engine ever executed them: L1 "Simple Sync" + * (`automation/sync.zod.ts`) in #4738, and L2 "ETL Pipeline" + * (`automation/etl.zod.ts`) in #6414. See + * `packages/spec/docs/SYNC_ARCHITECTURE.md`: * - **Enterprise Connector** (THIS FILE) - System integrators - Full SAP integration; connector-attached sync via `syncConfig` * * **SCOPE: Most comprehensive integration layer.** @@ -98,9 +100,11 @@ import { retiredKey } from '../shared/retired-key'; * - Microsoft Dynamics 365 connector * * **When to downgrade:** - * - Data transformation only → Use {@link file://../automation/etl.zod.ts | ETL Pipeline} - * - * @see {@link file://../automation/etl.zod.ts} for the ETL Pipeline layer (data engineering) + * - Per-field value conversion on import only → the import mapping's own + * `transform` (`data/mapping.zod.ts`), which the REST import path executes + * row by row. (This used to point at `automation/etl.zod.ts`; L2 was retired + * at #6414 for having no executor, so the pointer would have been a signpost + * landing nowhere — the same defect class this header names below.) * * ## There is no "Trigger Registry" alternative * @@ -113,9 +117,11 @@ import { retiredKey } from '../shared/retired-key'; * per-provider template cluster). The same defect class as the * `capabilities.readOnly` prescription #4487 corrected: a signpost must land * somewhere enforced. Lightweight cases are served HERE — a connector instance - * with simple `auth` — or by `automation/etl.zod.ts` for transformation - * pipelines. (The automation-side L1 "Simple Sync" file was itself retired as - * a dead end of the same class in #4738.) + * with simple `auth`. (Both automation-side layers were themselves retired as + * dead ends of the same class: L1 "Simple Sync" in #4738, L2 `etl.zod.ts` in + * #6414. This paragraph named L2 as the transformation destination until the + * second retirement; a signpost that must land somewhere enforced cannot make + * an exception for itself.) */ // ============================================================================ diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 06a9493d9f..3eb69d5972 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -1224,36 +1224,16 @@ const step17: MigrationStep = { + 'retry re-runs the handler with its writes and callouts. No job fails to register ' + 'with the retry-policy bound prescription.', }, - { - id: 'etl-retry-converged-onto-retry-policy', - surface: 'etlPipeline.retry.maxAttempts (and any count above 10)', - replacement: 'maxRetries, same number — plus an explicit count if you relied on the old default of 3', - reason: - 'An ETL pipeline\'s `retry` was a THIRD retry vocabulary that #4661\'s convergence never ' - + 'reached, because that pass was driven by duplicated exported NAMES and this block is an ' - + 'anonymous inline object (#4962). It now carries the shared `RetryPolicySchema` contract, ' - + 'which changes three things with no single lossless rewrite between them. The rename ' - + '`maxAttempts` → `maxRetries` IS lossless and the tombstone performs it — both keys ' - + 'counted the retries AFTER the initial attempt, so the number does not change, and ' - + 'subtracting one (correct for `integration/connector.zod.ts`\'s identically-spelled ' - + '`RetryConfig.maxAttempts`, which includes the first attempt) would silently run one ' - + 'attempt fewer than asked. What needs a human: the count now DEFAULTS TO 0 instead of 3, ' - + 'so a pipeline that wrote `retry: {}` or omitted the count bought three silent re-runs ' - + 'and now buys none. That is deliberate and the business case is the destination — an ETL ' - + 'destination is a foreign system by definition, and an implicit retry against a ' - + 'non-idempotent one is a duplicate write (a second invoice, a second export, a second ' - + 'webhook). Retrying is now something an author states and thereby claims idempotency for. ' - + 'The shared contract also caps `maxRetries` at 10, which this block never did; clamping ' - + 'a larger budget would silently halve a number its author chose, so it fails at parse ' - + 'with the bound named instead.', - acceptanceCriteria: - 'No ETL pipeline declares `retry.maxAttempts`; every one that wants retries declares ' - + '`maxRetries` >= 1 explicitly (the number carried over unchanged from `maxAttempts`), and ' - + 'every pipeline that was relying on the old implicit 3 has either written `maxRetries: 3` ' - + 'or been re-decided against the duplicate-write risk at its destination. No count exceeds ' - + '10. Pipelines that want the old flat 60s backoff state `backoffMs: 60000` explicitly, ' - + 'since the shared default is 1000.', - }, + // `etl-retry-converged-onto-retry-policy` (#4962) stood here and was + // ABSORBED by `etl-pipeline-layer-retired` below (#6414), the §0 same-major + // rule: both land in the unreleased protocol 17, and composed, the rename + // `ETLPipeline.retry.maxAttempts` -> `maxRetries` has no observable effect + // because the shape carrying it does not survive the major. Leaving both + // would tell an upgrader to rewrite a key on a schema this same upgrade + // deletes, and would break the fixture-disjointness the replay contract + // asserts. The `agent.knowledge` / `WidgetManifest.performance` precedent: + // a tombstone goes with the shape that carried it, which is strictly + // stronger than the tombstone. { id: 'flow-retry-max-retries-required', surface: "flow.errorHandling.maxRetries (under strategy: 'retry')", @@ -2473,6 +2453,171 @@ const step17: MigrationStep = { + 'envelope or a FilterArray in that slot still compiles and is rejected at run time ' + 'with INVALID_FILTER / 400.', }, + { + id: 'http-server-runtime-vocabulary-retired', + surface: + 'system.serverEvent / system.serverEventType / system.serverCapabilities / ' + + 'system.serverStatus (the lifecycle-event, capability-report and status vocabulary of ' + + 'system/http-server.zod.ts — 4 defs, 8 exported names)', + replacement: + '(removed — there is no replacement key, because there was never a key. Server lifecycle ' + + 'is the transport plugin\'s own start/stop seam; per-request and per-server ' + + 'observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus ' + + '`OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a ' + + 'transport plugin can DO it states by implementing the kernel plugin contract — the ' + + 'seams it registers are the capability statement, and a self-described capability ' + + 'record can only disagree with them. Server-level configuration that IS authorable ' + + 'lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)', + reason: + 'The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the ' + + 'CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring ' + + 'entry); this removes the RUNTIME half — a 7-member lifecycle event union with a ' + + 'timestamped envelope, an eight-boolean capability report, and a five-state status ' + + 'record with connection and request counters. Nothing ever emitted, consumed or ' + + 'parsed any of them. ' + + 'This card was HELD for four days rather than queued, on a specific and legitimate ' + + 'doubt: a response/capability vocabulary can be a REFERENCE surface for host ' + + 'implementers, so "zero consumers in this repo" is weaker evidence for one of those ' + + 'than for an authorable key (the CSS-variable rebuttal). The hold was lifted by ' + + 'measuring the reference reader itself rather than by re-running the same grep: ' + + '`plugin-hono-server`, the one in-tree host implementation, neither implements nor ' + + 'reports any of the three — it names no capability record, no status shape and no ' + + 'event union, and what it registers is routes and middleware through the kernel ' + + 'plugin contract. A declaration-site grep put every declaration in this one file, a ' + + 'quoted-name sweep across objectstack and objectui found no reader outside it, and ' + + 'the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, ' + + 'resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in ' + + 'this file when there was one. ' + + 'With no carrier key there is nothing to tombstone, and with no author there is no ' + + 'source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR ' + + 'plus this entry are the declaration — route 3, the same shape as #4938 in this very ' + + 'file, #4834, #4988 and #5055. If host-implementer conformance becomes a real ' + + 'requirement it returns through the ENFORCE route: an adapter contract with a checker ' + + 'behind it, vocabulary second. ADR-0049, #5295.', + acceptanceCriteria: + 'No source imports `ServerEvent`, `ServerEventType`, `ServerEventSchema`, ' + + '`ServerCapabilities`, `ServerCapabilitiesSchema`, `ServerCapabilitiesParsed`, ' + + '`ServerStatus` or `ServerStatusSchema` from `@objectstack/spec/system` — a grep over ' + + 'consumer code resolves none of them, and `tsc` reports TS2724/TS2305 on any that ' + + 'survives. The route-registration half of the same module still resolves ' + + '(`RouteHandlerMetadataSchema`, `MiddlewareType`, `MiddlewareConfigSchema`, ' + + '`MiddlewareConfig`), and `StackServerConfigSchema` — the one authorable server ' + + 'surface — is untouched: a stack declaring `server: { trustProxy, security }` parses ' + + 'exactly as it did in 16.x.', + }, + { + id: 'view-management-protocol-retired', + surface: + 'api.listViews / api.getView / api.createView / api.updateView / api.deleteView ' + + '(the ViewProtocol interface and its ten Request/Response schemas in ' + + 'api/protocol.zod.ts — 10 defs, 25 exported names)', + replacement: + 'the two view surfaces that are actually routed. For a view\'s STORED definition, the ' + + 'generic metadata methods with `type: \'view\'` — `getMetaItem` / `getMetaItems` / ' + + '`saveMetaItem` / `deleteMetaItem`, served at `/api/v1/meta/view/:name`. For the ' + + 'RESOLVED render-time view, `getUiView` (`GetUiViewRequest` / `GetUiViewResponse`), ' + + 'served at `/api/v1/ui/view/:object/:type`. Neither is addressed by a `viewId`, which ' + + 'is the one thing the retired surface offered and the one thing nothing implemented', + reason: + 'A complete viewId-addressed CRUD surface — list (with a list/form filter), read, ' + + 'create, patch, delete — with none of the three things a protocol method needs. ' + + 'Measured on origin/main immediately before the removal: no implementation ' + + '(`packages/metadata-protocol/src/protocol.ts` declares no `listViews` / `getView` / ' + + '`createView` / `updateView` / `deleteView`; its only view resolver is `getUiView`), ' + + 'no route (`packages/rest/src/rest-server.ts` never mentions `viewId`, so nothing ' + + 'viewId-addressed is reachable over HTTP at all), and no caller (the only ' + + '`ViewProtocol` mention outside its own file was the services checklist, which ' + + 'already recorded the five as declared-and-unrouted). The look-alike hits a bare-name ' + + 'grep turns up are all different contracts: `metadata-manager.ts`\'s ' + + '`getView(name: string)` is another class, and objectui\'s ' + + '`getView(objectName, viewId)` resolves through `client.meta.getItem(\'view\', …)`, ' + + 'i.e. the metadata route. ' + + 'What makes this worth a removal rather than a note is that the cost is already ' + + 'measured. A declared surface that is name-identical and semantics-adjacent to a real ' + + 'one is an attractive nuisance in every grep, and it mis-directed a decision once: ' + + '#5948\'s issue body AND its 2026-08-07 maintainer ruling both read ' + + '`GetViewResponseSchema` (zero implementations) as the contract of ' + + '`GET /ui/view/:object/:type`, whose declared response is `GetUiViewResponseSchema` — ' + + 'one word apart, 250 lines up. That ruling\'s reasoning happened to survive the ' + + 'mix-up ("nobody can consume `{object, view}` successfully today" was true, though ' + + 'not for the stated reason), which is the luck this removal stops relying on. ' + + 'Route 3: none of the ten was a key on an authorable shape, nothing parsed them, so ' + + 'there is no tombstone and no D2 conversion — RETIRED_DEFS_BY_MAJOR plus this entry ' + + 'are the declaration. If reading and writing ONE view by id becomes a real ' + + 'requirement it returns implementation-first. ADR-0049, ADR-0087, maintainer ruling ' + + '2026-08-07, #6239.', + acceptanceCriteria: + 'No source imports `ListViewsRequest(Schema)`, `ListViewsResponse(Schema)`, ' + + '`GetViewRequest(Schema)`, `GetViewResponse(Schema)`, `CreateViewRequest(Schema)`, ' + + '`CreateViewResponse(Schema)`, `UpdateViewRequest(Schema)`, ' + + '`UpdateViewResponse(Schema)`, `DeleteViewRequest(Schema)` or ' + + '`DeleteViewResponse(Schema)` from `@objectstack/spec/api`, and no host declares a ' + + '`ViewProtocol` member. Reading and writing views still works end to end through the ' + + 'surfaces that were always the live ones: `GET /api/v1/meta/view/:name` returns the ' + + 'stored definition and `GET /api/v1/ui/view/:object/:type` returns the resolved view, ' + + 'both unchanged by this removal. `GetUiViewRequestSchema` / `GetUiViewResponseSchema` ' + + 'still resolve — they are the shapes #5948 meant.', + }, + { + id: 'etl-pipeline-layer-retired', + surface: + 'automation.etlPipeline / automation.etlPipelineRun / automation.etlSource / ' + + 'automation.etlDestination / automation.etlTransformation (the whole L2 layer of ' + + 'automation/etl.zod.ts, its four enums and the `ETL` factory — 9 defs, 27 exported ' + + 'names)', + replacement: + '(removed — no protocol surface replaces it, deliberately. Layer by layer: ' + + 'connector-attached synchronisation is `ConnectorSchema.syncConfig` ' + + '(`integration/connector.zod.ts`), which IS parsed and executed; per-field value ' + + 'transformation on import is `shared/mapping.zod.ts`, whose `transform` is applied ' + + 'row by row by the REST import path and recorded key by key in ' + + '`packages/spec/liveness/mapping.json`; scheduling is `system/job.zod.ts`. What has ' + + 'NO replacement is multi-source, multi-stage movement with joins and aggregations — ' + + 'because it never had an implementation either. It returns through the ENFORCE route: ' + + 'the engine first, the vocabulary second)', + reason: + 'The reading #4738 used to retire L1 `DataSyncConfig`, re-measured one layer up and ' + + 'identical: narrative-only. No engine ever parsed, scheduled or executed an ' + + '`ETLPipeline`. Measured on origin/main immediately before the removal: the only ' + + 'non-spec references in this repo are two fumadocs-generated documentation sources ' + + '(`apps/docs/.source/*.ts`), not executors; objectui has no reference at all; there ' + + 'is no `liveness/etl.json` or `pipeline.json`, so no ADR-0049 gate ever had a reading ' + + 'on it — while the same file family\'s EXECUTED half does have one ' + + '(`liveness/mapping.json`), which is the contrast that makes the absence meaningful ' + + 'rather than an oversight. The `etl` string in this registry was the one untested ' + + 'link the finding named, and it is not a loader path: it was the id of the #4962 ' + + 'retry-vocabulary entry, absorbed here. ' + + 'The layer was ADR-0078\'s asymmetry in its purest form — an author could write a ' + + 'complete ten-stage pipeline, get no error, and get no execution. It was also ' + + 'advertised: `packages/spec/docs/SYNC_ARCHITECTURE.md` named `ETLPipeline` as the ' + + 'recommended destination for authors displaced by the L1 retirement (#4738) and ' + + 'listed ten transformation types with copyable examples down to ' + + '`script | Custom JavaScript/Python`. That document is rewritten in the same change; ' + + 'a retirement whose own doc still recommends the retired layer is self-contradictory, ' + + 'and forwarding L1\'s authors to a second layer with no executor was the defect ' + + 'compounding rather than closing. ' + + '⚠️ `etl-retry-converged-onto-retry-policy` (#4962) is SUBSUMED here, the ' + + '#4657/#4834/#5055 way: both land in the unreleased protocol 17, so composed, a ' + + 'rename of `retry.maxAttempts` on a shape that does not survive the major has no ' + + 'observable effect — and keeping both would tell an upgrader to rewrite a key on a ' + + 'schema the same upgrade deletes. The `maxAttempts` `retiredKey()` tombstone goes ' + + 'with the shape that carried it, which is strictly stronger than the tombstone: there ' + + 'is no longer a `retry` block to author the key into. Route 3 — no carrier key, no ' + + 'parse site, so no D2 conversion and no tombstone; RETIRED_DEFS_BY_MAJOR plus this ' + + 'entry are the declaration. ADR-0049, ADR-0078, #6414.', + acceptanceCriteria: + 'No source imports `ETLPipeline`, `ETLPipelineParsed`, `ETLPipelineSchema`, ' + + '`ETLPipelineRun(Schema)`, `ETLSource(Schema)`, `ETLDestination(Schema)`, ' + + '`ETLTransformation(Schema)`, `ETLEndpointType(Schema)`, ' + + '`ETLTransformationType(Schema)`, `ETLSyncMode(Schema)`, `ETLRunStatus(Schema)` or ' + + 'the `ETL` factory from `@objectstack/spec/automation`; `tsc` reports TS2724/TS2305 ' + + 'on any that survives. Every author who was pointed at L2 has been re-pointed by ' + + 'name: SYNC_ARCHITECTURE.md no longer lists an L2 row, no longer recommends ' + + '`ETLPipeline` as L1\'s destination and no longer advertises a transformation-type ' + + 'table. The surviving layers still parse unchanged — a connector declaring ' + + '`syncConfig` and an import declaring `mapping.transform` both behave exactly as they ' + + 'did in 16.x.', + }, ], }; @@ -2697,6 +2842,21 @@ export const RETIRED_DEFS_BY_MAJOR: Readonly> // contract, not authorable metadata, and it has a live compile-time consumer // in objectui (`packages/fields/src/__tests__/spec-symbol-batch7.test.ts`, // landed by objectui PR #3289). + // + // The 2026-08-08 ADR-0049 sweep (#6486) adds twenty-three more across three + // members (4 + 10 + 9), all route 3 and all whole-def: `system/http-server.zod.ts`'s + // runtime vocabulary (#5295, D3 `http-server-runtime-vocabulary-retired`), + // `api/protocol.zod.ts`'s viewId-addressed view CRUD (#6239, D3 + // `view-management-protocol-retired`) and the whole L2 ETL layer (#6414, D3 + // `etl-pipeline-layer-retired`). None had a carrier key and none was ever + // parsed outside its own unit tests, so again there is no tombstone and no D2 + // conversion — this table plus those three entries ARE the declaration. + // + // ⚠️ `system/ServerRateLimitConfig` is deliberately NOT here. It sits four + // lines from the retired `system/ServerCapabilities` in the same manifest and + // shares its prefix, but it belongs to `StackServerSecurity.rateLimit` — the + // LIVE server surface #5006 admitted, with an executor. Prefix adjacency is + // not evidence. 17: [ 'shared/FieldMappingTransform', 'ui/WidgetManifest', @@ -2709,5 +2869,31 @@ export const RETIRED_DEFS_BY_MAJOR: Readonly> 'ui/NumberFormat', 'ui/DateFormat', 'ui/LocaleConfig', + // #5295 — system/http-server.zod.ts runtime vocabulary + 'system/ServerEvent', + 'system/ServerEventType', + 'system/ServerCapabilities', + 'system/ServerStatus', + // #6239 — api/protocol.zod.ts view-management operations + 'api/ListViewsRequest', + 'api/ListViewsResponse', + 'api/GetViewRequest', + 'api/GetViewResponse', + 'api/CreateViewRequest', + 'api/CreateViewResponse', + 'api/UpdateViewRequest', + 'api/UpdateViewResponse', + 'api/DeleteViewRequest', + 'api/DeleteViewResponse', + // #6414 — automation/etl.zod.ts, the whole L2 layer + 'automation/ETLPipeline', + 'automation/ETLPipelineRun', + 'automation/ETLSource', + 'automation/ETLDestination', + 'automation/ETLTransformation', + 'automation/ETLEndpointType', + 'automation/ETLTransformationType', + 'automation/ETLSyncMode', + 'automation/ETLRunStatus', ], }; diff --git a/packages/spec/src/shared/alias-integrity.test.ts b/packages/spec/src/shared/alias-integrity.test.ts index a4932c57f5..9d5925db8c 100644 --- a/packages/spec/src/shared/alias-integrity.test.ts +++ b/packages/spec/src/shared/alias-integrity.test.ts @@ -31,9 +31,12 @@ * no source-literal reader can see, so the target check has to be suppressed * for most of the interesting schemas. `.shape` sees them. * 2. **Assembled tables.** Ten call sites build `aliases` (or `surface`) from - * something other than a literal — `data/field.zod.ts`, `ui/theme.zod.ts`, - * `automation/etl.zod.ts` and others. The AST reads those as empty and - * reports them clean. + * something other than a literal — `data/field.zod.ts`, `ui/theme.zod.ts` + * and others. The AST reads those as empty and reports them clean. + * (`automation/etl.zod.ts` was one of the ten when this was measured; the + * whole L2 layer was retired at #6414, which is why the count above is kept + * as the measurement it was rather than silently decremented — the argument + * is about the AST's blind spot, and it does not get weaker by one file.) * 3. **Colliding surfaces.** `'this field group'` names two different schemas * (`data/object.zod.ts`, `studio/object-designer.zod.ts`), so the surface * string is not a key and the hand-map silently judges one against the diff --git a/packages/spec/src/shared/retry-policy.test.ts b/packages/spec/src/shared/retry-policy.test.ts index 118d50fcc0..bb270ba074 100644 --- a/packages/spec/src/shared/retry-policy.test.ts +++ b/packages/spec/src/shared/retry-policy.test.ts @@ -117,6 +117,12 @@ describe('RetryPolicySchema — converged shape', () => { * a surviving dialect after a completed convergence reads as reviewed-and-kept * rather than missed. * + * `ETLPipeline.retry` no longer exists: the whole L2 ETL layer was retired at + * #6414 for having no executor. Its convergence is kept in this history + * deliberately — the two dialects were found by asking the concept-level + * question, and that is the instrument this block still is, whatever it happens + * to be pointed at today. + * * This block asks the concept-level question directly, against the four * surfaces that carry a retry policy. It is deliberately a PARSE comparison * rather than a source or `.shape` inspection: it is blind to how a surface @@ -136,23 +142,21 @@ describe('every retry surface carries ONE contract (#4661, #4964, #4962)', () => name: 'f', label: 'F', type: 'autolaunched' as const, nodes: [{ id: 'n1', type: 'start', label: 'S' }], edges: [], }; - const minimalPipeline = { - name: 'p', label: 'P', - source: { type: 'api' as const, connector: 'sf', config: {} }, - destination: { type: 'database' as const, connector: 'pg', config: {} }, - }; - - /** The parsed retry region of each surface, given an EMPTY authored block. */ + // ⚠️ THREE surfaces became TWO at #6414, and the subtraction is the reason + // this comment exists. `etlPipeline.retry` — the third encoding this block was + // written to catch (#4962) — left with the whole L2 ETL layer, retired for + // having no executor at all. The remaining two are the real ones, and the + // block's guarantee is unchanged: a FOURTH surface added without wiring + // `retryPolicyShape()` still fails here. What would NOT be caught, and is + // worth stating rather than discovering: a surface that never gets added, + // because a vocabulary with no engine behind it is invisible to a parse + // comparison. That is the ADR-0049 question, not this block's. const surfaces = (): ReadonlyArray]> => [ ['job.retryPolicy / try_catch retry', RetryPolicySchema.parse({}) as Record], [ 'flow.errorHandling', Automation.FlowSchema.parse({ ...minimalFlow, errorHandling: {} }).errorHandling as Record, ], - [ - 'etlPipeline.retry', - Automation.ETLPipelineSchema.parse({ ...minimalPipeline, retry: {} }).retry as Record, - ], ]; it('declares the same key set everywhere (modulo flow-only `strategy`)', () => { @@ -166,7 +170,8 @@ describe('every retry surface carries ONE contract (#4661, #4964, #4962)', () => it('applies the same defaults everywhere — including the opt-in count of 0', () => { // The half no gate can see: `authorable-surface.json` compares key sets and // a default is not a key. `ETLPipeline.retry` defaulted the count to 3 - // until #4962 while every sibling defaulted 0, and nothing failed. + // until #4962 while every sibling defaulted 0, and nothing failed. (That + // surface is gone as of #6414; the blind spot it demonstrated is not.) for (const [label, parsed] of surfaces()) { for (const [key, value] of Object.entries(POLICY_DEFAULTS)) { expect(parsed[key], `${label}.${key} must default to ${value}`).toBe(value); @@ -174,22 +179,24 @@ describe('every retry surface carries ONE contract (#4661, #4964, #4962)', () => } }); - it('retires the two pre-17 spellings wherever they were legal', () => { - // `retryDelayMs` (automation base delay) and `maxAttempts` (the ETL count). - // Both tombstoned rather than deleted, so the rejection carries the rename - // instead of a bare "unrecognized key" — or, on the non-strict surfaces, a - // silent strip back to the default. + it('retires the pre-17 spelling wherever it was legal', () => { + // `retryDelayMs`, the automation base delay: tombstoned rather than + // deleted, so the rejection carries the rename instead of a bare + // "unrecognized key" — or, on the non-strict surfaces, a silent strip back + // to the default. + // + // The ETL half of this test asserted the `maxAttempts` -> `maxRetries` + // tombstone on `ETLPipeline.retry`. It is deleted rather than re-pointed at + // another surface, because the tombstone went with the shape that carried + // it (#6414 absorbed #4962's conversion entry for exactly this reason): with + // no `retry` block to author the key INTO, a prescription for it is a + // message nobody can receive. Re-spelling this case onto `flow` would have + // duplicated the assertion above while looking like preserved coverage. const flowRetired = Automation.FlowSchema.safeParse({ ...minimalFlow, errorHandling: { strategy: 'retry', maxRetries: 2, retryDelayMs: 500 }, }); expect(flowRetired.success).toBe(false); expect(JSON.stringify(flowRetired.error!.issues)).toMatch(/backoffMs/); - - const etlRetired = Automation.ETLPipelineSchema.safeParse({ - ...minimalPipeline, retry: { maxAttempts: 3 }, - }); - expect(etlRetired.success).toBe(false); - expect(JSON.stringify(etlRetired.error!.issues)).toMatch(/maxRetries/); }); it('accepts a policy authored once and pasted onto any surface', () => { @@ -206,6 +213,5 @@ describe('every retry surface carries ONE contract (#4661, #4964, #4962)', () => expect(Automation.FlowSchema.safeParse({ ...minimalFlow, errorHandling: { strategy: 'retry', ...policy }, }).success).toBe(true); - expect(Automation.ETLPipelineSchema.safeParse({ ...minimalPipeline, retry: policy }).success).toBe(true); }); }); diff --git a/packages/spec/src/shared/retry-policy.zod.ts b/packages/spec/src/shared/retry-policy.zod.ts index 98497a7737..e81acb6202 100644 --- a/packages/spec/src/shared/retry-policy.zod.ts +++ b/packages/spec/src/shared/retry-policy.zod.ts @@ -27,7 +27,9 @@ * * - `automation/flow.zod.ts` → `Flow.errorHandling` (#4964) — spelled the base * delay `retryDelayMs`, every other key already identical. - * - `automation/etl.zod.ts` → `ETLPipeline.retry` (#4962) — spelled the count + * - `automation/etl.zod.ts` → `ETLPipeline.retry` (#4962; the whole L2 layer + * was retired at #6414, so this surface no longer exists — the convergence + * is recorded because it is how the divergence was FOUND) — spelled the count * `maxAttempts`, defaulted it to **3** (the opposite of the opt-in reading * below), and declared no `backoffMultiplier` / `maxRetryDelayMs` / `jitter` * at all, so its backoff was flat, uncapped and unjittered. diff --git a/packages/spec/src/system/http-server.test.ts b/packages/spec/src/system/http-server.test.ts index 1aa84970b9..7d031fcae7 100644 --- a/packages/spec/src/system/http-server.test.ts +++ b/packages/spec/src/system/http-server.test.ts @@ -4,10 +4,6 @@ import { RouteHandlerMetadataSchema, MiddlewareType, MiddlewareConfigSchema, - ServerEventType, - ServerEventSchema, - ServerCapabilitiesSchema, - ServerStatusSchema, } from './http-server.zod'; import * as sharedHttp from '../shared/http.zod'; import { RouterConfigSchema } from '../api/router.zod'; @@ -193,124 +189,74 @@ describe('MiddlewareConfigSchema', () => { }); }); -describe('ServerEventType', () => { - it('should accept valid event types', () => { - const types = ['starting', 'started', 'stopping', 'stopped', 'request', 'response', 'error']; - - types.forEach((type) => { - expect(() => ServerEventType.parse(type)).not.toThrow(); - }); - }); - - it('should reject invalid event types', () => { - expect(() => ServerEventType.parse('invalid')).toThrow(); - }); -}); - -describe('ServerEventSchema', () => { - it('should accept valid server event', () => { - const event = ServerEventSchema.parse({ - type: 'started', - timestamp: '2025-01-01T00:00:00Z', - }); - - expect(event.type).toBe('started'); - expect(event.timestamp).toBe('2025-01-01T00:00:00Z'); - }); - - it('should accept event with data', () => { - const event = ServerEventSchema.parse({ - type: 'error', - timestamp: '2025-01-01T00:00:00Z', - data: { message: 'Connection refused', code: 500 }, - }); - - expect(event.data).toEqual({ message: 'Connection refused', code: 500 }); - }); - - it('should reject invalid timestamp', () => { - expect(() => ServerEventSchema.parse({ type: 'started', timestamp: 'not-a-date' })).toThrow(); - }); - - it('should reject missing required fields', () => { - expect(() => ServerEventSchema.parse({})).toThrow(); - expect(() => ServerEventSchema.parse({ type: 'started' })).toThrow(); - }); -}); - -describe('ServerCapabilitiesSchema', () => { - it('should accept empty config with defaults', () => { - const caps = ServerCapabilitiesSchema.parse({}); - - expect(caps.httpVersions).toEqual(['1.1']); - expect(caps.websocket).toBe(false); - expect(caps.sse).toBe(false); - expect(caps.serverPush).toBe(false); - expect(caps.streaming).toBe(true); - expect(caps.middleware).toBe(true); - expect(caps.routeParams).toBe(true); - expect(caps.compression).toBe(true); - }); - - it('should accept full configuration', () => { - const caps = ServerCapabilitiesSchema.parse({ - httpVersions: ['1.1', '2.0'], - websocket: true, - sse: true, - serverPush: true, - streaming: false, - middleware: false, - routeParams: false, - compression: false, - }); - - expect(caps.httpVersions).toEqual(['1.1', '2.0']); - expect(caps.websocket).toBe(true); - expect(caps.sse).toBe(true); - }); - - it('should reject invalid HTTP versions', () => { - expect(() => ServerCapabilitiesSchema.parse({ httpVersions: ['4.0'] })).toThrow(); - }); -}); - -describe('ServerStatusSchema', () => { - it('should accept minimal status', () => { - const status = ServerStatusSchema.parse({ - state: 'running', - }); - - expect(status.state).toBe('running'); - }); - - it('should accept all state values', () => { - const states = ['stopped', 'starting', 'running', 'stopping', 'error']; - - states.forEach((state) => { - expect(() => ServerStatusSchema.parse({ state })).not.toThrow(); - }); +/** + * The rest of `http-server.zod.ts`'s runtime vocabulary was retired in v17 + * (#5295, ADR-0049 enforce-or-remove) — `ServerEventType` / `ServerEvent`, + * `ServerCapabilities` and `ServerStatus`. Same file, same route and the same + * absence of an authoring door as #4938 above, so again there is no + * `retiredKey()` tombstone to assert against: none of the three was a KEY on + * anything, so nobody could author one and nobody can receive a prescription. + * What is pinned here is that the exports are gone, that the removal stopped at + * the three shapes, and that the ONE doubt which held this card for four days — + * "a capability vocabulary may be a reference surface for host implementers" — + * was answered by measurement rather than by the absence of a grep hit. + * + * The unit tests that stood here (event-type enum, event parse, capability + * defaults, status states) are deliberately NOT re-pointed at a surviving + * schema: they asserted the shapes' own behaviour, and the shapes are the thing + * being removed. Replacing them wholesale with the pins below is the third + * fixture disposition in the retirement playbook. + */ +describe('server runtime vocabulary retirement (#5295)', () => { + // All four were runtime VALUES (`z.enum` and `lazySchema` both produce one), + // so an `in` check is a real witness. Reverse verification, direction + // predicted before running it: pasting any limb back turns exactly these + // assertions red — they are `false`-expecting existence checks, so the + // restored export is the failure. It is the plain red direction, not one of + // the two inverted ones, because nothing downstream COUNTS these names. + it.each([ + 'ServerEventType', + 'ServerEventSchema', + 'ServerCapabilitiesSchema', + 'ServerStatusSchema', + ])('no longer exports the value `%s`', (name) => { + expect(name in httpServer).toBe(false); }); - it('should accept full status', () => { - const status = ServerStatusSchema.parse({ - state: 'running', - uptime: 3600000, - server: { port: 3000, host: '0.0.0.0', url: 'http://localhost:3000' }, - connections: { active: 10, total: 500 }, - requests: { total: 1000, success: 990, errors: 10 }, - }); - - expect(status.uptime).toBe(3600000); - expect(status.server?.port).toBe(3000); - expect(status.connections?.active).toBe(10); - expect(status.requests?.total).toBe(1000); + it('does not re-export them from the system barrel either', async () => { + const system = await import('./index'); + for (const name of [ + 'ServerEventType', + 'ServerEventSchema', + 'ServerCapabilitiesSchema', + 'ServerStatusSchema', + ]) { + expect(name in system).toBe(false); + } }); - it('should reject invalid state', () => { - expect(() => ServerStatusSchema.parse({ state: 'invalid' })).toThrow(); + it('stops at the three shapes — the route/middleware half of the file survives', () => { + // The file is not retired; its ROUTE-REGISTRATION half has live consumers + // (`packages/rest/src/route-manager.ts`, `packages/runtime/src/middleware.ts`). + // This is the same "removal is the container, not the file" line #4938 drew, + // reasserted one layer in so a later sweep does not read the second + // retirement as licence to take the rest. + for (const name of [ + 'RouteHandlerMetadataSchema', + 'MiddlewareType', + 'MiddlewareConfigSchema', + 'MiddlewareConfig', + ]) { + expect(name in httpServer).toBe(true); + } }); - it('should reject missing required state', () => { - expect(() => ServerStatusSchema.parse({})).toThrow(); + it('does not touch the server config that IS live and authorable', async () => { + // `system/stack-server.zod.ts` is the one authoring door for server-level + // configuration (#5006) and grows a key at a time, each with its executor. + // A reader who sees two server retirements in this file must not conclude + // that server configuration itself was retired. + const stackServer = await import('./stack-server.zod'); + expect('StackServerConfigSchema' in stackServer).toBe(true); }); }); diff --git a/packages/spec/src/system/http-server.zod.ts b/packages/spec/src/system/http-server.zod.ts index 26a6119f32..9b77e5c8ed 100644 --- a/packages/spec/src/system/http-server.zod.ts +++ b/packages/spec/src/system/http-server.zod.ts @@ -192,148 +192,56 @@ export type MiddlewareConfig = z.input; export type MiddlewareConfigParsed = z.infer; // ========================================== -// Server Lifecycle Events +// Server Lifecycle Events / Capabilities / Status — RETIRED // ========================================== -/** - * Server Event Type Enum - */ -export const ServerEventType = z.enum([ - 'starting', // Server is starting - 'started', // Server has started and is listening - 'stopping', // Server is stopping - 'stopped', // Server has stopped - 'request', // Request received - 'response', // Response sent - 'error', // Error occurred -]); - -export type ServerEventType = z.input; - -/** - * Server Event Schema - * Events emitted by the HTTP server during lifecycle - */ -export const ServerEventSchema = lazySchema(() => z.object({ - /** - * Event type - */ - type: ServerEventType.describe('Event type'), - - /** - * Timestamp - */ - timestamp: z.string().datetime().describe('Event timestamp (ISO 8601)'), - - /** - * Event payload - */ - data: z.record(z.string(), z.unknown()).optional().describe('Event-specific data'), -})); - -export type ServerEvent = z.input; - -// ========================================== -// Server Capability Declaration -// ========================================== - -/** - * Server Capabilities Schema - * Declares what features a server implementation supports - */ -export const ServerCapabilitiesSchema = lazySchema(() => z.object({ - /** - * Supported HTTP versions - */ - httpVersions: z.array(z.enum(['1.0', '1.1', '2.0', '3.0'])).default(['1.1']).describe('Supported HTTP versions'), - - /** - * WebSocket support - */ - websocket: z.boolean().default(false).describe('WebSocket support'), - - /** - * Server-Sent Events support - */ - sse: z.boolean().default(false).describe('Server-Sent Events support'), - - /** - * HTTP/2 Server Push - */ - serverPush: z.boolean().default(false).describe('HTTP/2 Server Push support'), - - /** - * Streaming support - */ - streaming: z.boolean().default(true).describe('Response streaming support'), - - /** - * Middleware support - */ - middleware: z.boolean().default(true).describe('Middleware chain support'), - - /** - * Route parameterization - */ - routeParams: z.boolean().default(true).describe('URL parameter support (/users/:id)'), - - /** - * Built-in compression - */ - compression: z.boolean().default(true).describe('Built-in compression support'), -})); - -export type ServerCapabilities = z.input; -/** Post-parse shape of {@link ServerCapabilities} — defaults applied, transforms run (ADR-0122). */ -export type ServerCapabilitiesParsed = z.infer; - -// ========================================== -// Server Status & Metrics -// ========================================== - -/** - * Server Status Schema - * Current operational status of the server - */ -export const ServerStatusSchema = lazySchema(() => z.object({ - /** - * Server state - */ - state: z.enum(['stopped', 'starting', 'running', 'stopping', 'error']).describe('Current server state'), - - /** - * Uptime in milliseconds - */ - uptime: z.number().int().optional().describe('Server uptime in milliseconds'), - - /** - * Server information - */ - server: z.object({ - port: z.number().int().describe('Listening port'), - host: z.string().describe('Bound host'), - url: z.string().optional().describe('Full server URL'), - }).optional(), - - /** - * Connection metrics - */ - connections: z.object({ - active: z.number().int().describe('Active connections'), - total: z.number().int().describe('Total connections handled'), - }).optional(), - - /** - * Request metrics - */ - requests: z.object({ - total: z.number().int().describe('Total requests processed'), - success: z.number().int().describe('Successful requests'), - errors: z.number().int().describe('Failed requests'), - }).optional(), -})); - -export type ServerStatus = z.input; +// `ServerEventType`, `ServerEventSchema` / `ServerEvent`, +// `ServerCapabilitiesSchema` / `ServerCapabilities` / +// `ServerCapabilitiesParsed`, and `ServerStatusSchema` / `ServerStatus` were +// REMOVED per ADR-0049 enforce-or-remove (#5295, protocol 17) — the same route, +// in the same file, for the same reason as `HttpServerConfigSchema` above +// (#4938). Route 3 of the retirement playbook: nothing parsed them, so there is +// no author to hand a `retiredKey()` tombstone to and no stored or authored +// document for a D2 conversion to rewrite. `RETIRED_DEFS_BY_MAJOR[17]` plus the +// D3 `SemanticMigration` `http-server-runtime-vocabulary-retired` ARE the +// declaration. +// +// What each declared, and what actually decides it: +// +// | retired shape | declared | the live mechanism | +// |---|---|---| +// | `ServerEventType` / `ServerEvent` | a 7-value lifecycle/traffic event feed (`starting`, `started`, `stopping`, `stopped`, `request`, `response`, `error`) with a timestamp and a loose payload | nothing emitted it. Lifecycle is the transport plugin's own `start`/`stop` seam; per-request observability is `system/metrics.zod.ts` and `system/logging.zod.ts`, and `OS_SERVER_TIMING` for timings | +// | `ServerCapabilities` | eight booleans a server implementation would *report* about itself (`websocket`, `sse`, `serverPush`, `streaming`, `middleware`, `routeParams`, `compression`, plus `httpVersions`) | nothing reported or read them. A transport plugin declares what it provides by implementing the kernel plugin contract — the seams it registers ARE the capability statement, and a second self-described record can only disagree with them | +// | `ServerStatus` | a five-state machine plus uptime, bound host/port and connection/request counters | `/health` for liveness and the metrics surface for counters; no seam ever produced this shape | +// +// ## The measurement that made this a removal rather than a conformance surface +// +// #5295 was held, not queued, on one doubt: a response/capability vocabulary +// can legitimately be a REFERENCE surface for host implementers (the CSS-variable +// rebuttal), and "zero consumers in this repo" is weaker evidence for one of +// those than for an authorable key. The card's precondition was therefore to +// measure the reference reader itself. Re-run on `origin/main` immediately +// before this removal: +// +// 1. `plugin-hono-server` — the one in-tree host implementation — neither +// implements nor reports any of the three. Its source names no capability +// record, no status shape and no lifecycle event union; what it registers +// is routes and middleware through the kernel plugin contract. +// 2. Declaration-site grep (`^(export )?(const|type) `) puts every +// declaration in this file, and a quoted-name sweep across objectstack and +// objectui finds no reader outside this file: the only surviving mentions +// are three ADR-0122 isomorphism pins (deleted with the schemas) and the +// GENERATED reference pages, which document the export because it exists, +// not because anyone imports it. +// 3. The control ran green in the same sweep: `MiddlewareConfig`, declared +// twelve lines above these, resolves to a live consumer +// (`packages/runtime/src/middleware.ts:4,59`) — so the sweep can see a +// reader in this file when there is one. +// +// A reference surface with no referent is the #3950 shape: an exported schema +// with no consumer reads as a capability to whoever finds it. If host-implementer +// conformance becomes a real requirement, it returns through the ENFORCE route +// — an adapter contract with a checker behind it, vocabulary second. // ========================================== // Helper Functions diff --git a/packages/spec/src/type-alias-convention.pin.test.ts b/packages/spec/src/type-alias-convention.pin.test.ts index 2d46ec5d03..ab15273dfa 100644 --- a/packages/spec/src/type-alias-convention.pin.test.ts +++ b/packages/spec/src/type-alias-convention.pin.test.ts @@ -462,7 +462,6 @@ export type Iso145 = Assert, export type Iso146 = Assert, z.infer< typeof M28.UpdateDataResponseSchema > >>; export type Iso147 = Assert, z.infer< typeof M28.DeleteDataResponseSchema > >>; export type Iso148 = Assert, z.infer< typeof M28.CreateManyDataResponseSchema > >>; -export type Iso149 = Assert, z.infer< typeof M28.DeleteViewResponseSchema > >>; export type Iso150 = Assert, z.infer< typeof M28.CheckPermissionResponseSchema > >>; export type Iso151 = Assert, z.infer< typeof M28.GetEffectivePermissionsResponseSchema > >>; export type Iso152 = Assert, z.infer< typeof M28.RealtimeConnectResponseSchema > >>; @@ -1101,9 +1100,6 @@ export type Iso563 = Assert, z.inf // system/http-server.zod.ts export type Iso564 = Assert, z.infer< typeof M132.MiddlewareType > >>; -export type Iso565 = Assert, z.infer< typeof M132.ServerEventType > >>; -export type Iso566 = Assert, z.infer< typeof M132.ServerEventSchema > >>; -export type Iso567 = Assert, z.infer< typeof M132.ServerStatusSchema > >>; // system/incident-response.zod.ts export type Iso568 = Assert, z.infer< typeof M133.IncidentResponsePhaseSchema > >>; @@ -1345,9 +1341,6 @@ export type Iso721 = Assert, z export type Iso722 = Assert, z.infer< typeof M28.UpdateDataRequestSchema > >>; export type Iso723 = Assert, z.infer< typeof M28.DeleteDataRequestSchema > >>; export type Iso724 = Assert, z.infer< typeof M28.CreateManyDataRequestSchema > >>; -export type Iso725 = Assert, z.infer< typeof M28.ListViewsRequestSchema > >>; -export type Iso726 = Assert, z.infer< typeof M28.GetViewRequestSchema > >>; -export type Iso727 = Assert, z.infer< typeof M28.DeleteViewRequestSchema > >>; export type Iso728 = Assert, z.infer< typeof M28.CheckPermissionRequestSchema > >>; export type Iso729 = Assert, z.infer< typeof M28.GetObjectPermissionsRequestSchema > >>; export type Iso730 = Assert, z.infer< typeof M28.GetEffectivePermissionsRequestSchema > >>; @@ -1492,13 +1485,42 @@ describe('ADR-0122 type-alias convention', () => { // `ValidateDataResponse` — three new protocol shapes with no defaults or // transforms anywhere in their trees, i.e. the second (RISE) case above. // + // + // 751 -> 754 is #6037's `ValidateDataIssue` / `ValidateDataRequest` / + // `ValidateDataResponse` — three new protocol shapes with no defaults or + // transforms anywhere in their trees, i.e. the second (RISE) case above. + // // 754 -> 755 is #5933's `SpecifierValueDomain` — one new closed enum on // `SettingsManifest`'s SpecifierSchema, the same (RISE) case: a `z.enum` // has no default or transform, so its two shapes coincide and it gets a pin // rather than a `SpecifierValueDomainParsed` synonym. + // + // 755 -> 748 is the 2026-08-08 ADR-0049 retirement sweep (#6486), the + // first way again — a schema left the package, so its pin left with it. + // Written out per member, because a MULTI-member sweep is exactly where a + // count gets nudged to fit instead of recomputed: + // + // #5295 -3 ServerEventType, ServerEventSchema, ServerStatusSchema + // (`ServerCapabilities` has a `Parsed` alias, so it was never + // pinned here — a retired schema does not always cost a line) + // #6239 -4 DeleteViewResponseSchema, ListViewsRequestSchema, + // GetViewRequestSchema, DeleteViewRequestSchema (four of the + // ten view schemas were isomorphic; the other six were paired) + // #6414 0 every ETL alias already had a `Parsed` counterpart + // + // -7, and 755 - 7 = 748. Three things this one entry is worth stating, + // because they are the ways a MINUS gets miscomputed here. (1) The member + // count (3) and the pin count (7) have no relation to each other. (2) The + // -7 was computed against 751 at the branch point and had to be rebased + // TWICE before landing — onto #6037's 754, then onto #5933's 755 — so the + // subtrahend was the only stable operand. (3) A sibling retirement in the + // same window contributed ZERO: #6527 retired `array_agg` / `string_agg` + // from `AggregationFunction`, and an enum VALUE narrowing is invisible + // here, exactly as it is to the four surface ratchets. Recompute from the + // file; never from the changelog. const self = readFileSync(fileURLToPath(import.meta.url), 'utf8'); const pins = self.match(/^export type Iso\d+ = Assert {