From e76b8f7c0d96118e003bead9b77a33e474fcdf75 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Wed, 2 Sep 2026 12:27:54 -0300 Subject: [PATCH 01/48] docs: design review 22 remediation --- ...9-02-pr157-review-22-remediation-design.md | 365 ++++++++++++++++++ 1 file changed, 365 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md diff --git a/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md b/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md new file mode 100644 index 00000000..c84cfed8 --- /dev/null +++ b/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md @@ -0,0 +1,365 @@ +# PR #157 Review 22 Remediation Design + +**Date:** 2026-09-02 +**Status:** Approved for implementation +**Base branch:** `w-23692110-multi-engine-design` at `6661b96` +**Fix branch:** `w-23692110-review-22-fixes` +**PR target:** `w-23692110-multi-engine-design` +**Source:** `docs/pr-157-follow-up-code-review-22.md` + +## Goal + +Resolve all eight open findings from PR #157 review 22 without changing valid-input behavior or the public Node and Python APIs. The result must prevent host-process termination, make engine destruction safe for direct C ABI consumers, bind accepted operations to one engine generation, bound Node streaming memory, cover detach-poison recovery, and pass repository whitespace checks. + +## Scope + +In scope: + +- Finding 1: reject same-OS-thread DataWeave lifecycle or execution reentrancy from a native host callback in Node and Python. +- Finding 2: contain every Java `@CEntryPoint` throwable behind an explicit ABI sentinel. +- Finding 3: make Python operation admission atomic with lifecycle generation validation. +- Finding 4: give every Java engine a core-owned lifecycle record and operation leases. +- Finding 5: bind Node and Python lazy streams to the generation present at stream creation. +- Finding 6: bound Node output buffering across both the native TSFN queue and the JavaScript chunk queue. +- Finding 7: add deterministic failure-injection coverage for Node's detach-poison transitions. +- Finding 8: remove the two whitespace errors in the PR diff. +- Update native-lib, Node, Python, and consolidated design documentation for changed contracts. + +Out of scope: + +- Separate GraalVM isolates per engine. +- Enabling custom module resolvers in background streaming or transform workers. +- Changing public `DataWeave` method names, parameters, or result envelopes. +- Retaining compatibility with removed pre-GA singleton C entrypoints. +- Making application-retained output buffers part of the binding's memory bound. +- Guaranteeing bounded total DataWeave runtime memory for non-deferred scripts that materialize output before callback delivery. + +## Global Constraints + +- Use the checked-in `./gradlew` wrapper and GraalVM Community Java 24 for native verification. +- Java remains source/target 17; Scala remains 2.12; Python remains 3.9+; Node remains 18+. +- Preserve exported C names, argument order, callback semantics, and existing JSON wire fields. +- Normal script failures continue to return non-null `{"success":false,...}` envelopes. +- Never let Java, JavaScript, or Python exceptions unwind across C callbacks. +- Every OS thread calling Graal attaches its own isolate thread and detaches afterward unless a successful teardown has invalidated the attachment. +- Node shared C lifecycle state remains guarded by `g_mutex`; per-stream flow state uses its own mutex with explicit lock ordering. +- Python module isolate state remains guarded by `_isolate_lock`; user callbacks run without module locks held. +- All behavior changes use test-first red-green cycles. + +## Architecture + +```mermaid +flowchart TD + A[Public Node or Python call] --> B[Capture immutable engine token] + B --> C{Native callback active on this OS thread?} + C -->|yes| D[Throw DataWeaveError] + C -->|no| E[Binding admission validation] + E --> F[Java core lease acquisition] + F -->|closing or absent| G[Unknown engine envelope] + F -->|live| H[Execute with fixed runtime and handle] + H --> I[Release core lease] + I --> J[Destroy may finish draining] +``` + +The binding token prevents work accepted by one object generation from migrating to a replacement engine. The Java lease independently protects the raw ABI and resolver context even when callers bypass the bindings. These mechanisms are deliberately additive: neither replaces Node bridge pinning, Python instance serialization, or isolate reference accounting. + +## 1. Java C Entrypoint Exception Containment + +### Problem + +The exported methods in `NativeLib` use GraalVM's default `CEntryPoint.FatalExceptionHandler`. A recoverable Java validation error, such as a null resolver callback, therefore prints a fatal error and exits the embedding process instead of returning to C. + +### Design + +Add one package-private `CEntryPointExceptionHandlers` support class containing three nested handler classes. Each nested class declares exactly one static `@Uninterruptible` handler method, performs no allocation or logging, and returns one ABI-category sentinel: + +| Category | Sentinel | Entrypoints | +|---|---|---| +| Engine handle | `0L` | `create_engine`, `create_engine_with_resolver` | +| Result pointer | null `CCharPointer` | all three `run_*_engine` methods | +| Void | return | `destroy_engine`, `free_cstring` | + +Expected validation failures should still be handled in the entrypoint body and represented as ordinary error envelopes when allocation remains safe. The custom exception handler is the non-allocating last resort for unexpected throwables or a failure while constructing an envelope. + +The ABI contract becomes explicit: + +- Engine handles are positive; `0` means creation failed and no engine was registered. +- A null run-result pointer means the entrypoint could not create a JSON result. The caller must not free it. +- Unknown, closing, or destroyed handles return the existing non-null `Unknown engine handle` envelope. +- `free_cstring(NULL)` remains a no-op. + +### Tests + +- A native subprocess calls `create_engine_with_resolver(thread, NULL, NULL)`, observes `0`, then creates and uses a valid engine in the same process. +- Native malformed-pointer coverage observes a null result instead of process exit where a safe deterministic trigger exists. +- A hosted reflection test verifies every export names a non-default handler of the correct category. +- Native compilation validates GraalVM's handler shape and `@Uninterruptible` requirements. + +## 2. Core Engine Lifecycle Records and Leases + +### Problem + +`ScriptRuntime.get(handle)` and `ScriptRuntime.destroy(handle)` are unrelated map operations. A raw C run can retain the Java runtime, then a concurrent destroy can remove the registry entry and return. The caller may free the resolver `ctx` while the admitted run later invokes it. + +### Design + +Change the Java registry value from a bare `ScriptRuntime` to an `EngineRecord` with this lifecycle: + +```mermaid +stateDiagram-v2 + [*] --> LIVE: register + LIVE --> LIVE: acquire or release lease + LIVE --> CLOSING: destroy closes admission + CLOSING --> CLOSING: wait for active leases + CLOSING --> DESTROYED: final lease released + DESTROYED --> [*]: remove exact record +``` + +`ScriptRuntime.acquire(long handle)` returns an `EngineLease` only when the record is `LIVE`. The state check and active-lease increment occur under the record monitor. `EngineLease` implements `AutoCloseable`, exposes the fixed `ScriptRuntime`, and releases exactly once. + +Every `run_*_engine` entrypoint acquires a lease before converting required pointers or invoking the runtime and closes it with try-with-resources. The lease spans resolver calls, streaming callbacks, transform feeder cleanup, and final result allocation. It can end before `free_cstring()` because the returned allocation no longer depends on the engine. + +`destroy_engine` atomically changes `LIVE` to `CLOSING`, rejects later acquisitions, and waits until all admitted leases close. It continues waiting through interruption and restores the interrupt status only after lifetime safety has been re-established. Concurrent destroy calls coordinate on the same record. Exact-record map removal occurs only after `DESTROYED`. + +The public raw ABI contract is: + +- `destroy_engine` is idempotent for unknown or already destroyed handles. +- `destroy_engine` can block until all previously admitted operations finish. +- Resolver and callback contexts must remain valid through `destroy_engine` return and may be freed afterward. +- Calling `destroy_engine` for an engine synchronously from one of that engine's callbacks is prohibited because the callback owns a lease that destroy must drain. + +Node's `engine_bridge_t.in_flight` remains necessary for N-API reference and resolver-bridge lifetime. Python operation serialization remains necessary for wrapper lifecycle and generation correctness. + +### Tests + +- Hosted Java tests prove admission rejection after close, multiple-lease drain, concurrent destroy, interrupted wait restoration, and lease release on thrown bodies. +- A raw ctypes subprocess blocks inside a real resolver callback, calls destroy on another OS thread, and proves destroy does not return until the callback and run finish. + +## 3. Native Callback Reentrancy Guards + +### Problem + +A synchronous resolver invokes host JavaScript or Python while the outer Graal call is active. If host code invokes DataWeave on another engine on the same OS thread, the nested call attaches and detaches that thread. The outer call then resumes with invalid Graal thread state and terminates the process. + +The same hazard applies to lifecycle operations and other synchronous host callbacks that can enter another engine. + +### Node Design + +Add addon-level OS-thread-local callback depth using `uv_key_t`, initialized through the existing `uv_once` setup. Enter the scope immediately around each direct host callback invocation and restore it on every status or exception path. + +Before any entrypoint attaches, detaches, creates, destroys, or admits DataWeave work, reject when callback depth is nonzero. The addon throws an error with a stable internal code such as `ERR_DATAWEAVE_CALLBACK_REENTRANCY`. The TypeScript layer maps that code to the public `DataWeaveError` class. + +The C guard is authoritative because raw addon users and duplicate package copies must not bypass it. A process-global boolean is forbidden because independent Worker OS threads must remain concurrent. + +### Python Design + +Add module-level `threading.local()` callback depth shared by all `NativeRuntime` instances. Apply the scope directly around user resolver, read, and write callback invocation. Check the scope before public lifecycle mutation, operation-token capture, and native serialized admission. + +The guard fails before waiting on another engine's lock. A global execution mutex or `RLock` is not used because it would either deadlock or permit the unsafe reentry. + +### Tests + +- Node and Python isolated child processes attempt cross-engine nested `run()` from a resolver, catch `DataWeaveError`, return valid module source, and prove the outer run still succeeds with no fatal stderr. +- Tests cover uncaught resolver reentry, nested initialization, and raw addon/native admission where feasible. +- Unit tests prove callback depth is OS-thread-local and restored after callback errors. + +## 4. Immutable Engine Generations + +### Problem + +Both bindings hold a mutable current handle. Python validates initialization before entering its operation lock. Node and Python lazy streams defer body execution until first iteration. Cleanup and reinitialization can therefore replace the engine between acceptance and admission, silently moving work from generation A to B. + +### Shared Contract + +Each successful engine initialization increments a monotonic binding-instance generation. Each operation captures an immutable token: + +```text +EngineOperationToken { handle, generation } +``` + +Generation never resets during cleanup or failed initialization. Handle alone is insufficient because a newly created isolate can restart Java static handle allocation. + +At native admission, the binding compares the token with the current initialized token while holding the lock that excludes lifecycle mutation. Native calls use the token's handle, never a later mutable field. + +The race has two valid outcomes: + +- Admission wins: work completes on its captured engine; cleanup waits or refuses while it is active. +- Cleanup wins: later token validation raises `DataWeaveError` for a stale engine generation; the replacement engine is never called. + +### Python Design + +Add frozen internal `_EngineOperation(handle: int, generation: int)` state to `NativeRuntime`. Public buffered, callback, transform, and stream methods capture this token synchronously. `_serialized_native_operation(expected)` validates it under the per-instance operation lock and yields the immutable token. + +Stream worker registration validates the token atomically with `_stream_workers_lock` registration. If registration wins, cleanup sees the worker and follows current active-worker policy. If cleanup wins, registration rejects as stale. Lock order is `_stream_workers_lock` before the brief operation-token validation; native execution never reacquires `_stream_workers_lock` while holding the operation lock. + +### Node Design + +Add `engineGeneration` and `EngineOperationToken`. Convert `runStreaming` and `runTransform` from public async-generator methods into ordinary methods that capture the token at call time and return private async generators. The private generator validates the token immediately before synchronous native admission and invokes FFI with `token.handle`. + +`runTransform` validates the same token before and after async input pre-buffering. The second check must occur immediately before FFI admission. + +This intentionally changes stale or uninitialized stream failure timing to method call or first pull as documented by the concrete path; normal stream chunks and terminal metadata do not change. + +### Tests + +- Deterministic Python paused-admission test captures generation A, performs cleanup/reinitialize to B, resumes, and proves no call reaches handle B. +- Node and Python tests create a lazy stream, cleanup/reinitialize, then consume it and receive `DataWeaveError` before native invocation. +- Handle-reuse tests prove generation, not only numeric handle, controls identity. +- Transform tests pause during async pre-buffering and reject after generation replacement. +- Complementary tests prove already-admitted work finishes on its original engine. + +## 5. End-to-End Node Output Backpressure + +### Problem + +The output TSFNs use `max_queue_size = 0`, and `streamFromNative` appends delivered buffers to an unbounded array. The native producer can outrun a paused consumer and retain output-sized memory even if one of those queues is later bounded independently. + +### Design + +Retain the push architecture and add one reference-counted `output_flow_t` per stream or transform operation. It tracks outstanding bytes and chunks from immediately before TSFN enqueue until the async generator dequeues the corresponding buffer. + +```mermaid +sequenceDiagram + participant P as Native producer + participant F as Flow credits + participant Q as Bounded TSFN + participant J as JS chunk queue + participant C as Async consumer + P->>F: reserve bytes and one chunk + F-->>P: wait only on producer thread if full + P->>Q: enqueue payload + Q->>J: deliver Buffer on JS thread + C->>J: dequeue Buffer + C->>F: acknowledge bytes and one chunk + F-->>P: resume below low watermark +``` + +Internal defaults are fixed initially rather than public configuration: + +- High watermark: 1 MiB or 128 chunks. +- Low watermark: 512 KiB and 64 chunks. +- A single oversized chunk is admitted when the window is empty so it cannot deadlock permanently. + +The output TSFN receives a finite queue capacity with room for normal outstanding chunks and the terminal sentinel. The native producer may wait on a per-flow condition variable; the JS thread only performs short acknowledge/cancel updates and never waits. + +The internal FFI start contract returns an operation controller with: + +```ts +interface NativeStreamingOperation { + readonly completion: Promise; + acknowledge(bytes: number): void; + cancel(): void; + close(): void; +} +``` + +`streamFromNative` acknowledges a chunk when dequeuing it for delivery. This bounds binding-owned memory but not buffers retained by application code after `yield`. + +Cancellation is mandatory because a producer blocked on credit cannot finish if the consumer abandons the stream. Generator `finally`, early `return()`, DataWeave cleanup, env teardown, and JS callback failure all cancel the flow, signal the producer, release queued credit exactly once, and allow native completion to settle. Cleanup semantics become cancel abandoned streams and drain their native completion, never wait indefinitely for consumer pulls. + +Native flow lock ordering is: + +- Never wait on the flow condition while holding `g_mutex`. +- If a path needs both locks, acquire `g_mutex` before the flow mutex and release the flow mutex before later global completion accounting. +- Refcounted flow ownership prevents callbacks, worker completion, cancellation, or TSFN finalization from freeing shared state twice. + +### Tests + +- TypeScript unit tests verify acknowledgment occurs only at dequeue, not JS enqueue; early return cancels; rejection drains accounted chunks; close/cancel are idempotent. +- Native test hooks expose current and peak outstanding bytes/chunks, pause, and cancellation state. +- Real integration tests pause a large deferred stream and transform, assert peak credits stay within the configured limit plus the one-oversized-chunk allowance, then resume and verify ordered complete output. +- Early generator return and cleanup of a paused stream complete under a bounded timeout and permit healthy reinitialization. +- The Node README writable-stream example waits for `drain` when `write()` returns false. + +## 6. Detach-Poison Failure Injection + +### Problem + +The addon now poisons an isolate when ordinary detach returns nonzero, but no test forces the status. Cleanup skipping teardown, isolate abandonment, and fresh-isolate recovery are unverified. + +### Design + +Centralize ordinary detach calls behind `detach_thread_checked(detach_site_t, thread)`. Under `DATAWEAVE_TEST_HOOKS`, a site-specific one-shot injection calls the real detach first and, when it succeeds, substitutes a nonzero observed status. Production builds remain a thin wrapper over the real function. + +Test-only counters record forced failures, isolate creation, teardown attempts, and isolate abandonment. Hooks expose state without making test behavior part of the product ABI. + +The safe hook verifies the addon's response to a detach status; it does not claim to reproduce every physical consequence of a thread that truly remained attached. + +Once poisoned, new DataWeave admission should fail closed rather than assume the isolate remains safe for new work. Already-produced triggering results may surface, active operations drain, final cleanup abandons the isolate without teardown, and later initialization creates a fresh isolate. + +### Tests + +- Child-process synchronous-run test forces one detach failure, observes poison, completes cleanup without teardown or hang, reinitializes, and runs successfully on a fresh isolate. +- Child-process transform test forces a background detach failure while final cleanup is waiting, proving deferred cleanup skips teardown and resolves. +- Lower-cost cases exercise create, resolver-create, bridge-finalize, and unknown-destroy detach sites where deterministic setup exists. + +## 7. Whitespace and Documentation + +Remove trailing spaces in `docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md` and the extra final blank line in `native-lib/python/src/dataweave/native.py` without unrelated formatting churn. + +Update: + +- `native-lib/README.md` with C sentinel, lease/drain, callback-context, and destroy-blocking contracts. +- `native-lib/node/README.md` with callback reentrancy, bounded streaming, cancellation, and writable `drain` guidance. +- `native-lib/python/README.md` with callback reentrancy and stale-generation behavior. +- `docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md` so the final-state architecture uses core leases, immutable generation tokens, bounded flow control, and the new test-hook posture. + +## Implementation Sequence + +```mermaid +flowchart TD + A[Java lifecycle tests] --> B[Core leases] + C[Native exception tests] --> D[C entrypoint handlers] + B --> E[Raw ABI concurrency verification] + D --> E + F[Python red tests] --> G[Callback and generation guards] + H[Node callback and generation red tests] --> I[Node guards and tokens] + J[Stream credit unit tests] --> K[Native credit protocol] + K --> L[Slow-consumer integration tests] + M[Detach injection red tests] --> N[Central detach wrapper and recovery] + E --> O[Docs and full native verification] + G --> O + I --> O + L --> O + N --> O +``` + +Use focused commits for independently reviewable behavior. The final PR contains the complete cohesive hardening set and targets `w-23692110-multi-engine-design`; do not squash unless requested during review. + +## Verification Matrix + +| Layer | Required verification | +|---|---| +| Java hosted | Focused lifecycle and existing `native-lib:test` suites | +| Native image | `native-lib:nativeCompile` with GraalVM Community Java 24 | +| Raw ABI | Isolated ctypes exception and resolver-context lease subprocess tests | +| Node unit | Generation, stream-credit, cancellation, and error mapping tests | +| Node integration | Resolver reentry, stale streams, slow consumer, cleanup, poison recovery | +| Node typecheck | `npm run build:ts` | +| Python unit | Token admission, callback TLS, worker registration, lifecycle tests | +| Python integration | Resolver reentry and stale real-native streams | +| Repository | `git diff --check w-23692110-multi-engine-design...HEAD` | + +Run the smallest focused test after each red-green cycle, then the nearest module suite. Before PR creation, run Java, Node, Python, native-image, and diff verification from a clean branch and review every commit in the PR range. + +## Risks and Mitigations + +- **Destroy self-deadlock:** same-engine callback destroy waits for its own lease. Reject binding callback reentrancy and document the raw ABI prohibition. +- **Lease leak:** one missed close blocks destroy forever. Use try-with-resources in every entrypoint and tests that throw inside the leased body. +- **Flow-control deadlock:** waiting from the JS thread or while holding `g_mutex` prevents progress. Only the native producer waits, with explicit lock ordering and cancellation broadcasts. +- **Double free/use-after-free:** worker, TSFN callback, generator cancellation, and finalizer share flow state. Use reference counting and idempotent cancel/close transitions. +- **Generation false acceptance:** numeric handles can repeat after isolate replacement. Compare both monotonic generation and handle. +- **Fault-hook overclaim:** real detach then substitute failure validates status handling but not a physically attached dead thread. State this limitation in tests and docs. +- **Throughput regression:** high/low watermarks intentionally slow fast producers behind slow consumers. Integration tests verify correctness; benchmark only if normal-consumer throughput changes materially. +- **Native handler fragility:** GraalVM custom exception handlers have strict shape requirements. Keep handlers allocation-free and prove them through native compilation and subprocess execution. + +## Success Criteria + +- The three previously reproduced exit-99 cases remain alive and return documented errors or sentinels. +- Direct raw-ABI destroy cannot return while an admitted resolver callback may still use its context. +- No buffered, callback, streaming, or transform operation can migrate to a replacement engine generation. +- Node binding-owned output buffering remains within configured byte/chunk watermarks for paused consumers. +- Abandoned or cleanup-cancelled Node streams settle without deadlock. +- Forced ordinary detach failures poison and abandon the old isolate, skip unsafe teardown, and allow fresh initialization. +- Existing valid-input Node, Python, Java, native, and TCK behavior remains green. +- `git diff --check` passes for the full PR range. From 1b3a19aab82dd0d658eff33a36ad50f44cc11d5f Mon Sep 17 00:00:00 2001 From: mlischetti Date: Wed, 2 Sep 2026 12:57:19 -0300 Subject: [PATCH 02/48] docs: plan review 22 remediation --- .../2026-09-02-pr157-review-22-remediation.md | 1208 +++++++++++++++++ 1 file changed, 1208 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md diff --git a/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md b/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md new file mode 100644 index 00000000..bdfee097 --- /dev/null +++ b/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md @@ -0,0 +1,1208 @@ +# PR #157 Review 22 Remediation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Resolve all eight PR #157 review-22 findings and deliver a verified PR from `w-23692110-review-22-fixes` to `w-23692110-multi-engine-design`. + +**Architecture:** The Java engine registry owns `LIVE -> CLOSING -> DESTROYED` records and leases so the raw C ABI is lifetime-safe. Node and Python add same-OS-thread native-callback guards and immutable `{handle, generation}` operation tokens; Node additionally uses one credit/ack flow-control object per asynchronous stream to bound native and JavaScript buffering and to cancel abandoned consumers safely. + +**Tech Stack:** Java 17, GraalVM Community Java 24 Native Image, C11/N-API 8/libuv, TypeScript 5.5/Node.js 18+, Python 3.9+/ctypes, Gradle, JUnit 5, Vitest 3, pytest. + +**Spec:** `docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md` + +## Global Constraints + +- Use the checked-in `./gradlew` wrapper and GraalVM Community Java 24 for native verification. +- Java remains source/target 17; Scala remains 2.12; Python remains 3.9+; Node remains 18+. +- Preserve exported C names, argument order, callback semantics, and existing JSON wire fields. +- Normal script failures continue to return non-null `{"success":false,...}` envelopes. +- Never let Java, JavaScript, or Python exceptions unwind across C callbacks. +- Every OS thread calling Graal attaches its own isolate thread and detaches afterward unless a successful teardown has invalidated the attachment. +- Node shared C lifecycle state remains guarded by `g_mutex`; per-stream flow state uses its own mutex with explicit lock ordering. +- Python module isolate state remains guarded by `_isolate_lock`; user callbacks run without module locks held. +- Use test-first red-green cycles for every behavior change. Run the named failing test before editing production code and record the expected failure. +- Work only in `/private/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review-22-fixes` on `w-23692110-review-22-fixes`. +- Do not modify unrelated untracked or generated artifacts. Do not commit `node_modules`, staged `native/`, `dist`, native build outputs, coverage, wheels, or downloaded TCK suites. + +## File Map + +- `native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java`: owns the Java engine registry, lifecycle records, and operation leases. +- `native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java`: contains allocation-free GraalVM exception handlers by ABI return category. +- `native-lib/src/main/java/org/mule/weave/lib/NativeLib.java`: validates exported entrypoint inputs, acquires core leases, and applies exception sentinels. +- `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java`: exercises hosted Java lease and destroy concurrency. +- `native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java`: verifies every exported entrypoint declares the intended exception handler. +- `native-lib/python/src/dataweave/native.py`: owns Python immutable operation tokens, serialized admission, and callback-thread-local state. +- `native-lib/python/src/dataweave/runtime.py`: captures operation generations at public API entry and binds stream workers to them. +- `native-lib/python/tests/unit/test_native.py`: tests immutable native admission and callback TLS. +- `native-lib/python/tests/unit/test_streaming.py`: tests stale stream rejection and callback wrappers. +- `native-lib/python/tests/integration/test_module_resolver.py`: isolates resolver reentrancy so a regression cannot kill pytest. +- `native-lib/python/tests/integration/test_lifecycle.py`: holds direct ctypes ABI sentinel and lease/drain subprocess tests. +- `native-lib/node/src/addon.c`: owns authoritative callback TLS, asynchronous stream flow control, cancellation, and detach fault injection. +- `native-lib/node/src/ffi.ts`: types and wraps the internal native streaming operation controller. +- `native-lib/node/src/stream.ts`: acknowledges chunks at dequeue and cancels abandoned generators. +- `native-lib/node/src/dataweave.ts`: owns Node generations, token validation, public error mapping, and active-stream cleanup. +- `native-lib/node/tests/unit/dataweave-initialize.test.ts`: tests Node generation capture, handle reuse, and active-stream cleanup. +- `native-lib/node/tests/unit/stream.test.ts`: tests credit acknowledgment and cancellation semantics without native code. +- `native-lib/node/tests/integration/resolver-reentrancy.test.ts`: child-process proof that nested resolver execution no longer exits 99. +- `native-lib/node/tests/integration/stream-backpressure.test.ts`: real-addon bounded-buffer and cancellation tests. +- `native-lib/node/tests/integration/detach-poison-hook.test.ts`: child-process detach-poison and fresh-isolate recovery tests. +- `native-lib/README.md`, `native-lib/node/README.md`, `native-lib/python/README.md`, and the consolidated design: public and maintainer-facing contract updates. + +--- + +### Task 1: Core Java Engine Leases + +**Files:** +- Create: `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java` +- Modify: `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java:629-691` +- Modify: `native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java:40-59` + +**Interfaces:** +- Consumes: existing `ScriptRuntime.register(ScriptRuntime)` and `ScriptRuntime.destroy(long)` semantics. +- Produces: `ScriptRuntime.EngineLease acquire(long handle)`, `EngineLease.runtime()`, `EngineLease.close()`, and blocking/idempotent `destroy(long handle)`. + +- [ ] **Step 1: Write lifecycle tests that expose the missing lease** + +Create `ScriptRuntimeLifecycleTest` with deterministic latches. The central test shape is: + +```java +@Test +void destroyWaitsForAnAdmittedLeaseAndRejectsNewAdmission() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + assertNotNull(lease); + + CountDownLatch destroyStarted = new CountDownLatch(1); + AtomicBoolean destroyReturned = new AtomicBoolean(false); + Thread destroyer = new Thread(() -> { + destroyStarted.countDown(); + ScriptRuntime.destroy(handle); + destroyReturned.set(true); + }); + destroyer.start(); + + assertTrue(destroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> ScriptRuntime.acquire(handle) == null); + assertFalse(destroyReturned.get()); + + lease.close(); + destroyer.join(1_000); + assertFalse(destroyer.isAlive()); + assertTrue(destroyReturned.get()); + assertNull(ScriptRuntime.acquire(handle)); +} +``` + +Add focused cases for: + +```java +multipleLeasesMustAllDrainBeforeDestroyReturns(); +concurrentDestroyCallsCoordinateAndComplete(); +closingAnEngineLeaseTwiceIsHarmless(); +interruptedDestroyRestoresInterruptAfterTheLeaseDrains(); +unknownHandleCannotAcquireALease(); +``` + +Use a local polling helper with a one-second deadline instead of sleeps. Every test must close admitted leases in `finally`. + +- [ ] **Step 2: Run the lifecycle test and verify RED** + +Run: + +```bash +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true +``` + +Expected: compilation fails because `ScriptRuntime.EngineLease` and `ScriptRuntime.acquire(long)` do not exist. + +- [ ] **Step 3: Implement the lifecycle record and lease** + +Replace `ConcurrentHashMap` with `ConcurrentHashMap`. Keep all lifecycle types in `ScriptRuntime.java`: + +```java +public static final class EngineLease implements AutoCloseable { + private final EngineRecord record; + private final ScriptRuntime runtime; + private boolean closed; + + private EngineLease(EngineRecord record, ScriptRuntime runtime) { + this.record = record; + this.runtime = runtime; + } + + public ScriptRuntime runtime() { + return runtime; + } + + @Override + public void close() { + synchronized (this) { + if (closed) { + return; + } + closed = true; + } + record.release(); + } +} +``` + +The record owns `State { LIVE, CLOSING, DESTROYED }`, `activeLeases`, `tryAcquire()`, `closeAndAwait()`, and `release()`. The state check and increment are one synchronized action. `closeAndAwait()` keeps waiting after `InterruptedException`, sets `DESTROYED` only after `activeLeases == 0`, and restores the interrupt bit after leaving the monitor. + +`destroy(handle)` obtains the record, invokes `closeAndAwait()`, and removes exactly that record with `REGISTRY.remove(handle, record)`. Do not remove the record before draining because concurrent destroy callers need the same coordination object. + +Make `register` reject null runtimes and ensure generated handles remain positive. If `NEXT_HANDLE.getAndIncrement()` returns a non-positive value after overflow, fail registration rather than publishing an invalid ABI handle. + +- [ ] **Step 4: Update existing registry tests to use leases** + +Replace every `ScriptRuntime.get(handle)` in `ScriptRuntimeTest` with scoped acquisition: + +```java +try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(hA)) { + assertNotNull(lease); + assertEquals("\"A:X\"", Result.parse(lease.runtime().run(IMPORT_A)).result); +} +``` + +Use `assertNull(ScriptRuntime.acquire(handle))` for absent or destroyed handles. Delete the Javadoc claim that checking `UNKNOWN_ENGINE_HANDLE_JSON` and `get()` verifies the complete entrypoint contract; retain only the exact-string assertion. + +- [ ] **Step 5: Run focused and module Java tests and verify GREEN** + +Run: + +```bash +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeTest" -PskipNodeTests=true -PskipPythonTests=true +``` + +Expected: both commands exit `0`; the new lifecycle class reports all tests passed. + +- [ ] **Step 6: Commit the core lease** + +```bash +git add native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java \ + native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java \ + native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java +git commit -m "fix(native-lib): lease engines across admitted operations" +``` + +### Task 2: C Entrypoint Exception Sentinels and Lease Adoption + +**Files:** +- Create: `native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java` +- Create: `native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java` +- Modify: `native-lib/src/main/java/org/mule/weave/lib/NativeLib.java:39-45,540-665` +- Modify: `native-lib/node/src/addon.c:2269-2273,2409-2412` + +**Interfaces:** +- Consumes: `ScriptRuntime.acquire(long)` and `EngineLease` from Task 1. +- Produces: explicit `ReturnZero`, `ReturnNullPointer`, and `ReturnVoid` C-entrypoint handlers; every run entrypoint is leased. + +- [ ] **Step 1: Write reflection tests for all exported handlers** + +Create `NativeLibEntryPointContractTest`. Use `NativeLib.class.getDeclaredMethod(...)` and `getAnnotation(CEntryPoint.class)` to assert exact handlers: + +```java +assertEquals( + CEntryPointExceptionHandlers.ReturnZero.class, + annotation("createEngine", IsolateThread.class).exceptionHandler()); +assertEquals( + CEntryPointExceptionHandlers.ReturnNullPointer.class, + annotation("runScriptEngine", IsolateThread.class, long.class, + CCharPointer.class, CCharPointer.class).exceptionHandler()); +assertEquals( + CEntryPointExceptionHandlers.ReturnVoid.class, + annotation("destroyEngine", IsolateThread.class, long.class).exceptionHandler()); +``` + +Cover all seven exports, including `freeCString` and both callback run entrypoints. + +- [ ] **Step 2: Run the handler contract test and verify RED** + +Run: + +```bash +./gradlew native-lib:test --tests "org.mule.weave.lib.NativeLibEntryPointContractTest" -PskipNodeTests=true -PskipPythonTests=true +``` + +Expected: compilation fails because `CEntryPointExceptionHandlers` does not exist. + +- [ ] **Step 3: Add allocation-free GraalVM handlers** + +Create the support class using `com.oracle.svm.core.Uninterruptible`: + +```java +final class CEntryPointExceptionHandlers { + private CEntryPointExceptionHandlers() { + } + + static final class ReturnZero implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") + static long handle(Throwable ignored) { + return 0L; + } + } + + static final class ReturnNullPointer implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") + static CCharPointer handle(Throwable ignored) { + return WordFactory.nullPointer(); + } + } + + static final class ReturnVoid implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Contain an exception at the C ABI boundary") + static void handle(Throwable ignored) { + } + } +} +``` + +Each nested handler class must declare exactly one method. Do not allocate, log, or throw in these methods. + +- [ ] **Step 4: Apply handlers and leases to all entrypoints** + +Annotate every export with the intended `exceptionHandler`. Change each run entrypoint from `ScriptRuntime.get(handle)` to: + +```java +try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle)) { + if (lease == null) { + return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + } + // Convert required pointers and execute through lease.runtime(). +} +``` + +Validate required pointer and callback arguments before dereference. Return a non-null `success:false` envelope for expected invalid arguments when possible. Keep null-pointer result only as the exception-handler fallback. + +The lease must include the callback loop, feeder join, and final result allocation. Keep `destroyEngine` idempotent and blocking through `ScriptRuntime.destroy(handle)`. + +Update the two addon comments so they state that `0` is the explicit Java ABI exception sentinel, not GraalVM default-value behavior. + +- [ ] **Step 5: Run Java tests and verify GREEN** + +Run: + +```bash +./gradlew native-lib:test --tests "org.mule.weave.lib.NativeLibEntryPointContractTest" -PskipNodeTests=true -PskipPythonTests=true +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true +``` + +Expected: both commands exit `0`. + +- [ ] **Step 6: Build the native library and inspect generated exports** + +Run with the checked-in GraalVM: + +```bash +export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" +export JAVA_HOME="$GRAALVM_HOME" +./gradlew native-lib:nativeCompile -PskipStripDebug=true +``` + +Expected: exit `0`; Native Image accepts all custom handlers. Verify `native-lib/build/native/nativeCompile/dwlib.h` still exports the same seven C names with unchanged signatures. + +- [ ] **Step 7: Commit the ABI containment** + +```bash +git add native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java \ + native-lib/src/main/java/org/mule/weave/lib/NativeLib.java \ + native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java \ + native-lib/node/src/addon.c +git commit -m "fix(native-lib): contain C entrypoint exceptions" +``` + +### Task 3: Raw ABI Native Regression Tests + +**Files:** +- Modify: `native-lib/python/tests/integration/test_lifecycle.py` + +**Interfaces:** +- Consumes: native sentinels from Task 2 and blocking `destroy_engine` lease semantics from Task 1. +- Produces: isolated subprocess tests for process survival and resolver-context drain. + +- [ ] **Step 1: Add null-resolver subprocess coverage** + +Add a helper that runs Python code with `DATAWEAVE_NATIVE_LIB` and the package source on `PYTHONPATH`. In the child, bind the raw ABI with `ctypes`, create an isolate, detach bootstrap, attach the current thread, and call: + +```python +null_resolver = ctypes.cast(None, RESOLVE_MODULE_CALLBACK) +handle = lib.create_engine_with_resolver(thread, null_resolver, None) +assert handle == 0 +healthy = lib.create_engine(thread) +assert healthy > 0 +null_result = lib.run_script_engine(thread, healthy, None, None) +assert not null_result +lib.destroy_engine(thread, healthy) +``` + +The child must not call `free_cstring` for the null result pointer. It then cleans up/detaches correctly and prints one JSON object. Parent assertions require exit `0`, `handle == 0`, a valid later handle, a null run result, and no `Fatal error` in stderr. + +- [ ] **Step 2: Add a resolver-context lease/drain subprocess test** + +Use two OS threads and direct ctypes, bypassing `NativeRuntime` serialization: + +```python +resolver_entered = Event() +release_resolver = Event() +destroy_returned = Event() + +@RESOLVE_MODULE_CALLBACK +def resolver(_thread, _ctx, _path): + resolver_entered.set() + assert release_resolver.wait(5) + return module_source_address +``` + +Thread A attaches and calls `run_script_engine` on a resolver-backed engine. Thread B attaches only after `resolver_entered`, calls `destroy_engine`, and sets `destroy_returned` afterward. Assert in the parent process logic that `destroy_returned.wait(0.1)` is false before releasing the callback, then true after run completion. Retain the resolver source buffer until destroy returns. + +- [ ] **Step 3: Prove the new tests fail against an unmodified base-branch native library** + +Create a disposable verification worktree at the base branch and build its native library: + +```bash +git worktree add --detach \ + "/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red" \ + w-23692110-multi-engine-design +GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" \ +JAVA_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" \ + ./gradlew native-lib:nativeCompile -PskipStripDebug=true +``` + +Run the fix branch's new tests against that base library: + +```bash +cd native-lib/python +DATAWEAVE_NATIVE_LIB="/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red/native-lib/build/native/nativeCompile/dwlib.dylib" \ + python3 -m pytest tests/integration/test_lifecycle.py -k "raw_abi" -q +``` + +Expected on the old implementation: the null-resolver child exits `99`, and the destroy-drain assertion reports that destroy returned while the resolver remained blocked. Remove the disposable worktree afterward with `git worktree remove "/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red"`; do not edit or commit from it. + +- [ ] **Step 4: Run the native tests against the fixed library and verify GREEN** + +Run: + +```bash +export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" +export JAVA_HOME="$GRAALVM_HOME" +./gradlew native-lib:nativeCompile -PskipStripDebug=true +cd native-lib/python +DATAWEAVE_NATIVE_LIB="../build/native/nativeCompile/dwlib.dylib" \ + python3 -m pytest tests/integration/test_lifecycle.py -k "raw_abi" -q +``` + +Expected: child processes exit `0`; destroy remains blocked until the callback is released. + +- [ ] **Step 5: Commit the raw ABI regressions** + +```bash +git add native-lib/python/tests/integration/test_lifecycle.py +git commit -m "test(native-lib): cover C ABI failure and lease contracts" +``` + +### Task 4: Python Callback Reentrancy Guard + +**Files:** +- Modify: `native-lib/python/src/dataweave/native.py:1-12,344-623` +- Modify: `native-lib/python/src/dataweave/runtime.py:44-75,122-134,225-249,263-287` +- Modify: `native-lib/python/tests/unit/test_native.py` +- Modify: `native-lib/python/tests/unit/test_streaming.py:98-175` +- Modify: `native-lib/python/tests/integration/test_module_resolver.py:240-323` + +**Interfaces:** +- Consumes: existing `DataWeaveError` and callback wrappers. +- Produces: `_native_callback_scope()`, `_raise_if_native_callback_active()`, and process-wide thread-local callback depth. + +- [ ] **Step 1: Add isolated cross-engine resolver reentry test** + +Add a child-process test next to `test_overlapping_resolver_aware_runs_are_serialized`. The child initializes `inner` and resolver-backed `outer`; the outer resolver calls `inner.run("40 + 2")`, catches `DataWeaveError`, and returns `%dw 2.0\nfun answer() = 42`. Print JSON containing nested error type/message and outer result. + +Parent assertions: + +```python +assert completed.returncode == 0 +assert response["nested_type"] == "DataWeaveError" +assert "native callback" in response["nested_error"].lower() +assert response["outer"] == {"success": True, "value": "42"} +assert "Fatal error" not in completed.stderr +``` + +- [ ] **Step 2: Run the integration test and verify RED** + +Run against the staged native library: + +```bash +cd native-lib/python +python3 -m pytest tests/integration/test_module_resolver.py -k "cross_engine_resolver_reentry" -q +``` + +Expected: child exits `99` with GraalVM thread-state fatal stderr. + +- [ ] **Step 3: Add unit tests for shared thread-local guard behavior** + +Test that callback scope on one thread rejects `capture_operation`, `initialize`, and `cleanup` on any instance on that same thread, but does not reject another Python thread. Extend write/read callback tests so reentry through a second `DataWeave` instance returns callback status `-1` and never calls its native attach function. + +- [ ] **Step 4: Implement callback TLS and guard all isolate-touching public paths** + +Import `local` and define: + +```python +_native_callback_state = local() + +def _raise_if_native_callback_active() -> None: + if getattr(_native_callback_state, "depth", 0) > 0: + raise DataWeaveError( + "DataWeave lifecycle and execution are not allowed from a native callback on the same thread." + ) + +@contextmanager +def _native_callback_scope(): + previous = getattr(_native_callback_state, "depth", 0) + _native_callback_state.depth = previous + 1 + try: + yield + finally: + if previous == 0: + del _native_callback_state.depth + else: + _native_callback_state.depth = previous +``` + +Wrap only the direct user callback invocation, not parsing or diagnostics. Apply it to resolver, public write callback, and public read callback wrappers. Check `_raise_if_native_callback_active()` before lifecycle locks and before serialized native admission. Do not hold `_isolate_lock`, `_resolver_lock_global`, or an instance operation lock while user code runs. + +- [ ] **Step 5: Run focused Python tests and verify GREEN** + +```bash +cd native-lib/python +python3 -m pytest tests/unit/test_native.py -k "native_callback" -q +python3 -m pytest tests/unit/test_streaming.py -k "reentry" -q +python3 -m pytest tests/integration/test_module_resolver.py -k "cross_engine_resolver_reentry" -q +``` + +Expected: all selected tests pass; the child process remains alive. + +- [ ] **Step 6: Commit the Python callback guard** + +```bash +git add native-lib/python/src/dataweave/native.py \ + native-lib/python/src/dataweave/runtime.py \ + native-lib/python/tests/unit/test_native.py \ + native-lib/python/tests/unit/test_streaming.py \ + native-lib/python/tests/integration/test_module_resolver.py +git commit -m "fix(python): reject native callback reentrancy" +``` + +### Task 5: Python Atomic Admission and Generation-Bound Streams + +**Files:** +- Modify: `native-lib/python/src/dataweave/native.py:344-492,552-600` +- Modify: `native-lib/python/src/dataweave/runtime.py:87-103,109-223,251-287` +- Modify: `native-lib/python/tests/unit/test_native.py` +- Modify: `native-lib/python/tests/unit/test_streaming.py` +- Modify: `native-lib/python/tests/integration/test_streaming.py` + +**Interfaces:** +- Consumes: `_raise_if_native_callback_active()` from Task 4. +- Produces: frozen `_EngineOperation(handle, generation)`, `capture_operation()`, `validate_operation()`, and token-taking native run methods. + +- [ ] **Step 1: Write deterministic paused-admission test** + +Initialize a wrapper with `FakeLibrary`. Patch `_require_initialized` so it captures and returns generation A, signals an event, and waits. While the run thread is paused, cleanup and reinitialize to generation B, then resume. Assert `DataWeaveError` contains `stale engine generation` and the fake `run_script_engine` was never called with B. + +Add a complementary test that blocks after serialized admission, starts cleanup, and proves the admitted operation uses A before cleanup destroys A. + +- [ ] **Step 2: Write stale stream tests before implementation** + +Parameterize `run_streaming` and `run_transform`: + +```python +stream = create_stream(runtime) +old_handle = runtime._native.handle +runtime.cleanup() +runtime.initialize() +assert runtime._native.handle != old_handle +with pytest.raises(dataweave.DataWeaveError, match="stale engine generation"): + next(stream) +``` + +Assert no worker registered, no attach occurred for the stale stream, and no callback entrypoint received the replacement handle. Add a handle-reuse variant where the fake returns the same numeric handle for both generations. + +- [ ] **Step 3: Run focused tests and verify RED** + +```bash +cd native-lib/python +python3 -m pytest tests/unit/test_native.py -k "admission_generation" -q +python3 -m pytest tests/unit/test_streaming.py -k "stale_generation" -q +``` + +Expected: the paused run executes replacement handle B, and old streams execute instead of raising. + +- [ ] **Step 4: Implement immutable operation tokens** + +Add: + +```python +@dataclass(frozen=True) +class _EngineOperation: + handle: int + generation: int +``` + +`NativeRuntime` keeps monotonic `_generation` and canonical `_engine_operation`. Publish a new token only after successful engine creation. Clear `_engine_operation` during cleanup but never reset `_generation`. + +`capture_operation()` fails when uninitialized and returns the immutable token. `_serialized_native_operation(expected)` checks callback TLS before waiting, acquires the existing per-instance lock, validates exact token equality, and yields the token. All native run methods accept `operation` explicitly and call `operation.handle`; none reads `self.handle` after admission. + +- [ ] **Step 5: Bind public calls and worker registration to captured tokens** + +Change `_require_initialized` to return `_EngineOperation`. Capture it in `run`, `run_callback`, `run_input_output_callback`, `run_streaming`, and `run_transform` before constructing callbacks or generators. + +Change `_stream_worker(operation, invoke, cancelled)` and `_register_stream_worker(worker, operation)`. Under `_stream_workers_lock`, reject `_cleaning_up`, validate the operation, then register. Native worker closures carry `operation` into the token-taking native method. + +Keep lock order `_stream_workers_lock -> brief token validation`; native execution must release its operation lock before `_unregister_stream_worker` takes the worker lock. + +- [ ] **Step 6: Run unit and real-native stale-stream tests and verify GREEN** + +```bash +cd native-lib/python +python3 -m pytest tests/unit/test_native.py -k "admission_generation" -q +python3 -m pytest tests/unit/test_streaming.py -k "stale_generation" -q +python3 -m pytest tests/integration/test_streaming.py -k "stale_generation" -q +``` + +Expected: stale operations fail before replacement-handle invocation; admitted work stays on its captured engine. + +- [ ] **Step 7: Run the complete Python unit lane** + +```bash +cd native-lib/python +python3 -m pytest -m unit -q +``` + +Expected: all unit tests pass. Update `configured_runtime` test helpers to initialize `_generation` and `_engine_operation` explicitly rather than weakening production fallbacks. + +- [ ] **Step 8: Commit Python generations** + +```bash +git add native-lib/python/src/dataweave/native.py \ + native-lib/python/src/dataweave/runtime.py \ + native-lib/python/tests/unit/test_native.py \ + native-lib/python/tests/unit/test_streaming.py \ + native-lib/python/tests/integration/test_streaming.py +git commit -m "fix(python): bind operations to engine generations" +``` + +### Task 6: Node Callback Reentrancy Guard + +**Files:** +- Modify: `native-lib/node/src/addon.c:31-59,2094-2231,2235-2719,3373-3453` +- Modify: `native-lib/node/src/ffi.ts` +- Modify: `native-lib/node/src/dataweave.ts:73-139,229-323` +- Create: `native-lib/node/tests/integration/resolver-reentrancy.test.ts` +- Create: `native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs` + +**Interfaces:** +- Consumes: existing N-API methods and `DataWeaveError`. +- Produces: addon error code `ERR_DATAWEAVE_CALLBACK_REENTRANCY` and TypeScript `callNative()` error mapping. + +- [ ] **Step 1: Write the child-process resolver reentry regression** + +The fixture creates inner and outer `DataWeave` instances. The resolver attempts `inner.run`, catches the error, returns a valid module, prints JSON, and cleans both instances in `finally`. Parent asserts exit `0`, nested error name `DataWeaveError`, outer result `42`, and no Graal fatal stderr. + +Add a raw-addon case in the same fixture that recursively calls `runScriptEngine` from `createEngineWithResolver` and records the addon's stable error code. + +- [ ] **Step 2: Run the resolver test and verify RED** + +```bash +cd native-lib/node +npm run test:integration -- tests/integration/resolver-reentrancy.test.ts +``` + +Expected: child exits `99` with `Must either be at a safepoint or in native mode`. + +- [ ] **Step 3: Implement OS-thread-local callback depth in the addon** + +Initialize a `uv_key_t` in the existing `uv_once` initializer. Add helpers: + +```c +static unsigned native_callback_depth(void); +static void native_callback_enter(void); +static void native_callback_exit(void); +static bool native_callback_active(void); +static napi_value throw_callback_reentrancy(napi_env env); +``` + +Wrap `napi_call_function` in `resolve_module_callback`, `call_js_read`, and output callback bridges with enter/exit on all statuses. Reject callback reentry at the start of create, create-with-resolver, destroy, synchronous run, stream start, transform start, and cleanup before any state mutation or attachment. Set JavaScript error property `code` to `ERR_DATAWEAVE_CALLBACK_REENTRANCY` before throwing. + +- [ ] **Step 4: Map the native error to DataWeaveError** + +In `ffi.ts`, normalize calls through: + +```ts +function callNative(invoke: () => T): T { + try { + return invoke(); + } catch (error) { + if (error && typeof error === "object" && + "code" in error && error.code === "ERR_DATAWEAVE_CALLBACK_REENTRANCY") { + throw new DataWeaveError(String((error as Error).message)); + } + throw error; + } +} +``` + +Apply it to every isolate-touching wrapper. Avoid wrapping promise rejections twice; only normalize synchronous native admission errors. + +- [ ] **Step 5: Build and run resolver tests and verify GREEN** + +```bash +cd native-lib/node +npm run build:addon +npm run build:ts +npm run test:integration -- tests/integration/resolver-reentrancy.test.ts +``` + +Expected: build exits `0`; child processes survive and return the intended typed errors. + +- [ ] **Step 6: Commit the Node callback guard** + +```bash +git add native-lib/node/src/addon.c native-lib/node/src/ffi.ts \ + native-lib/node/src/dataweave.ts \ + native-lib/node/tests/integration/resolver-reentrancy.test.ts \ + native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs +git commit -m "fix(node): reject native callback reentrancy" +``` + +### Task 7: Node Generation-Bound Lazy Streams + +**Files:** +- Modify: `native-lib/node/src/dataweave.ts:49-55,81-139,186-216,229-323` +- Modify: `native-lib/node/tests/unit/dataweave-initialize.test.ts` +- Modify: `native-lib/node/tests/integration/instance-lifecycle.test.ts` + +**Interfaces:** +- Consumes: existing handle-based FFI and `DataWeaveError`. +- Produces: internal `EngineOperationToken`, `captureOperationToken()`, `assertCurrentOperation(token)`, and private token-taking async generators. + +- [ ] **Step 1: Add unit tests for stale streams and handle reuse** + +Mock FFI to return handle `2`, create `runStreaming()` without iterating, cleanup, initialize with handle `3`, and assert first pull rejects before `runScriptStreamingEngine` is called. Repeat with handle `2` reused; generation must still reject. Mirror both cases for `runTransform`. + +Add an async transform-input test that pauses `createChunkReader` consumption, performs cleanup/reinitialize, resumes input, and asserts no transform native call. + +- [ ] **Step 2: Run the Node generation tests and verify RED** + +```bash +cd native-lib/node +npm run test:unit -- tests/unit/dataweave-initialize.test.ts -t "stale engine generation" +``` + +Expected: old streams call FFI with the replacement handle or otherwise fail the no-call assertion. + +- [ ] **Step 3: Implement immutable Node tokens** + +Add: + +```ts +interface EngineOperationToken { + readonly handle: number; + readonly generation: number; +} +``` + +Increment `engineGeneration` only after successful handle publication. Never reset it in cleanup. `captureOperationToken()` checks readiness and returns current identity. `assertCurrentOperation()` requires state ready, exact handle, and exact generation; stale work throws `DataWeaveError("DataWeave operation belongs to a stale engine generation.")`. + +Convert public stream APIs to ordinary methods: + +```ts +runStreaming(...): AsyncGenerator { + const token = this.captureOperationToken(); + return this.runStreamingInternal(token, script, inputs); +} +``` + +Private async generators validate immediately before native admission and use `token.handle`. `runTransformInternal` validates before and after async input pre-buffering. + +- [ ] **Step 4: Add real-native stale stream coverage** + +In `instance-lifecycle.test.ts`, hold an anchor instance initialized so Java handles do not reset with isolate teardown. Create target stream under old handle, cleanup/reinitialize target, then consume old stream and assert `DataWeaveError`; a new stream must still succeed. Cover stream and transform. + +- [ ] **Step 5: Run focused tests and typecheck and verify GREEN** + +```bash +cd native-lib/node +npm run test:unit -- tests/unit/dataweave-initialize.test.ts +npm run test:integration -- tests/integration/instance-lifecycle.test.ts +npm run build:ts +``` + +Expected: all commands exit `0` and FFI receives captured handles only. + +- [ ] **Step 6: Commit Node generations** + +```bash +git add native-lib/node/src/dataweave.ts \ + native-lib/node/tests/unit/dataweave-initialize.test.ts \ + native-lib/node/tests/integration/instance-lifecycle.test.ts +git commit -m "fix(node): bind lazy streams to engine generations" +``` + +### Task 8: TypeScript Streaming Operation and Consumer Credits + +**Files:** +- Modify: `native-lib/node/src/ffi.ts:4-27,58-87` +- Modify: `native-lib/node/src/stream.ts` +- Modify: `native-lib/node/src/dataweave.ts:49-56,160-216,254-313` +- Modify: `native-lib/node/tests/unit/stream.test.ts` +- Modify: `native-lib/node/tests/unit/dataweave-initialize.test.ts` + +**Interfaces:** +- Consumes: token-taking stream methods from Task 7. +- Produces: internal `NativeStreamingOperation` with `completion`, `acknowledge`, `cancel`, and `close`; active-operation cleanup tracking. + +- [ ] **Step 1: Rewrite unit fakes around a streaming controller and add credit assertions** + +Define a test helper: + +```ts +function operation(completion: Promise) { + return { + completion, + acknowledge: vi.fn(), + cancel: vi.fn(), + close: vi.fn(), + }; +} +``` + +Add tests proving: + +- pushing a chunk into the JS queue does not acknowledge it; +- the first `.next()` acknowledges exactly that chunk's byte length before yielding; +- draining buffered chunks acknowledges each once; +- `generator.return(undefined)` cancels and closes once; +- native rejection still acknowledges already-buffered chunks before throwing; +- zero-chunk completion closes without cancellation; +- cancel/close paths are idempotent. + +Rename the current test containing `(backpressure)` to describe parking/wakeup only. + +- [ ] **Step 2: Run stream unit tests and verify RED** + +```bash +cd native-lib/node +npm run test:unit -- tests/unit/stream.test.ts +``` + +Expected: TypeScript compilation fails because the start callback still returns `Promise` and has no credit methods. + +- [ ] **Step 3: Implement the internal controller contract** + +Export from `ffi.ts`: + +```ts +export interface NativeStreamingOperation { + readonly completion: Promise; + acknowledge(bytes: number): void; + cancel(): void; + close(): void; +} +``` + +Change addon method typings and wrappers to return this object for stream and transform. Change `StartStreaming` accordingly. + +In `streamFromNative`, call `start` once, attach completion handlers, acknowledge immediately when dequeuing, and use `try/finally` to cancel only when iteration ended before native completion. Always `close()` exactly once after settlement/finalization. When abandoning buffered chunks, acknowledge their lengths before clearing them. + +- [ ] **Step 4: Track and cancel active operations during DataWeave cleanup** + +Add `activeStreams: Set`. Register an operation when native start returns and unregister it from `streamFromNative`'s close callback. In `doCleanup()`, synchronously cancel a snapshot before `ffi.destroyEngine`; await their completion settlements without masking the primary destroy/cleanup error. + +Do not start native work at stream method call. Registration still occurs on first iteration after generation validation. + +- [ ] **Step 5: Run TypeScript tests and verify GREEN** + +```bash +cd native-lib/node +npm run test:unit -- tests/unit/stream.test.ts +npm run test:unit -- tests/unit/dataweave-initialize.test.ts +npm run build:ts +``` + +Expected: all commands exit `0`. + +- [ ] **Step 6: Commit the TypeScript flow contract** + +```bash +git add native-lib/node/src/ffi.ts native-lib/node/src/stream.ts \ + native-lib/node/src/dataweave.ts \ + native-lib/node/tests/unit/stream.test.ts \ + native-lib/node/tests/unit/dataweave-initialize.test.ts +git commit -m "fix(node): propagate streaming consumer credits" +``` + +### Task 9: Native Node Output Flow Control + +**Files:** +- Modify: `native-lib/node/src/addon.c:1124-2092,3378-3453` +- Create: `native-lib/node/tests/integration/stream-backpressure.test.ts` + +**Interfaces:** +- Consumes: TypeScript controller contract from Task 8. +- Produces: `output_flow_t`, finite TSFN queues, native acknowledge/cancel/close methods, and test-only flow statistics. + +- [ ] **Step 1: Add integration tests that inspect a paused producer** + +Extend the test-only addon interface with flow stats and fixed watermarks. Start a large `deferred=true` output, consume one chunk, pause, and poll until `paused` is true. Assert: + +```ts +expect(stats.peakBufferedChunks).toBeLessThanOrEqual(stats.highChunks + 1); +expect(stats.peakBufferedBytes).toBeLessThanOrEqual( + stats.highBytes + stats.largestChunkBytes +); +expect(completionSettled).toBe(false); +``` + +Resume slowly and verify complete ordered output and successful metadata. Mirror the pause/drain assertion for `runTransform`. Add early-return and cleanup-while-paused tests with bounded timeouts. + +- [ ] **Step 2: Run the backpressure test and verify RED** + +```bash +cd native-lib/node +npm run test:integration -- tests/integration/stream-backpressure.test.ts +``` + +Expected: native controller/stat hooks are missing, or peak buffering exceeds the intended bound. + +- [ ] **Step 3: Implement `output_flow_t` and lifetime rules** + +Add fixed constants: + +```c +#define OUTPUT_HIGH_BYTES (1024 * 1024) +#define OUTPUT_LOW_BYTES (512 * 1024) +#define OUTPUT_HIGH_CHUNKS 128 +#define OUTPUT_LOW_CHUNKS 64 +#define OUTPUT_TSFN_QUEUE_SIZE 129 +``` + +`output_flow_t` contains mutex, condition, outstanding/peak counters, largest chunk, paused/cancelled/done flags, and refcount. Implement create/retain/release/reserve/acknowledge/cancel/mark_done. `reserve` waits only on the native producer thread and admits one oversized chunk when the window is empty. + +Each `chunk_data` records its flow pointer and accounted length. Reserve before allocation/enqueue; on OOM or TSFN enqueue failure, release the reserved credit. Non-sentinel JS callbacks keep credit outstanding after copying into a Buffer. Sentinel and env-dead paths cancel/settle and release ownership exactly once. + +- [ ] **Step 4: Return native controller objects** + +Instead of returning the bare promise, create an object with named properties/methods: + +```text +completion: Promise +acknowledge(bytes): void +cancel(): void +close(): void +``` + +Each method resolves the operation flow through N-API external data or a finalizer-safe holder. Validate bytes as a non-negative integer and make cancel/close idempotent. The holder keeps the flow alive until both worker and JS controller release it. + +Use finite output TSFN queues for streaming and transform writes. Keep transform read TSFN behavior unchanged. + +- [ ] **Step 5: Make cancellation unblock every producer path** + +Cancellation broadcasts the flow condition. Write callbacks return `-1` after cancellation. Generator abandonment, cleanup, env teardown, JS callback allocation/call failure, and controller finalization all route through the same idempotent cancel. Never wait on a flow condition while holding `g_mutex`; release the flow mutex before bridge/global completion accounting. + +- [ ] **Step 6: Run addon, focused unit, and integration tests and verify GREEN** + +```bash +cd native-lib/node +npm run build:addon +npm run build:ts +npm run test:unit -- tests/unit/stream.test.ts +npm run test:integration -- tests/integration/stream-backpressure.test.ts +npm run test:integration -- tests/integration/teardown-deadlock.test.ts +``` + +Expected: all commands exit `0`; paused producer counters remain bounded; early return and cleanup do not hang. + +- [ ] **Step 7: Commit native backpressure** + +```bash +git add native-lib/node/src/addon.c \ + native-lib/node/tests/integration/stream-backpressure.test.ts +git commit -m "fix(node): bound asynchronous output buffering" +``` + +### Task 10: Detach-Poison Failure Injection and Fail-Closed Admission + +**Files:** +- Modify: `native-lib/node/src/addon.c:135-154,237-297,455-505,1241-1341,1767-1866,2235-2719,2761-2876,3378-3453` +- Create: `native-lib/node/tests/integration/detach-poison-hook.test.ts` +- Create: `native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs` +- Create: `native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs` +- Modify: `native-lib/node/vitest.config.ts:27-33` + +**Interfaces:** +- Consumes: existing `g_isolate_poisoned` cleanup behavior. +- Produces: `detach_thread_checked(detach_site_t, void*)`, one-shot fault hooks, counters, and rejection of new work on a poisoned isolate. + +- [ ] **Step 1: Add child-process poison tests** + +Synchronous fixture sequence: + +```text +initialize -> create engine -> arm sync-run detach failure -> run succeeds +-> poison is true -> new run/create rejects -> destroy + cleanup settle +-> teardown count unchanged, abandon count +1 -> initialize fresh -> run succeeds +``` + +Transform fixture arms the transform-worker site, starts final cleanup while the operation is active, completes the worker, and asserts cleanup skips teardown and resolves. Parent tests enforce timeouts and reject any exit `99`, signal, or fatal stderr. + +- [ ] **Step 2: Run the poison tests and verify RED** + +```bash +cd native-lib/node +npm run test:integration -- tests/integration/detach-poison-hook.test.ts +``` + +Expected: required test hooks are undefined. + +- [ ] **Step 3: Centralize ordinary detach calls** + +Add `detach_site_t` entries for bridge finalize, stream worker, transform worker, create engine, create rollback, resolver create, unknown destroy, and sync run. Replace the eight ordinary detach sites with `detach_thread_checked(site, thread)`. Leave teardown-helper follow-up detach calls direct because they classify teardown-plus-detach double failure rather than ordinary operation poison. + +Under test hooks, call real detach first and substitute nonzero only when the selected one-shot site is armed and real detach succeeded. Count forced failures, isolate creates, teardown calls, and abandon operations under `g_mutex`. + +- [ ] **Step 4: Reject new admission after poison** + +In create/run/stream/transform admission critical sections, add `g_isolate_poisoned` to the rejection condition. Use a stable error `DataWeave isolate is unavailable after a thread detach failure; clean up and initialize again.` Existing admitted work continues draining; final cleanup follows `CLEANUP_UNRECOVERABLE` and abandons published state. + +- [ ] **Step 5: Export test-only hooks and stats** + +When `DATAWEAVE_TEST_HOOKS` is enabled, export: + +```text +__test_forceDetachFailureOnce(site) +__test_isolatePoisoned() +__test_isolateCreationCount() +__test_teardownCallCount() +__test_abandonedIsolateCount() +``` + +Reject unknown site strings synchronously. Update the Vitest comment listing enabled hooks. + +- [ ] **Step 6: Build and run poison/lifecycle tests and verify GREEN** + +```bash +cd native-lib/node +npm run build:addon +npm run test:integration -- tests/integration/detach-poison-hook.test.ts +npm run test:integration -- tests/integration/engine-strand-hook.test.ts +npm run test:integration -- tests/integration/instance-lifecycle.test.ts +``` + +Expected: all commands exit `0`; cleanup never hangs; fresh isolate recovery succeeds. + +- [ ] **Step 7: Commit detach-poison coverage** + +```bash +git add native-lib/node/src/addon.c native-lib/node/vitest.config.ts \ + native-lib/node/tests/integration/detach-poison-hook.test.ts \ + native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs \ + native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs +git commit -m "test(node): inject detach failures across isolate recovery" +``` + +### Task 11: Documentation and Whitespace Contract + +**Files:** +- Modify: `docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md:3-4,369-382,465` +- Modify: `native-lib/python/src/dataweave/native.py:624` +- Modify: `native-lib/README.md` +- Modify: `native-lib/node/README.md` +- Modify: `native-lib/python/README.md` +- Modify: `docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md` + +**Interfaces:** +- Consumes: final behavior from Tasks 1-10. +- Produces: accurate public ABI, lifecycle, reentrancy, streaming, cancellation, and failure-injection documentation. + +- [ ] **Step 1: Remove only the reported whitespace errors** + +Remove trailing spaces from the superseded Node design and remove the extra final blank line in `native.py`. Do not run a repository-wide formatter. + +- [ ] **Step 2: Update raw ABI documentation** + +In `native-lib/README.md`, document: + +```text +create_engine* returns 0 on entrypoint failure. +run_*_engine returns NULL on entrypoint-level failure; do not free NULL. +Normal script failures remain non-NULL JSON envelopes. +destroy_engine closes admission and blocks until admitted operations drain. +Resolver/read/write ctx storage must remain valid through destroy_engine return. +destroy_engine must not be called synchronously from that engine's callback. +``` + +- [ ] **Step 3: Update binding documentation** + +In both binding READMEs, document same-thread callback reentrancy rejection and stale-generation stream behavior. In Node docs, document internal byte/chunk watermarks as implementation details, cleanup cancellation of abandoned streams, and the fact that yielded buffers retained by user code are outside the bound. + +Change the large-file writable example to await `drain`: + +```ts +if (!output.write(chunk)) { + await once(output, "drain"); +} +``` + +Import `once` from `node:events` and preserve existing ESM/CommonJS style in that example. + +- [ ] **Step 4: Update the consolidated design** + +Replace `ConcurrentHashMap`/`ScriptRuntime.get` descriptions with core lifecycle records and leases. Add binding generations, callback TLS, bounded output flow, cancel-on-cleanup, explicit exception sentinels, and detach test hooks. Correct Python stream cleanup wording to match active-worker refusal plus generation-safe registration. + +- [ ] **Step 5: Verify documentation claims against symbols and tests** + +Search all documented C names against `NativeLib.java`, `_bind_abi`, and `addon.c`; remove stale or invented names. Confirm every new error phrase matches production source exactly. + +- [ ] **Step 6: Run whitespace verification and commit** + +```bash +git diff --check w-23692110-multi-engine-design...HEAD +git add docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md \ + docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md \ + native-lib/README.md native-lib/node/README.md native-lib/python/README.md \ + native-lib/python/src/dataweave/native.py +git commit -m "docs(native-lib): document hardened multi-engine contracts" +``` + +Expected: `git diff --check` exits `0` before commit. + +### Task 12: Full Verification, Review, Push, and PR + +**Files:** +- Modify only if verification or review reveals a defect in files already in scope. + +**Interfaces:** +- Consumes: every preceding task. +- Produces: clean, reviewed branch and PR to `w-23692110-multi-engine-design`. + +- [ ] **Step 1: Inspect branch state and commit range** + +```bash +git status --short +git log --oneline --decorate w-23692110-multi-engine-design..HEAD +git diff --stat w-23692110-multi-engine-design...HEAD +git diff --check w-23692110-multi-engine-design...HEAD +``` + +Expected: no unintended tracked changes, focused commits only, and no whitespace errors. + +- [ ] **Step 2: Run complete hosted Java verification** + +```bash +./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true +``` + +Expected: exit `0`. If this task still triggers `nativeCompile` through plugin wiring, use the GraalVM environment from the next step rather than falling back to JDK 17. + +- [ ] **Step 3: Build native library with GraalVM 24** + +```bash +export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" +export JAVA_HOME="$GRAALVM_HOME" +./gradlew native-lib:nativeCompile -PskipStripDebug=true +``` + +Expected: exit `0` with `dwlib.dylib` generated. + +- [ ] **Step 4: Run complete Python unit and integration lane** + +```bash +cd native-lib/python +DATAWEAVE_NATIVE_LIB="../build/native/nativeCompile/dwlib.dylib" \ + python3 -m pytest -m "unit or integration" -q +``` + +Expected: all selected tests pass with no hangs or fatal child exits. + +- [ ] **Step 5: Build and run complete Node unit/integration lanes** + +```bash +cd native-lib/node +npm install +npm run build:addon +npm run build:ts +npm run test:unit +npm run test:integration +``` + +Expected: all commands exit `0`; no unhandled rejections, Worker nonzero exits, or timeout hangs. + +- [ ] **Step 6: Run normal native-lib Gradle verification** + +```bash +export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" +export JAVA_HOME="$GRAALVM_HOME" +./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true +./gradlew build -PskipNodeTests=true +``` + +Expected: both commands exit `0`. The second command includes the repository's configured Python lane; Node was already run directly to preserve focused output. + +- [ ] **Step 7: Perform three-lens code review** + +Review the complete PR diff for: + +```text +General correctness: stale state, exception masking, duplicate cleanup, error timing. +Native/concurrency: lock order, lease/flow refcounts, callback thread affinity, env death, cancellation. +Security: pointer validation, callback input lengths, tenant data in logs, raw ABI nullability. +``` + +For each accepted finding, add a failing regression test first, implement the smallest correction, rerun the focused test, then rerun the affected module lane. Commit review fixes separately with a concise message. + +- [ ] **Step 8: Verify final clean evidence** + +```bash +git status --short +git diff --check w-23692110-multi-engine-design...HEAD +git log --oneline --decorate w-23692110-multi-engine-design..HEAD +git diff --stat w-23692110-multi-engine-design...HEAD +``` + +Expected: only ignored build artifacts may exist; no uncommitted source/doc changes; diff check exits `0`. + +- [ ] **Step 9: Push the branch** + +```bash +git push -u origin w-23692110-review-22-fixes +``` + +Expected: remote tracking branch created without force. + +- [ ] **Step 10: Create the PR targeting the multi-engine branch** + +Before creation, inspect remote tracking and the full range: + +```bash +git status --short +git branch -vv +git log --oneline origin/w-23692110-multi-engine-design..HEAD +git diff --stat origin/w-23692110-multi-engine-design...HEAD +``` + +Create the PR: + +```bash +gh pr create \ + --base w-23692110-multi-engine-design \ + --head w-23692110-review-22-fixes \ + --title "@W-23692110: Harden multi-engine lifecycle and streaming" \ + --body-file /tmp/pr157-review22-body.md +``` + +The body must summarize all eight resolved findings, list exact verification commands/results, call out the intentional destroy-blocking and callback-reentrancy contracts, and state that this PR layers onto PR #157 rather than targeting `master`. + +- [ ] **Step 11: Report the PR URL and residual risks** + +Return the PR URL, commit count, final test counts, and any unrun platform-only checks. Do not claim hosted CI passes until GitHub reports it. From 46b947dad5d3022409d364ed6a5399fda733e189 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 09:43:44 -0300 Subject: [PATCH 03/48] fix(native-lib): lease engines across admitted operations --- .../org/mule/weave/lib/ScriptRuntime.java | 106 +++++- .../weave/lib/ScriptRuntimeLifecycleTest.java | 310 ++++++++++++++++++ .../org/mule/weave/lib/ScriptRuntimeTest.java | 79 +++-- 3 files changed, 458 insertions(+), 37 deletions(-) create mode 100644 native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java diff --git a/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java b/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java index b8857313..bc0bbdad 100644 --- a/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java +++ b/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java @@ -20,6 +20,7 @@ import java.io.InputStream; import java.nio.charset.Charset; import java.util.Base64; +import java.util.Objects; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.AtomicLong; @@ -38,24 +39,119 @@ public class ScriptRuntime { // ── Handle registry ────────────────────────────────────────────────── - private static final ConcurrentHashMap REGISTRY = new ConcurrentHashMap<>(); + private static final ConcurrentHashMap REGISTRY = new ConcurrentHashMap<>(); private static final AtomicLong NEXT_HANDLE = new AtomicLong(1); /** Registers a runtime and returns its non-zero handle. */ public static long register(ScriptRuntime runtime) { + Objects.requireNonNull(runtime, "runtime"); long handle = NEXT_HANDLE.getAndIncrement(); - REGISTRY.put(handle, runtime); + if (handle <= 0) { + throw new IllegalStateException("Engine handle space exhausted"); + } + REGISTRY.put(handle, new EngineRecord(runtime)); return handle; } /** Returns the runtime for a handle, or {@code null} if unknown/destroyed. */ public static ScriptRuntime get(long handle) { - return REGISTRY.get(handle); + EngineRecord record = REGISTRY.get(handle); + return record == null ? null : record.runtime; + } + + /** Acquires a lease for a live runtime, or returns {@code null} if admission is closed. */ + public static EngineLease acquire(long handle) { + EngineRecord record = REGISTRY.get(handle); + return record == null ? null : record.tryAcquire(); } - /** Removes a runtime; returns {@code true} if one was present. */ + /** Closes admission, drains active leases, and removes the runtime. */ public static boolean destroy(long handle) { - return REGISTRY.remove(handle) != null; + EngineRecord record = REGISTRY.get(handle); + if (record == null) { + return false; + } + record.closeAndAwait(); + REGISTRY.remove(handle, record); + return true; + } + + public static final class EngineLease implements AutoCloseable { + private final EngineRecord record; + private final ScriptRuntime runtime; + private boolean closed; + + private EngineLease(EngineRecord record, ScriptRuntime runtime) { + this.record = record; + this.runtime = runtime; + } + + public ScriptRuntime runtime() { + return runtime; + } + + @Override + public void close() { + synchronized (this) { + if (closed) { + return; + } + closed = true; + } + record.release(); + } + } + + private static final class EngineRecord { + private enum State { + LIVE, + CLOSING, + DESTROYED + } + + private final ScriptRuntime runtime; + private State state = State.LIVE; + private int activeLeases; + + private EngineRecord(ScriptRuntime runtime) { + this.runtime = runtime; + } + + private synchronized EngineLease tryAcquire() { + if (state != State.LIVE) { + return null; + } + activeLeases++; + return new EngineLease(this, runtime); + } + + private void closeAndAwait() { + boolean interrupted = false; + synchronized (this) { + if (state == State.LIVE) { + state = State.CLOSING; + } + while (state != State.DESTROYED && activeLeases > 0) { + try { + wait(); + } catch (InterruptedException e) { + interrupted = true; + } + } + state = State.DESTROYED; + notifyAll(); + } + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + + private synchronized void release() { + activeLeases--; + if (activeLeases == 0) { + notifyAll(); + } + } } // ── Per-instance engine ─────────────────────────────────────────────── diff --git a/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java b/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java new file mode 100644 index 00000000..7953c391 --- /dev/null +++ b/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java @@ -0,0 +1,310 @@ +package org.mule.weave.lib; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Test; + +import java.lang.reflect.Field; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.BooleanSupplier; + +class ScriptRuntimeLifecycleTest { + + @Test + void destroyWaitsForAnAdmittedLeaseAndRejectsNewAdmission() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + CountDownLatch destroyStarted = new CountDownLatch(1); + AtomicBoolean destroyReturned = new AtomicBoolean(false); + Thread destroyer = daemonThread(() -> { + destroyStarted.countDown(); + ScriptRuntime.destroy(handle); + destroyReturned.set(true); + }); + + try { + assertNotNull(lease); + destroyer.start(); + assertTrue(destroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> admissionIsClosed(handle)); + assertFalse(destroyReturned.get()); + } finally { + if (lease != null) { + lease.close(); + } + if (destroyer.getState() != Thread.State.NEW) { + destroyer.join(1_000); + } + ScriptRuntime.destroy(handle); + } + + assertFalse(destroyer.isAlive()); + assertTrue(destroyReturned.get()); + assertCannotAcquire(handle); + } + + @Test + void multipleLeasesMustAllDrainBeforeDestroyReturns() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease firstLease = ScriptRuntime.acquire(handle); + ScriptRuntime.EngineLease secondLease = ScriptRuntime.acquire(handle); + + Thread destroyer = null; + AtomicBoolean destroyReturned = new AtomicBoolean(false); + try { + assertNotNull(firstLease); + assertNotNull(secondLease); + CountDownLatch destroyStarted = new CountDownLatch(1); + destroyer = daemonThread(() -> { + destroyStarted.countDown(); + ScriptRuntime.destroy(handle); + destroyReturned.set(true); + }); + destroyer.start(); + assertTrue(destroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> admissionIsClosed(handle)); + firstLease.close(); + assertFalse(destroyReturned.get()); + secondLease.close(); + destroyer.join(1_000); + } finally { + if (firstLease != null) { + firstLease.close(); + } + if (secondLease != null) { + secondLease.close(); + } + if (destroyer != null) { + destroyer.join(1_000); + } + ScriptRuntime.destroy(handle); + } + + assertFalse(destroyer.isAlive()); + assertTrue(destroyReturned.get()); + } + + @Test + void concurrentDestroyCallsCoordinateAndComplete() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + CountDownLatch firstDestroyStarted = new CountDownLatch(1); + CountDownLatch secondDestroyStarted = new CountDownLatch(1); + AtomicBoolean firstResult = new AtomicBoolean(false); + AtomicBoolean secondResult = new AtomicBoolean(false); + Thread firstDestroyer = destroyThread(handle, firstDestroyStarted, firstResult); + Thread secondDestroyer = destroyThread(handle, secondDestroyStarted, secondResult); + + try { + assertNotNull(lease); + firstDestroyer.start(); + assertTrue(firstDestroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> admissionIsClosed(handle)); + awaitCondition(() -> isWaiting(firstDestroyer)); + secondDestroyer.start(); + assertTrue(secondDestroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> isWaiting(secondDestroyer)); + } finally { + if (lease != null) { + lease.close(); + } + if (firstDestroyer.getState() != Thread.State.NEW) { + firstDestroyer.join(1_000); + } + if (secondDestroyer.getState() != Thread.State.NEW) { + secondDestroyer.join(1_000); + } + ScriptRuntime.destroy(handle); + } + + assertFalse(firstDestroyer.isAlive()); + assertFalse(secondDestroyer.isAlive()); + assertTrue(firstResult.get()); + assertTrue(secondResult.get()); + assertCannotAcquire(handle); + } + + @Test + void closingAnEngineLeaseTwiceIsHarmless() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease firstLease = ScriptRuntime.acquire(handle); + ScriptRuntime.EngineLease secondLease = null; + AtomicBoolean destroyReturned = new AtomicBoolean(false); + Thread destroyer = null; + + try { + assertNotNull(firstLease); + firstLease.close(); + firstLease.close(); + secondLease = ScriptRuntime.acquire(handle); + assertNotNull(secondLease); + destroyer = daemonThread(() -> { + ScriptRuntime.destroy(handle); + destroyReturned.set(true); + }); + destroyer.start(); + awaitCondition(() -> admissionIsClosed(handle)); + assertFalse(destroyReturned.get()); + } finally { + if (firstLease != null) { + firstLease.close(); + } + if (secondLease != null) { + secondLease.close(); + } + if (destroyer != null) { + destroyer.join(1_000); + } + ScriptRuntime.destroy(handle); + } + + assertFalse(destroyer.isAlive()); + assertTrue(destroyReturned.get()); + } + + @Test + void interruptedDestroyRestoresInterruptAfterTheLeaseDrains() throws Exception { + long handle = ScriptRuntime.register(new ScriptRuntime()); + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + CountDownLatch destroyStarted = new CountDownLatch(1); + AtomicBoolean destroyResult = new AtomicBoolean(false); + AtomicBoolean interruptedOnReturn = new AtomicBoolean(false); + Thread destroyer = daemonThread(() -> { + destroyStarted.countDown(); + destroyResult.set(ScriptRuntime.destroy(handle)); + interruptedOnReturn.set(Thread.currentThread().isInterrupted()); + }); + + try { + assertNotNull(lease); + destroyer.start(); + assertTrue(destroyStarted.await(1, TimeUnit.SECONDS)); + awaitCondition(() -> admissionIsClosed(handle)); + awaitCondition(() -> isWaiting(destroyer)); + destroyer.interrupt(); + awaitCondition(() -> isWaiting(destroyer) && !destroyer.isInterrupted()); + assertFalse(interruptedOnReturn.get()); + } finally { + if (lease != null) { + lease.close(); + } + if (destroyer.getState() != Thread.State.NEW) { + destroyer.join(1_000); + } + ScriptRuntime.destroy(handle); + } + + assertFalse(destroyer.isAlive()); + assertTrue(destroyResult.get()); + assertTrue(interruptedOnReturn.get()); + } + + @Test + void unknownHandleCannotAcquireALease() { + assertCannotAcquire(Long.MAX_VALUE); + } + + @Test + void leaseExposesTheRegisteredRuntime() { + ScriptRuntime runtime = new ScriptRuntime(); + long handle = ScriptRuntime.register(runtime); + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + + try { + assertNotNull(lease); + assertSame(runtime, lease.runtime()); + } finally { + if (lease != null) { + lease.close(); + } + ScriptRuntime.destroy(handle); + } + } + + @Test + void registerRejectsNullRuntime() { + assertThrows(NullPointerException.class, () -> ScriptRuntime.register(null)); + } + + @Test + void registerDoesNotPublishANonPositiveHandleAfterOverflow() throws Exception { + Field nextHandleField = ScriptRuntime.class.getDeclaredField("NEXT_HANDLE"); + nextHandleField.setAccessible(true); + AtomicLong nextHandle = (AtomicLong) nextHandleField.get(null); + long previousHandle = nextHandle.getAndSet(Long.MAX_VALUE); + long lastPositiveHandle = 0; + + try { + lastPositiveHandle = ScriptRuntime.register(new ScriptRuntime()); + assertEquals(Long.MAX_VALUE, lastPositiveHandle); + assertThrows(IllegalStateException.class, + () -> ScriptRuntime.register(new ScriptRuntime())); + assertCannotAcquire(Long.MIN_VALUE); + } finally { + if (lastPositiveHandle > 0) { + ScriptRuntime.destroy(lastPositiveHandle); + } + nextHandle.set(previousHandle); + } + } + + private static Thread destroyThread(long handle, CountDownLatch started, AtomicBoolean result) { + return daemonThread(() -> { + started.countDown(); + result.set(ScriptRuntime.destroy(handle)); + }); + } + + private static Thread daemonThread(Runnable action) { + Thread thread = new Thread(action); + thread.setDaemon(true); + return thread; + } + + private static boolean admissionIsClosed(long handle) { + ScriptRuntime.EngineLease probe = ScriptRuntime.acquire(handle); + if (probe == null) { + return true; + } + try { + return false; + } finally { + probe.close(); + } + } + + private static void assertCannotAcquire(long handle) { + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + try { + assertNull(lease); + } finally { + if (lease != null) { + lease.close(); + } + } + } + + private static boolean isWaiting(Thread thread) { + return thread.getState() == Thread.State.WAITING; + } + + private static void awaitCondition(BooleanSupplier condition) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(1); + while (System.nanoTime() < deadline) { + if (condition.getAsBoolean()) { + return; + } + Thread.onSpinWait(); + } + assertTrue(condition.getAsBoolean(), "Condition was not met within one second"); + } +} diff --git a/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java b/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java index 77fa2a43..fd439052 100644 --- a/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java +++ b/native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java @@ -635,34 +635,48 @@ void twoEnginesResolveOnlyTheirOwnModule() { long hA = ScriptRuntime.register(engineA); long hB = ScriptRuntime.register(engineB); - assertNotNull(ScriptRuntime.get(hA)); - assertNotNull(ScriptRuntime.get(hB)); - - // Each engine resolves its own module... - assertEquals("\"A:X\"", Result.parse(ScriptRuntime.get(hA).run(IMPORT_A)).result); - assertEquals("\"B:X\"", Result.parse(ScriptRuntime.get(hB).run(IMPORT_B)).result); - - // ...and NOT the other's (no cross-talk). - assertNotNull(Result.parse(ScriptRuntime.get(hA).run(IMPORT_B)).error); - assertNotNull(Result.parse(ScriptRuntime.get(hB).run(IMPORT_A)).error); - - // destroy removes it; a fresh handle is distinct. - assertTrue(ScriptRuntime.destroy(hA)); - assertNull(ScriptRuntime.get(hA)); - assertFalse(ScriptRuntime.destroy(hA)); // already gone - assertNotNull(ScriptRuntime.get(hB)); + try { + try (ScriptRuntime.EngineLease leaseA = ScriptRuntime.acquire(hA); + ScriptRuntime.EngineLease leaseB = ScriptRuntime.acquire(hB)) { + assertNotNull(leaseA); + assertNotNull(leaseB); + + // Each engine resolves its own module... + assertEquals("\"A:X\"", Result.parse(leaseA.runtime().run(IMPORT_A)).result); + assertEquals("\"B:X\"", Result.parse(leaseB.runtime().run(IMPORT_B)).result); + + // ...and NOT the other's (no cross-talk). + assertNotNull(Result.parse(leaseA.runtime().run(IMPORT_B)).error); + assertNotNull(Result.parse(leaseB.runtime().run(IMPORT_A)).error); + } - ScriptRuntime.destroy(hB); + // destroy removes it; a fresh handle is distinct. + assertTrue(ScriptRuntime.destroy(hA)); + assertCannotAcquire(hA); + assertFalse(ScriptRuntime.destroy(hA)); // already gone + try (ScriptRuntime.EngineLease leaseB = ScriptRuntime.acquire(hB)) { + assertNotNull(leaseB); + } + } finally { + ScriptRuntime.destroy(hA); + ScriptRuntime.destroy(hB); + } } @Test void engineWithoutResolverStillRunsBuiltins() { ScriptRuntime engine = new ScriptRuntime(); // ClassLoader-only long h = ScriptRuntime.register(engine); - String r = ScriptRuntime.get(h).run( - "%dw 2.0\nimport dw::core::Strings\noutput application/json\n---\nStrings::capitalize(\"hello\")"); - assertEquals("\"Hello\"", Result.parse(r).result); - ScriptRuntime.destroy(h); + try { + try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(h)) { + assertNotNull(lease); + String r = lease.runtime().run( + "%dw 2.0\nimport dw::core::Strings\noutput application/json\n---\nStrings::capitalize(\"hello\")"); + assertEquals("\"Hello\"", Result.parse(r).result); + } + } finally { + ScriptRuntime.destroy(h); + } } /** @@ -671,25 +685,26 @@ void engineWithoutResolverStillRunsBuiltins() { * {@code run_script_input_output_callback_engine} in {@link NativeLib}): running a * script against an unknown or already-destroyed engine handle must return exactly * {@code {"success":false,"error":"Unknown engine handle"}} rather than throwing. - * - *

The {@code @CEntryPoint} methods themselves cannot be invoked from a plain JVM - * unit test — they take GraalVM word types ({@code IsolateThread}, {@code CCharPointer}) - * whose boxing infrastructure is only initialized inside a compiled native image (calling - * e.g. {@code WordFactory.nullPointer()} from a hosted JVM test throws - * {@code NullPointerException} from {@code WordBoxFactory}). All three entrypoints funnel - * the unknown-handle case through the same {@code UNKNOWN_ENGINE_HANDLE_JSON} constant, so - * asserting on that constant — combined with {@link #twoEnginesResolveOnlyTheirOwnModule} - * proving {@link ScriptRuntime#get} returns {@code null} for an unregistered/destroyed - * handle — verifies the full contract without needing the native runtime.

*/ @Test void unknownEngineHandleProducesExactErrorJson() { long unregisteredHandle = Long.MAX_VALUE; - assertNull(ScriptRuntime.get(unregisteredHandle)); + assertCannotAcquire(unregisteredHandle); assertEquals("{\"success\":false,\"error\":\"Unknown engine handle\"}", NativeLib.UNKNOWN_ENGINE_HANDLE_JSON); } + private static void assertCannotAcquire(long handle) { + ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); + try { + assertNull(lease); + } finally { + if (lease != null) { + lease.close(); + } + } + } + // ── Fail-closed input parsing (review #11 #5) ────────────────────────── /** (a) Malformed inputs JSON must fail closed, not silently run on empty bindings. */ From 59e43405fde5b77cfe805b4d78989515977eb31e Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 10:15:15 -0300 Subject: [PATCH 04/48] fix(native-lib): contain C entrypoint exceptions --- native-lib/node/src/addon.c | 15 ++- .../lib/CEntryPointExceptionHandlers.java | 31 ++++++ .../java/org/mule/weave/lib/NativeLib.java | 95 +++++++++++++------ .../lib/NativeLibEntryPointContractTest.java | 50 ++++++++++ 4 files changed, 154 insertions(+), 37 deletions(-) create mode 100644 native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java create mode 100644 native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index aa1f0f2d..29544806 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -2266,11 +2266,10 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { long long handle = fn_create_engine(thread); int detach_rc = fn_detach_thread(thread); if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); - // A GraalVM @CEntryPoint that throws on the Java side returns the return - // type's default value instead of propagating the exception — 0 for a - // long long. The real handle registry only ever hands out handles >= 1, so - // any handle <= 0 means construction failed; never hand that back to JS as - // if it were usable. + // The Java @CEntryPoint exception handler explicitly returns 0 as its ABI + // exception sentinel when engine construction throws. The real handle + // registry only ever hands out handles >= 1, so any handle <= 0 means + // construction failed; never hand that back to JS as if it were usable. if (handle <= 0) { uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "create_engine returned an invalid handle"); return NULL; @@ -2406,9 +2405,9 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i int detach_rc = fn_detach_thread(thread); if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); - // Same invalid-handle guard as napi_create_engine: a Java-side construction - // failure surfaces here as handle == 0 (GraalVM @CEntryPoint default-value - // semantics), and any handle <= 0 is never valid. Reject before this bridge + // Same invalid-handle guard as napi_create_engine: the Java @CEntryPoint + // exception handler explicitly returns handle == 0 as its ABI sentinel, + // and any handle <= 0 is never valid. Reject before this bridge // is linked into g_bridges or a cleanup hook is registered for it — at this // point neither has happened, so there's nothing to unlink/unhook. Still use // bridge_finalize (not a manual napi_delete_reference+free) because the failed diff --git a/native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java b/native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java new file mode 100644 index 00000000..412d49e7 --- /dev/null +++ b/native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java @@ -0,0 +1,31 @@ +package org.mule.weave.lib; + +import com.oracle.svm.core.Uninterruptible; +import org.graalvm.nativeimage.c.function.CEntryPoint; +import org.graalvm.nativeimage.c.type.CCharPointer; +import org.graalvm.word.WordFactory; + +final class CEntryPointExceptionHandlers { + private CEntryPointExceptionHandlers() { + } + + static final class ReturnZero implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") + static long handle(Throwable ignored) { + return 0L; + } + } + + static final class ReturnNullPointer implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") + static CCharPointer handle(Throwable ignored) { + return WordFactory.nullPointer(); + } + } + + static final class ReturnVoid implements CEntryPoint.ExceptionHandler { + @Uninterruptible(reason = "Contain an exception at the C ABI boundary") + static void handle(Throwable ignored) { + } + } +} diff --git a/native-lib/src/main/java/org/mule/weave/lib/NativeLib.java b/native-lib/src/main/java/org/mule/weave/lib/NativeLib.java index 0770d087..4fd8e8c1 100644 --- a/native-lib/src/main/java/org/mule/weave/lib/NativeLib.java +++ b/native-lib/src/main/java/org/mule/weave/lib/NativeLib.java @@ -36,7 +36,8 @@ public class NativeLib { * @param thread the isolate thread (automatically provided by GraalVM) * @param pointer the pointer to the unmanaged C string to free; if null, this is a no-op */ - @CEntryPoint(name = "free_cstring") + @CEntryPoint(name = "free_cstring", + exceptionHandler = CEntryPointExceptionHandlers.ReturnVoid.class) public static void freeCString(IsolateThread thread, CCharPointer pointer) { if (pointer.isNull()) { return; @@ -537,7 +538,8 @@ private static CCharPointer toUnmanagedCString(String value) { * @param thread the isolate thread * @return a non-zero handle identifying the new engine */ - @CEntryPoint(name = "create_engine") + @CEntryPoint(name = "create_engine", + exceptionHandler = CEntryPointExceptionHandlers.ReturnZero.class) public static long createEngine(IsolateThread thread) { return ScriptRuntime.register(new ScriptRuntime()); } @@ -551,11 +553,15 @@ public static long createEngine(IsolateThread thread) { * @param ctx opaque context pointer forwarded to every resolver invocation * @return a non-zero handle identifying the new engine */ - @CEntryPoint(name = "create_engine_with_resolver") + @CEntryPoint(name = "create_engine_with_resolver", + exceptionHandler = CEntryPointExceptionHandlers.ReturnZero.class) public static long createEngineWithResolver( IsolateThread thread, NativeCallbacks.ResolveModuleCallback resolverCallback, PointerBase ctx) { + if (resolverCallback.isNull()) { + return 0L; + } CallbackWeaveResourceResolver resolver = new CallbackWeaveResourceResolver(resolverCallback, ctx); return ScriptRuntime.register(new ScriptRuntime(resolver)); @@ -568,7 +574,8 @@ public static long createEngineWithResolver( * @param thread the isolate thread * @param handle the engine handle to remove */ - @CEntryPoint(name = "destroy_engine") + @CEntryPoint(name = "destroy_engine", + exceptionHandler = CEntryPointExceptionHandlers.ReturnVoid.class) public static void destroyEngine(IsolateThread thread, long handle) { ScriptRuntime.destroy(handle); } @@ -585,16 +592,21 @@ public static void destroyEngine(IsolateThread thread, long handle) { * @param inputsJson JSON-encoded inputs map (C string), may be null * @return the script execution result (unmanaged C string, must be freed) */ - @CEntryPoint(name = "run_script_engine") + @CEntryPoint(name = "run_script_engine", + exceptionHandler = CEntryPointExceptionHandlers.ReturnNullPointer.class) public static CCharPointer runScriptEngine( IsolateThread thread, long handle, CCharPointer script, CCharPointer inputsJson) { - ScriptRuntime runtime = ScriptRuntime.get(handle); - if (runtime == null) { - return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle)) { + if (lease == null) { + return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + } + if (script.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Script cannot be null\"}"); + } + String dwScript = CTypeConversion.toJavaString(script); + String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); + return toUnmanagedCString(lease.runtime().run(dwScript, inputs)); } - String dwScript = CTypeConversion.toJavaString(script); - String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); - return toUnmanagedCString(runtime.run(dwScript, inputs)); } /** @@ -612,17 +624,25 @@ public static CCharPointer runScriptEngine( * @param ctx opaque context pointer forwarded to every callback invocation * @return an unmanaged C string with JSON metadata/error */ - @CEntryPoint(name = "run_script_callback_engine") + @CEntryPoint(name = "run_script_callback_engine", + exceptionHandler = CEntryPointExceptionHandlers.ReturnNullPointer.class) public static CCharPointer runScriptCallbackEngine( IsolateThread thread, long handle, CCharPointer script, CCharPointer inputsJson, NativeCallbacks.WriteCallback writeCallback, PointerBase ctx) { - ScriptRuntime runtime = ScriptRuntime.get(handle); - if (runtime == null) { - return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle)) { + if (lease == null) { + return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + } + if (script.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Script cannot be null\"}"); + } + if (writeCallback.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Write callback cannot be null\"}"); + } + String dwScript = CTypeConversion.toJavaString(script); + String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); + return streamToWriteCallback(lease.runtime(), dwScript, inputs, writeCallback, ctx); } - String dwScript = CTypeConversion.toJavaString(script); - String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); - return streamToWriteCallback(runtime, dwScript, inputs, writeCallback, ctx); } /** @@ -645,23 +665,40 @@ public static CCharPointer runScriptCallbackEngine( * @param ctx opaque context pointer forwarded to every callback invocation * @return an unmanaged C string with JSON metadata/error */ - @CEntryPoint(name = "run_script_input_output_callback_engine") + @CEntryPoint(name = "run_script_input_output_callback_engine", + exceptionHandler = CEntryPointExceptionHandlers.ReturnNullPointer.class) public static CCharPointer runScriptInputOutputCallbackEngine( IsolateThread thread, long handle, CCharPointer script, CCharPointer inputsJson, CCharPointer inputName, CCharPointer inputMimeType, CCharPointer inputCharset, NativeCallbacks.ReadCallback readCallback, NativeCallbacks.WriteCallback writeCallback, PointerBase ctx) { - ScriptRuntime runtime = ScriptRuntime.get(handle); - if (runtime == null) { - return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle)) { + if (lease == null) { + return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); + } + if (script.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Script cannot be null\"}"); + } + if (inputName.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Input name cannot be null\"}"); + } + if (inputMimeType.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Input MIME type cannot be null\"}"); + } + if (readCallback.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Read callback cannot be null\"}"); + } + if (writeCallback.isNull()) { + return toUnmanagedCString("{\"success\":false,\"error\":\"Write callback cannot be null\"}"); + } + String dwScript = CTypeConversion.toJavaString(script); + String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); + String inName = CTypeConversion.toJavaString(inputName); + String inMime = CTypeConversion.toJavaString(inputMimeType); + String inCharset = inputCharset.isNull() ? null : CTypeConversion.toJavaString(inputCharset); + return transformViaCallbacks(lease.runtime(), dwScript, inputs, inName, inMime, inCharset, + readCallback, writeCallback, ctx); } - String dwScript = CTypeConversion.toJavaString(script); - String inputs = inputsJson.isNull() ? null : CTypeConversion.toJavaString(inputsJson); - String inName = CTypeConversion.toJavaString(inputName); - String inMime = CTypeConversion.toJavaString(inputMimeType); - String inCharset = inputCharset.isNull() ? null : CTypeConversion.toJavaString(inputCharset); - return transformViaCallbacks(runtime, dwScript, inputs, inName, inMime, inCharset, - readCallback, writeCallback, ctx); } } diff --git a/native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java b/native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java new file mode 100644 index 00000000..7d3ae496 --- /dev/null +++ b/native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java @@ -0,0 +1,50 @@ +package org.mule.weave.lib; + +import org.graalvm.nativeimage.IsolateThread; +import org.graalvm.nativeimage.c.function.CEntryPoint; +import org.graalvm.nativeimage.c.type.CCharPointer; +import org.graalvm.word.PointerBase; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class NativeLibEntryPointContractTest { + + @Test + void allExportsDeclareExplicitExceptionHandlers() throws NoSuchMethodException { + assertEquals( + CEntryPointExceptionHandlers.ReturnVoid.class, + annotation("freeCString", IsolateThread.class, CCharPointer.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnZero.class, + annotation("createEngine", IsolateThread.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnZero.class, + annotation("createEngineWithResolver", IsolateThread.class, + NativeCallbacks.ResolveModuleCallback.class, PointerBase.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnVoid.class, + annotation("destroyEngine", IsolateThread.class, long.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnNullPointer.class, + annotation("runScriptEngine", IsolateThread.class, long.class, + CCharPointer.class, CCharPointer.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnNullPointer.class, + annotation("runScriptCallbackEngine", IsolateThread.class, long.class, + CCharPointer.class, CCharPointer.class, + NativeCallbacks.WriteCallback.class, PointerBase.class).exceptionHandler()); + assertEquals( + CEntryPointExceptionHandlers.ReturnNullPointer.class, + annotation("runScriptInputOutputCallbackEngine", IsolateThread.class, long.class, + CCharPointer.class, CCharPointer.class, CCharPointer.class, + CCharPointer.class, CCharPointer.class, NativeCallbacks.ReadCallback.class, + NativeCallbacks.WriteCallback.class, PointerBase.class).exceptionHandler()); + } + + private static CEntryPoint annotation(String methodName, Class... parameterTypes) + throws NoSuchMethodException { + return NativeLib.class.getDeclaredMethod(methodName, parameterTypes) + .getAnnotation(CEntryPoint.class); + } +} From 09071f8b3e151acb3f01e77f47c618e98bb334e1 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 11:29:04 -0300 Subject: [PATCH 05/48] test(native-lib): cover C ABI failure and lease contracts --- .../tests/integration/test_lifecycle.py | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) diff --git a/native-lib/python/tests/integration/test_lifecycle.py b/native-lib/python/tests/integration/test_lifecycle.py index cfabda1a..9f9d6cb0 100644 --- a/native-lib/python/tests/integration/test_lifecycle.py +++ b/native-lib/python/tests/integration/test_lifecycle.py @@ -1,8 +1,39 @@ +import json +import os +from pathlib import Path +import subprocess +import sys +import textwrap + import pytest import dataweave +def _run_raw_abi_child(code): + source_dir = Path(__file__).resolve().parents[2] / "src" + environment = os.environ.copy() + environment["DATAWEAVE_NATIVE_LIB"] = os.environ["DATAWEAVE_NATIVE_LIB"] + environment["PYTHONPATH"] = ( + str(source_dir) + os.pathsep + environment.get("PYTHONPATH", "") + ) + + return subprocess.run( + [sys.executable, "-c", textwrap.dedent(code)], + capture_output=True, + check=False, + env=environment, + text=True, + timeout=30, + ) + + +def _raw_abi_response(completed): + assert completed.returncode == 0, completed.stderr + assert "Fatal error" not in completed.stderr + return json.loads(completed.stdout) + + @pytest.mark.integration def test_context_manager_runs_multiple_scripts(): with dataweave.DataWeave() as dw: @@ -28,3 +59,228 @@ def test_context_exit_surfaces_cleanup_failure_without_body_exception(monkeypatc with pytest.raises(dataweave.DataWeaveError, match="cleanup failed"): runtime.__exit__(None, None, None) + + +@pytest.mark.integration +def test_raw_abi_contains_null_arguments_without_terminating_process(): + completed = _run_raw_abi_child( + """ + import ctypes + import json + import os + + from dataweave.models import RESOLVE_MODULE_CALLBACK + from dataweave.native import ( + GraalIsolatePointer, + GraalIsolateThreadPointer, + _bind_abi, + ) + + lib = ctypes.CDLL(os.environ["DATAWEAVE_NATIVE_LIB"]) + _bind_abi(lib) + isolate = GraalIsolatePointer() + bootstrap = GraalIsolateThreadPointer() + assert lib.graal_create_isolate( + None, ctypes.byref(isolate), ctypes.byref(bootstrap) + ) == 0 + assert lib.graal_detach_thread(bootstrap) == 0 + + thread = GraalIsolateThreadPointer() + attached = False + null_resolver_handle = 0 + healthy_handle = 0 + null_script_result = None + try: + assert lib.graal_attach_thread(isolate, ctypes.byref(thread)) == 0 + attached = True + null_resolver = ctypes.cast(None, RESOLVE_MODULE_CALLBACK) + null_resolver_handle = lib.create_engine_with_resolver( + thread, null_resolver, None + ) + assert null_resolver_handle == 0 + healthy_handle = lib.create_engine(thread) + assert healthy_handle > 0 + result_pointer = lib.run_script_engine( + thread, healthy_handle, None, None + ) + if result_pointer: + try: + null_script_result = json.loads( + ctypes.string_at(result_pointer).decode("utf-8") + ) + finally: + lib.free_cstring(thread, result_pointer) + finally: + if attached: + if healthy_handle > 0: + lib.destroy_engine(thread, healthy_handle) + if null_resolver_handle > 0: + lib.destroy_engine(thread, null_resolver_handle) + assert lib.graal_detach_thread(thread) == 0 + + teardown_thread = GraalIsolateThreadPointer() + assert lib.graal_attach_thread( + isolate, ctypes.byref(teardown_thread) + ) == 0 + assert lib.graal_tear_down_isolate(teardown_thread) == 0 + print(json.dumps({ + "null_resolver_handle": null_resolver_handle, + "healthy_handle": healthy_handle, + "null_script_result": null_script_result, + })) + """ + ) + + response = _raw_abi_response(completed) + assert response["null_resolver_handle"] == 0 + assert response["healthy_handle"] > 0 + assert response["null_script_result"] == { + "success": False, + "error": "Script cannot be null", + } + + +@pytest.mark.integration +def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): + completed = _run_raw_abi_child( + """ + import ctypes + import json + import os + from threading import Event, Thread + + from dataweave.models import RESOLVE_MODULE_CALLBACK + from dataweave.native import ( + GraalIsolatePointer, + GraalIsolateThreadPointer, + _bind_abi, + ) + + lib = ctypes.CDLL(os.environ["DATAWEAVE_NATIVE_LIB"]) + _bind_abi(lib) + isolate = GraalIsolatePointer() + bootstrap = GraalIsolateThreadPointer() + assert lib.graal_create_isolate( + None, ctypes.byref(isolate), ctypes.byref(bootstrap) + ) == 0 + assert lib.graal_detach_thread(bootstrap) == 0 + + resolver_entered = Event() + release_resolver = Event() + destroy_returned = Event() + errors = [] + run_result = None + context_matched = False + module_source = ctypes.create_string_buffer( + b"%dw 2.0\\nfun answer() = 42" + ) + module_source_address = ctypes.addressof(module_source) + resolver_context = ctypes.c_int(157) + resolver_context_address = ctypes.addressof(resolver_context) + + @RESOLVE_MODULE_CALLBACK + def resolver(_thread, ctx, _path): + global context_matched + context_matched = ctx == resolver_context_address + resolver_entered.set() + if not release_resolver.wait(5): + return 0 + return module_source_address + + creator_thread = GraalIsolateThreadPointer() + assert lib.graal_attach_thread( + isolate, ctypes.byref(creator_thread) + ) == 0 + handle = lib.create_engine_with_resolver( + creator_thread, + resolver, + ctypes.cast(ctypes.pointer(resolver_context), ctypes.c_void_p), + ) + assert handle > 0 + assert lib.graal_detach_thread(creator_thread) == 0 + + def run_script(): + global run_result + thread = GraalIsolateThreadPointer() + attached = False + try: + assert lib.graal_attach_thread( + isolate, ctypes.byref(thread) + ) == 0 + attached = True + result_pointer = lib.run_script_engine( + thread, + handle, + b"%dw 2.0\\n" + b"import org::test::lib\\n" + b"output application/json\\n" + b"---\\n" + b"lib::answer()", + None, + ) + assert result_pointer + try: + run_result = json.loads( + ctypes.string_at(result_pointer).decode("utf-8") + ) + finally: + lib.free_cstring(thread, result_pointer) + except BaseException as error: + errors.append("run: " + repr(error)) + finally: + if attached and lib.graal_detach_thread(thread) != 0: + errors.append("run: failed to detach") + + def destroy_engine(): + thread = GraalIsolateThreadPointer() + attached = False + try: + assert lib.graal_attach_thread( + isolate, ctypes.byref(thread) + ) == 0 + attached = True + lib.destroy_engine(thread, handle) + destroy_returned.set() + except BaseException as error: + errors.append("destroy: " + repr(error)) + finally: + if attached and lib.graal_detach_thread(thread) != 0: + errors.append("destroy: failed to detach") + + run_thread = Thread(target=run_script, daemon=True) + destroy_thread = Thread(target=destroy_engine, daemon=True) + run_thread.start() + try: + assert resolver_entered.wait(5) + destroy_thread.start() + destroy_blocked_before_release = not destroy_returned.wait(0.1) + finally: + release_resolver.set() + + run_thread.join(5) + destroy_thread.join(5) + assert not run_thread.is_alive() + assert not destroy_thread.is_alive() + assert destroy_returned.is_set() + + teardown_thread = GraalIsolateThreadPointer() + assert lib.graal_attach_thread( + isolate, ctypes.byref(teardown_thread) + ) == 0 + assert lib.graal_tear_down_isolate(teardown_thread) == 0 + print(json.dumps({ + "destroy_blocked_before_release": destroy_blocked_before_release, + "destroy_returned": destroy_returned.is_set(), + "context_matched": context_matched, + "run_result": run_result, + "errors": errors, + })) + """ + ) + + response = _raw_abi_response(completed) + assert response["destroy_blocked_before_release"] is True + assert response["destroy_returned"] is True + assert response["context_matched"] is True + assert response["run_result"]["success"] is True + assert response["errors"] == [] From 7e9cf337ea538b92983ba901f59410fad858baed Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 11:48:14 -0300 Subject: [PATCH 06/48] test(native-lib): enforce destroy lease ordering --- .../tests/integration/test_lifecycle.py | 27 ++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/native-lib/python/tests/integration/test_lifecycle.py b/native-lib/python/tests/integration/test_lifecycle.py index 9f9d6cb0..b8f4aad2 100644 --- a/native-lib/python/tests/integration/test_lifecycle.py +++ b/native-lib/python/tests/integration/test_lifecycle.py @@ -147,7 +147,7 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): import ctypes import json import os - from threading import Event, Thread + from threading import Event, Lock, Thread from dataweave.models import RESOLVE_MODULE_CALLBACK from dataweave.native import ( @@ -167,7 +167,13 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): resolver_entered = Event() release_resolver = Event() + destroy_ready = Event() + enter_destroy_call = Event() + destroy_call_started = Event() + run_returned = Event() destroy_returned = Event() + completion_order = [] + completion_lock = Lock() errors = [] run_result = None context_matched = False @@ -218,6 +224,9 @@ def run_script(): b"lib::answer()", None, ) + with completion_lock: + completion_order.append("run_script_engine") + run_returned.set() assert result_pointer try: run_result = json.loads( @@ -239,7 +248,12 @@ def destroy_engine(): isolate, ctypes.byref(thread) ) == 0 attached = True + destroy_ready.set() + assert enter_destroy_call.wait(5) + destroy_call_started.set() lib.destroy_engine(thread, handle) + with completion_lock: + completion_order.append("destroy_engine") destroy_returned.set() except BaseException as error: errors.append("destroy: " + repr(error)) @@ -253,6 +267,9 @@ def destroy_engine(): try: assert resolver_entered.wait(5) destroy_thread.start() + assert destroy_ready.wait(5) + enter_destroy_call.set() + assert destroy_call_started.wait(5) destroy_blocked_before_release = not destroy_returned.wait(0.1) finally: release_resolver.set() @@ -261,6 +278,7 @@ def destroy_engine(): destroy_thread.join(5) assert not run_thread.is_alive() assert not destroy_thread.is_alive() + assert run_returned.is_set() assert destroy_returned.is_set() teardown_thread = GraalIsolateThreadPointer() @@ -270,7 +288,9 @@ def destroy_engine(): assert lib.graal_tear_down_isolate(teardown_thread) == 0 print(json.dumps({ "destroy_blocked_before_release": destroy_blocked_before_release, + "destroy_call_started": destroy_call_started.is_set(), "destroy_returned": destroy_returned.is_set(), + "completion_order": completion_order, "context_matched": context_matched, "run_result": run_result, "errors": errors, @@ -279,6 +299,11 @@ def destroy_engine(): ) response = _raw_abi_response(completed) + assert response["destroy_call_started"] is True + assert response["completion_order"] == [ + "run_script_engine", + "destroy_engine", + ], response assert response["destroy_blocked_before_release"] is True assert response["destroy_returned"] is True assert response["context_matched"] is True From 17f18a5dd3424ec55d4aa1fd0d591128b7c4d1c9 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 12:08:21 -0300 Subject: [PATCH 07/48] test(native-lib): observe closed engine admission --- .../tests/integration/test_lifecycle.py | 89 ++++++++++++++----- 1 file changed, 65 insertions(+), 24 deletions(-) diff --git a/native-lib/python/tests/integration/test_lifecycle.py b/native-lib/python/tests/integration/test_lifecycle.py index b8f4aad2..9759aa46 100644 --- a/native-lib/python/tests/integration/test_lifecycle.py +++ b/native-lib/python/tests/integration/test_lifecycle.py @@ -147,7 +147,8 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): import ctypes import json import os - from threading import Event, Lock, Thread + from threading import Event, Thread + import time from dataweave.models import RESOLVE_MODULE_CALLBACK from dataweave.native import ( @@ -168,14 +169,12 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): resolver_entered = Event() release_resolver = Event() destroy_ready = Event() - enter_destroy_call = Event() - destroy_call_started = Event() + admission_closed = Event() run_returned = Event() destroy_returned = Event() - completion_order = [] - completion_lock = Lock() errors = [] run_result = None + probe_result = None context_matched = False module_source = ctypes.create_string_buffer( b"%dw 2.0\\nfun answer() = 42" @@ -224,8 +223,6 @@ def run_script(): b"lib::answer()", None, ) - with completion_lock: - completion_order.append("run_script_engine") run_returned.set() assert result_pointer try: @@ -249,11 +246,7 @@ def destroy_engine(): ) == 0 attached = True destroy_ready.set() - assert enter_destroy_call.wait(5) - destroy_call_started.set() lib.destroy_engine(thread, handle) - with completion_lock: - completion_order.append("destroy_engine") destroy_returned.set() except BaseException as error: errors.append("destroy: " + repr(error)) @@ -261,23 +254,67 @@ def destroy_engine(): if attached and lib.graal_detach_thread(thread) != 0: errors.append("destroy: failed to detach") + def probe_admission(): + global probe_result + thread = GraalIsolateThreadPointer() + attached = False + try: + assert lib.graal_attach_thread( + isolate, ctypes.byref(thread) + ) == 0 + attached = True + assert destroy_ready.wait(5) + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + result_pointer = lib.run_script_engine( + thread, handle, b"1", None + ) + assert result_pointer + try: + result = json.loads( + ctypes.string_at(result_pointer).decode("utf-8") + ) + finally: + lib.free_cstring(thread, result_pointer) + if result == { + "success": False, + "error": "Unknown engine handle", + }: + probe_result = result + admission_closed.set() + return + time.sleep(0.01) + errors.append("probe: admission did not close") + except BaseException as error: + errors.append("probe: " + repr(error)) + finally: + if attached and lib.graal_detach_thread(thread) != 0: + errors.append("probe: failed to detach") + run_thread = Thread(target=run_script, daemon=True) destroy_thread = Thread(target=destroy_engine, daemon=True) + probe_thread = Thread(target=probe_admission, daemon=True) run_thread.start() try: assert resolver_entered.wait(5) destroy_thread.start() - assert destroy_ready.wait(5) - enter_destroy_call.set() - assert destroy_call_started.wait(5) - destroy_blocked_before_release = not destroy_returned.wait(0.1) + probe_thread.start() + admission_closed_while_resolver_blocked = ( + admission_closed.wait(5) and not release_resolver.is_set() + ) + destroy_blocked_with_admission_closed = ( + admission_closed_while_resolver_blocked + and not destroy_returned.wait(0.1) + ) finally: release_resolver.set() run_thread.join(5) destroy_thread.join(5) + probe_thread.join(5) assert not run_thread.is_alive() assert not destroy_thread.is_alive() + assert not probe_thread.is_alive() assert run_returned.is_set() assert destroy_returned.is_set() @@ -287,11 +324,15 @@ def destroy_engine(): ) == 0 assert lib.graal_tear_down_isolate(teardown_thread) == 0 print(json.dumps({ - "destroy_blocked_before_release": destroy_blocked_before_release, - "destroy_call_started": destroy_call_started.is_set(), + "admission_closed_while_resolver_blocked": ( + admission_closed_while_resolver_blocked + ), + "destroy_blocked_with_admission_closed": ( + destroy_blocked_with_admission_closed + ), "destroy_returned": destroy_returned.is_set(), - "completion_order": completion_order, "context_matched": context_matched, + "probe_result": probe_result, "run_result": run_result, "errors": errors, })) @@ -299,13 +340,13 @@ def destroy_engine(): ) response = _raw_abi_response(completed) - assert response["destroy_call_started"] is True - assert response["completion_order"] == [ - "run_script_engine", - "destroy_engine", - ], response - assert response["destroy_blocked_before_release"] is True + assert response["admission_closed_while_resolver_blocked"] is True + assert response["destroy_blocked_with_admission_closed"] is True assert response["destroy_returned"] is True assert response["context_matched"] is True + assert response["probe_result"] == { + "success": False, + "error": "Unknown engine handle", + } assert response["run_result"]["success"] is True assert response["errors"] == [] From 9e0ec26063327a77939b8a024439852265a2378f Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 13:08:12 -0300 Subject: [PATCH 08/48] test(native-lib): observe native destroy return --- .../tests/integration/test_lifecycle.py | 206 ++++++++++++++---- 1 file changed, 160 insertions(+), 46 deletions(-) diff --git a/native-lib/python/tests/integration/test_lifecycle.py b/native-lib/python/tests/integration/test_lifecycle.py index 9759aa46..501309e5 100644 --- a/native-lib/python/tests/integration/test_lifecycle.py +++ b/native-lib/python/tests/integration/test_lifecycle.py @@ -10,13 +10,14 @@ import dataweave -def _run_raw_abi_child(code): +def _run_raw_abi_child(code, extra_environment=None): source_dir = Path(__file__).resolve().parents[2] / "src" environment = os.environ.copy() environment["DATAWEAVE_NATIVE_LIB"] = os.environ["DATAWEAVE_NATIVE_LIB"] environment["PYTHONPATH"] = ( str(source_dir) + os.pathsep + environment.get("PYTHONPATH", "") ) + environment.update(extra_environment or {}) return subprocess.run( [sys.executable, "-c", textwrap.dedent(code)], @@ -28,6 +29,100 @@ def _run_raw_abi_child(code): ) +def _build_destroy_observer(directory): + source = directory / "destroy_observer.c" + source.write_text( + textwrap.dedent( + """ + #ifdef _WIN32 + #include + #define EXPORT __declspec(dllexport) + static volatile LONG destroy_returned; + static volatile LONG resolver_released; + #define STORE_VALUE(value, new_value) \\ + InterlockedExchange(&(value), (new_value)) + #define LOAD_VALUE(value) InterlockedCompareExchange(&(value), 0, 0) + #else + #define EXPORT __attribute__((visibility("default"))) + static int destroy_returned; + static int resolver_released; + #define STORE_VALUE(value, new_value) \\ + __atomic_store_n(&(value), (new_value), __ATOMIC_RELEASE) + #define LOAD_VALUE(value) __atomic_load_n(&(value), __ATOMIC_ACQUIRE) + #endif + + typedef void (*destroy_engine_fn)(void *, long long); + + EXPORT void reset_destroy_returned(void) { + STORE_VALUE(destroy_returned, 0); + STORE_VALUE(resolver_released, 0); + } + + EXPORT int has_destroy_returned(void) { + return LOAD_VALUE(destroy_returned); + } + + EXPORT void mark_resolver_released(void) { + STORE_VALUE(resolver_released, 1); + } + + EXPORT int observe_destroy_return( + destroy_engine_fn destroy_engine, void *thread, long long handle) { + int released_at_return; + destroy_engine(thread, handle); + released_at_return = LOAD_VALUE(resolver_released); + STORE_VALUE(destroy_returned, 1); + return released_at_return; + } + """ + ), + encoding="ascii", + ) + + if os.name == "nt": + library = directory / "destroy_observer.dll" + command = [ + os.environ.get("CC", "cl"), + "/nologo", + "/LD", + "/O2", + str(source), + f"/Fe:{library}", + ] + elif sys.platform == "darwin": + library = directory / "libdestroy_observer.dylib" + command = [ + os.environ.get("CC", "cc"), + "-dynamiclib", + "-O2", + str(source), + "-o", + str(library), + ] + else: + library = directory / "libdestroy_observer.so" + command = [ + os.environ.get("CC", "cc"), + "-shared", + "-fPIC", + "-O2", + str(source), + "-o", + str(library), + ] + + completed = subprocess.run( + command, + capture_output=True, + check=False, + cwd=directory, + text=True, + timeout=30, + ) + assert completed.returncode == 0, completed.stdout + completed.stderr + return library + + def _raw_abi_response(completed): assert completed.returncode == 0, completed.stderr assert "Fatal error" not in completed.stderr @@ -141,7 +236,8 @@ def test_raw_abi_contains_null_arguments_without_terminating_process(): @pytest.mark.integration -def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): +def test_raw_abi_destroy_waits_for_resolver_context_to_drain(tmp_path): + destroy_observer = _build_destroy_observer(tmp_path) completed = _run_raw_abi_child( """ import ctypes @@ -159,6 +255,19 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): lib = ctypes.CDLL(os.environ["DATAWEAVE_NATIVE_LIB"]) _bind_abi(lib) + observer = ctypes.CDLL(os.environ["DATAWEAVE_DESTROY_OBSERVER"]) + observer.reset_destroy_returned.argtypes = [] + observer.reset_destroy_returned.restype = None + observer.has_destroy_returned.argtypes = [] + observer.has_destroy_returned.restype = ctypes.c_int + observer.mark_resolver_released.argtypes = [] + observer.mark_resolver_released.restype = None + observer.observe_destroy_return.argtypes = [ + ctypes.c_void_p, + GraalIsolateThreadPointer, + ctypes.c_int64, + ] + observer.observe_destroy_return.restype = ctypes.c_int isolate = GraalIsolatePointer() bootstrap = GraalIsolateThreadPointer() assert lib.graal_create_isolate( @@ -168,13 +277,14 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): resolver_entered = Event() release_resolver = Event() - destroy_ready = Event() - admission_closed = Event() + probe_ready = Event() + start_probe = Event() run_returned = Event() - destroy_returned = Event() errors = [] run_result = None probe_result = None + resolver_released_at_destroy_return = None + destroy_returned_before_release = None context_matched = False module_source = ctypes.create_string_buffer( b"%dw 2.0\\nfun answer() = 42" @@ -237,25 +347,8 @@ def run_script(): if attached and lib.graal_detach_thread(thread) != 0: errors.append("run: failed to detach") - def destroy_engine(): - thread = GraalIsolateThreadPointer() - attached = False - try: - assert lib.graal_attach_thread( - isolate, ctypes.byref(thread) - ) == 0 - attached = True - destroy_ready.set() - lib.destroy_engine(thread, handle) - destroy_returned.set() - except BaseException as error: - errors.append("destroy: " + repr(error)) - finally: - if attached and lib.graal_detach_thread(thread) != 0: - errors.append("destroy: failed to detach") - def probe_admission(): - global probe_result + global destroy_returned_before_release, probe_result thread = GraalIsolateThreadPointer() attached = False try: @@ -263,7 +356,8 @@ def probe_admission(): isolate, ctypes.byref(thread) ) == 0 attached = True - assert destroy_ready.wait(5) + probe_ready.set() + assert start_probe.wait(5) deadline = time.monotonic() + 5 while time.monotonic() < deadline: result_pointer = lib.run_script_engine( @@ -281,42 +375,61 @@ def probe_admission(): "error": "Unknown engine handle", }: probe_result = result - admission_closed.set() + return_deadline = time.monotonic() + 1 + while time.monotonic() < return_deadline: + if observer.has_destroy_returned(): + destroy_returned_before_release = True + break + time.sleep(0.001) + else: + destroy_returned_before_release = False + observer.mark_resolver_released() + release_resolver.set() return time.sleep(0.01) errors.append("probe: admission did not close") except BaseException as error: errors.append("probe: " + repr(error)) finally: + release_resolver.set() if attached and lib.graal_detach_thread(thread) != 0: errors.append("probe: failed to detach") run_thread = Thread(target=run_script, daemon=True) - destroy_thread = Thread(target=destroy_engine, daemon=True) probe_thread = Thread(target=probe_admission, daemon=True) run_thread.start() + controller_thread = GraalIsolateThreadPointer() + controller_attached = False try: assert resolver_entered.wait(5) - destroy_thread.start() probe_thread.start() - admission_closed_while_resolver_blocked = ( - admission_closed.wait(5) and not release_resolver.is_set() - ) - destroy_blocked_with_admission_closed = ( - admission_closed_while_resolver_blocked - and not destroy_returned.wait(0.1) + assert probe_ready.wait(5) + assert lib.graal_attach_thread( + isolate, ctypes.byref(controller_thread) + ) == 0 + controller_attached = True + observer.reset_destroy_returned() + start_probe.set() + resolver_released_at_destroy_return = bool( + observer.observe_destroy_return( + ctypes.cast(lib.destroy_engine, ctypes.c_void_p), + controller_thread, + handle, + ) ) finally: release_resolver.set() + if ( + controller_attached + and lib.graal_detach_thread(controller_thread) != 0 + ): + errors.append("controller: failed to detach") run_thread.join(5) - destroy_thread.join(5) probe_thread.join(5) assert not run_thread.is_alive() - assert not destroy_thread.is_alive() assert not probe_thread.is_alive() assert run_returned.is_set() - assert destroy_returned.is_set() teardown_thread = GraalIsolateThreadPointer() assert lib.graal_attach_thread( @@ -324,26 +437,27 @@ def probe_admission(): ) == 0 assert lib.graal_tear_down_isolate(teardown_thread) == 0 print(json.dumps({ - "admission_closed_while_resolver_blocked": ( - admission_closed_while_resolver_blocked + "context_matched": context_matched, + "destroy_returned_before_release": destroy_returned_before_release, + "native_destroy_return_observed": bool( + observer.has_destroy_returned() ), - "destroy_blocked_with_admission_closed": ( - destroy_blocked_with_admission_closed + "resolver_released_at_destroy_return": ( + resolver_released_at_destroy_return ), - "destroy_returned": destroy_returned.is_set(), - "context_matched": context_matched, "probe_result": probe_result, "run_result": run_result, "errors": errors, })) - """ + """, + {"DATAWEAVE_DESTROY_OBSERVER": str(destroy_observer)}, ) response = _raw_abi_response(completed) - assert response["admission_closed_while_resolver_blocked"] is True - assert response["destroy_blocked_with_admission_closed"] is True - assert response["destroy_returned"] is True assert response["context_matched"] is True + assert response["destroy_returned_before_release"] is False + assert response["native_destroy_return_observed"] is True + assert response["resolver_released_at_destroy_return"] is True assert response["probe_result"] == { "success": False, "error": "Unknown engine handle", From 78c3a44d42b42ce38f61aa1cfc02f11bcaff328c Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 14:12:47 -0300 Subject: [PATCH 09/48] test(native-lib): restore bounded destroy drain check --- .../tests/integration/test_lifecycle.py | 230 +++--------------- 1 file changed, 30 insertions(+), 200 deletions(-) diff --git a/native-lib/python/tests/integration/test_lifecycle.py b/native-lib/python/tests/integration/test_lifecycle.py index 501309e5..bb31948e 100644 --- a/native-lib/python/tests/integration/test_lifecycle.py +++ b/native-lib/python/tests/integration/test_lifecycle.py @@ -10,15 +10,13 @@ import dataweave -def _run_raw_abi_child(code, extra_environment=None): +def _run_raw_abi_child(code): source_dir = Path(__file__).resolve().parents[2] / "src" environment = os.environ.copy() environment["DATAWEAVE_NATIVE_LIB"] = os.environ["DATAWEAVE_NATIVE_LIB"] environment["PYTHONPATH"] = ( str(source_dir) + os.pathsep + environment.get("PYTHONPATH", "") ) - environment.update(extra_environment or {}) - return subprocess.run( [sys.executable, "-c", textwrap.dedent(code)], capture_output=True, @@ -29,100 +27,6 @@ def _run_raw_abi_child(code, extra_environment=None): ) -def _build_destroy_observer(directory): - source = directory / "destroy_observer.c" - source.write_text( - textwrap.dedent( - """ - #ifdef _WIN32 - #include - #define EXPORT __declspec(dllexport) - static volatile LONG destroy_returned; - static volatile LONG resolver_released; - #define STORE_VALUE(value, new_value) \\ - InterlockedExchange(&(value), (new_value)) - #define LOAD_VALUE(value) InterlockedCompareExchange(&(value), 0, 0) - #else - #define EXPORT __attribute__((visibility("default"))) - static int destroy_returned; - static int resolver_released; - #define STORE_VALUE(value, new_value) \\ - __atomic_store_n(&(value), (new_value), __ATOMIC_RELEASE) - #define LOAD_VALUE(value) __atomic_load_n(&(value), __ATOMIC_ACQUIRE) - #endif - - typedef void (*destroy_engine_fn)(void *, long long); - - EXPORT void reset_destroy_returned(void) { - STORE_VALUE(destroy_returned, 0); - STORE_VALUE(resolver_released, 0); - } - - EXPORT int has_destroy_returned(void) { - return LOAD_VALUE(destroy_returned); - } - - EXPORT void mark_resolver_released(void) { - STORE_VALUE(resolver_released, 1); - } - - EXPORT int observe_destroy_return( - destroy_engine_fn destroy_engine, void *thread, long long handle) { - int released_at_return; - destroy_engine(thread, handle); - released_at_return = LOAD_VALUE(resolver_released); - STORE_VALUE(destroy_returned, 1); - return released_at_return; - } - """ - ), - encoding="ascii", - ) - - if os.name == "nt": - library = directory / "destroy_observer.dll" - command = [ - os.environ.get("CC", "cl"), - "/nologo", - "/LD", - "/O2", - str(source), - f"/Fe:{library}", - ] - elif sys.platform == "darwin": - library = directory / "libdestroy_observer.dylib" - command = [ - os.environ.get("CC", "cc"), - "-dynamiclib", - "-O2", - str(source), - "-o", - str(library), - ] - else: - library = directory / "libdestroy_observer.so" - command = [ - os.environ.get("CC", "cc"), - "-shared", - "-fPIC", - "-O2", - str(source), - "-o", - str(library), - ] - - completed = subprocess.run( - command, - capture_output=True, - check=False, - cwd=directory, - text=True, - timeout=30, - ) - assert completed.returncode == 0, completed.stdout + completed.stderr - return library - - def _raw_abi_response(completed): assert completed.returncode == 0, completed.stderr assert "Fatal error" not in completed.stderr @@ -236,15 +140,13 @@ def test_raw_abi_contains_null_arguments_without_terminating_process(): @pytest.mark.integration -def test_raw_abi_destroy_waits_for_resolver_context_to_drain(tmp_path): - destroy_observer = _build_destroy_observer(tmp_path) +def test_raw_abi_destroy_waits_for_resolver_context_to_drain(): completed = _run_raw_abi_child( """ import ctypes import json import os from threading import Event, Thread - import time from dataweave.models import RESOLVE_MODULE_CALLBACK from dataweave.native import ( @@ -255,19 +157,6 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(tmp_path): lib = ctypes.CDLL(os.environ["DATAWEAVE_NATIVE_LIB"]) _bind_abi(lib) - observer = ctypes.CDLL(os.environ["DATAWEAVE_DESTROY_OBSERVER"]) - observer.reset_destroy_returned.argtypes = [] - observer.reset_destroy_returned.restype = None - observer.has_destroy_returned.argtypes = [] - observer.has_destroy_returned.restype = ctypes.c_int - observer.mark_resolver_released.argtypes = [] - observer.mark_resolver_released.restype = None - observer.observe_destroy_return.argtypes = [ - ctypes.c_void_p, - GraalIsolateThreadPointer, - ctypes.c_int64, - ] - observer.observe_destroy_return.restype = ctypes.c_int isolate = GraalIsolatePointer() bootstrap = GraalIsolateThreadPointer() assert lib.graal_create_isolate( @@ -277,14 +166,12 @@ def test_raw_abi_destroy_waits_for_resolver_context_to_drain(tmp_path): resolver_entered = Event() release_resolver = Event() - probe_ready = Event() - start_probe = Event() - run_returned = Event() + destroy_ready = Event() + start_destroy = Event() + destroy_call_started = Event() + destroy_returned = Event() errors = [] run_result = None - probe_result = None - resolver_released_at_destroy_return = None - destroy_returned_before_release = None context_matched = False module_source = ctypes.create_string_buffer( b"%dw 2.0\\nfun answer() = 42" @@ -333,7 +220,6 @@ def run_script(): b"lib::answer()", None, ) - run_returned.set() assert result_pointer try: run_result = json.loads( @@ -347,8 +233,7 @@ def run_script(): if attached and lib.graal_detach_thread(thread) != 0: errors.append("run: failed to detach") - def probe_admission(): - global destroy_returned_before_release, probe_result + def destroy_engine(): thread = GraalIsolateThreadPointer() attached = False try: @@ -356,80 +241,35 @@ def probe_admission(): isolate, ctypes.byref(thread) ) == 0 attached = True - probe_ready.set() - assert start_probe.wait(5) - deadline = time.monotonic() + 5 - while time.monotonic() < deadline: - result_pointer = lib.run_script_engine( - thread, handle, b"1", None - ) - assert result_pointer - try: - result = json.loads( - ctypes.string_at(result_pointer).decode("utf-8") - ) - finally: - lib.free_cstring(thread, result_pointer) - if result == { - "success": False, - "error": "Unknown engine handle", - }: - probe_result = result - return_deadline = time.monotonic() + 1 - while time.monotonic() < return_deadline: - if observer.has_destroy_returned(): - destroy_returned_before_release = True - break - time.sleep(0.001) - else: - destroy_returned_before_release = False - observer.mark_resolver_released() - release_resolver.set() - return - time.sleep(0.01) - errors.append("probe: admission did not close") + destroy_ready.set() + assert start_destroy.wait(5) + destroy_call_started.set() + lib.destroy_engine(thread, handle) + destroy_returned.set() except BaseException as error: - errors.append("probe: " + repr(error)) + errors.append("destroy: " + repr(error)) finally: - release_resolver.set() if attached and lib.graal_detach_thread(thread) != 0: - errors.append("probe: failed to detach") + errors.append("destroy: failed to detach") run_thread = Thread(target=run_script, daemon=True) - probe_thread = Thread(target=probe_admission, daemon=True) + destroy_thread = Thread(target=destroy_engine, daemon=True) run_thread.start() - controller_thread = GraalIsolateThreadPointer() - controller_attached = False try: assert resolver_entered.wait(5) - probe_thread.start() - assert probe_ready.wait(5) - assert lib.graal_attach_thread( - isolate, ctypes.byref(controller_thread) - ) == 0 - controller_attached = True - observer.reset_destroy_returned() - start_probe.set() - resolver_released_at_destroy_return = bool( - observer.observe_destroy_return( - ctypes.cast(lib.destroy_engine, ctypes.c_void_p), - controller_thread, - handle, - ) - ) + destroy_thread.start() + assert destroy_ready.wait(5) + start_destroy.set() + assert destroy_call_started.wait(5) + destroy_blocked_before_release = not destroy_returned.wait(0.1) finally: release_resolver.set() - if ( - controller_attached - and lib.graal_detach_thread(controller_thread) != 0 - ): - errors.append("controller: failed to detach") run_thread.join(5) - probe_thread.join(5) + destroy_thread.join(5) assert not run_thread.is_alive() - assert not probe_thread.is_alive() - assert run_returned.is_set() + assert not destroy_thread.is_alive() + assert destroy_returned.is_set() teardown_thread = GraalIsolateThreadPointer() assert lib.graal_attach_thread( @@ -438,29 +278,19 @@ def probe_admission(): assert lib.graal_tear_down_isolate(teardown_thread) == 0 print(json.dumps({ "context_matched": context_matched, - "destroy_returned_before_release": destroy_returned_before_release, - "native_destroy_return_observed": bool( - observer.has_destroy_returned() - ), - "resolver_released_at_destroy_return": ( - resolver_released_at_destroy_return - ), - "probe_result": probe_result, + "destroy_blocked_before_release": destroy_blocked_before_release, + "destroy_call_started": destroy_call_started.is_set(), + "destroy_returned": destroy_returned.is_set(), "run_result": run_result, "errors": errors, })) - """, - {"DATAWEAVE_DESTROY_OBSERVER": str(destroy_observer)}, + """ ) response = _raw_abi_response(completed) assert response["context_matched"] is True - assert response["destroy_returned_before_release"] is False - assert response["native_destroy_return_observed"] is True - assert response["resolver_released_at_destroy_return"] is True - assert response["probe_result"] == { - "success": False, - "error": "Unknown engine handle", - } + assert response["destroy_call_started"] is True + assert response["destroy_blocked_before_release"] is True + assert response["destroy_returned"] is True assert response["run_result"]["success"] is True assert response["errors"] == [] From 5b383b22aaf563850a145f7b25ff294bdb9b7317 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 3 Sep 2026 17:03:58 -0300 Subject: [PATCH 10/48] refactor(native-lib): remove unleased runtime lookup --- .../src/main/java/org/mule/weave/lib/ScriptRuntime.java | 6 ------ 1 file changed, 6 deletions(-) diff --git a/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java b/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java index bc0bbdad..ff703d3c 100644 --- a/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java +++ b/native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java @@ -53,12 +53,6 @@ public static long register(ScriptRuntime runtime) { return handle; } - /** Returns the runtime for a handle, or {@code null} if unknown/destroyed. */ - public static ScriptRuntime get(long handle) { - EngineRecord record = REGISTRY.get(handle); - return record == null ? null : record.runtime; - } - /** Acquires a lease for a live runtime, or returns {@code null} if admission is closed. */ public static EngineLease acquire(long handle) { EngineRecord record = REGISTRY.get(handle); From 2444240a1206fbbe5fc953fb011a980acc527786 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 15:56:08 -0300 Subject: [PATCH 11/48] fix(python): reject native callback reentrancy --- native-lib/python/src/dataweave/native.py | 49 +++++-- native-lib/python/src/dataweave/runtime.py | 31 +++-- .../tests/integration/test_module_resolver.py | 56 ++++++++ native-lib/python/tests/unit/test_native.py | 122 +++++++++++++++++- .../python/tests/unit/test_streaming.py | 88 ++++++------- 5 files changed, 284 insertions(+), 62 deletions(-) diff --git a/native-lib/python/src/dataweave/native.py b/native-lib/python/src/dataweave/native.py index afacd88d..b958da10 100644 --- a/native-lib/python/src/dataweave/native.py +++ b/native-lib/python/src/dataweave/native.py @@ -3,7 +3,7 @@ import os from pathlib import Path import sys -from threading import get_ident, Lock +from threading import get_ident, local, Lock import traceback from typing import Optional @@ -12,6 +12,27 @@ _ENV_NATIVE_LIB = "DATAWEAVE_NATIVE_LIB" +_native_callback_state = local() + + +def _raise_if_native_callback_active() -> None: + if getattr(_native_callback_state, "depth", 0) > 0: + raise DataWeaveError( + "DataWeave lifecycle and execution are not allowed from a native callback on the same thread." + ) + + +@contextmanager +def _native_callback_scope(): + previous = getattr(_native_callback_state, "depth", 0) + _native_callback_state.depth = previous + 1 + try: + yield + finally: + if previous == 0: + del _native_callback_state.depth + else: + _native_callback_state.depth = previous class graal_isolate_t(ctypes.Structure): @@ -361,7 +382,7 @@ def __init__(self, lib_path: Optional[str] = None): self._resolver_buffers = [] self._resolver_active = False self._resolver_active_ident = None - self._resolver_lock = Lock() + self._operation_lock = Lock() self._execution_owner = None # Guards this instance's initialize()/cleanup() lifecycle transitions # (the initialized-check -> acquire -> create-engine -> publish @@ -373,6 +394,7 @@ def __init__(self, lib_path: Optional[str] = None): self._init_lock = Lock() def initialize(self) -> None: + _raise_if_native_callback_active() if self.initialized: return with self._init_lock: @@ -493,6 +515,7 @@ def run_input_output_callback_engine_and_decode( def install_resolver(self, resolver: ModuleResolver) -> None: """Binds a module resolver to this engine. Must be called before initialize().""" + _raise_if_native_callback_active() if self.initialized: raise DataWeaveError("Cannot install a resolver after initialize().") self._resolver = resolver @@ -517,7 +540,8 @@ def resolve(_thread, _ctx, module_path): path = module_path.decode("utf-8") if path.startswith("/"): path = path[1:] - source = entry._resolver(path) + with _native_callback_scope(): + source = entry._resolver(path) if not isinstance(source, str): return None buffer = ctypes.create_string_buffer(source.encode("utf-8")) @@ -550,10 +574,11 @@ def _resolver_scope(self): self._resolver_buffers = [] def cleanup(self) -> None: + _raise_if_native_callback_active() with self._serialized_native_operation(): - # _init_lock is nested INSIDE _resolver_lock here (never the + # _init_lock is nested INSIDE _operation_lock here (never the # reverse -- initialize() only ever takes _init_lock alone, and - # never takes _resolver_lock), so there is no lock-ordering + # never takes _operation_lock), so there is no lock-ordering # inversion between the two. Only the initialized-clearing flag # flip needs the lock; the actual destroy/release below stays # outside it, guarded by _serialized_native_operation as before. @@ -587,18 +612,25 @@ def cleanup(self) -> None: @contextmanager def _serialized_native_operation(self): + _raise_if_native_callback_active() owner = get_ident() if getattr(self, "_execution_owner", None) == owner: raise DataWeaveError("Reentrant DataWeave execution is not supported.") - if not hasattr(self, "_resolver_lock"): - self._resolver_lock = Lock() - with self._resolver_lock: + if not hasattr(self, "_operation_lock"): + self._operation_lock = Lock() + with self._operation_lock: self._execution_owner = owner try: yield finally: self._execution_owner = None + def capture_operation(self) -> None: + _raise_if_native_callback_active() + if not self.initialized: + raise DataWeaveError("DataWeave runtime not initialized. Call initialize() first.") + return None + @contextmanager def _current_thread_attachment(self, thread): # A non-None thread is one the caller already attached (a streaming worker @@ -621,4 +653,3 @@ def _current_thread_attachment(self, thread): except Exception: if primary_error is None: raise - diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index 377c60f1..fe9a7f07 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -16,7 +16,7 @@ StreamingResult, WriteCallback, ) -from .native import NativeRuntime +from .native import _native_callback_scope, _raise_if_native_callback_active, NativeRuntime from .resolver import ModuleResolver @@ -42,6 +42,7 @@ def __init__( self._lifecycle_lock = Lock() def initialize(self): + _raise_if_native_callback_active() # Holds the lock across the whole install-resolver + native-init # transition so two concurrent initialize() calls on this instance # cannot both pass the `initialized` guard and both call @@ -58,6 +59,7 @@ def initialize(self): self._native.initialize() def cleanup(self): + _raise_if_native_callback_active() # Symmetric with initialize(): install_resolver() and # NativeRuntime.cleanup() both mutate the module-global resolver # registry, so cleanup() takes the same instance-level lock. @@ -97,6 +99,7 @@ def _unregister_stream_worker(self, worker: Optional[Thread] = None) -> None: workers.discard(worker or current_thread()) def _require_initialized(self, supported: bool, api_name: str) -> None: + _raise_if_native_callback_active() if not self._native.initialized: raise DataWeaveError("DataWeave runtime not initialized. Call initialize() first.") if not supported: @@ -113,8 +116,10 @@ def run(self, script: str, inputs: Optional[Dict[str, Any]] = None, raise_on_err script.encode("utf-8"), self._inputs_json(inputs) ) result = parse_native_encoded_response(raw) + except DataWeaveError: + raise except Exception as error: - raise DataWeaveError(f"Failed to execute script: {error}") + raise DataWeaveError(f"Failed to execute script: {error}") from error if raise_on_error and not result.success: raise DataWeaveScriptError(result) return result @@ -124,14 +129,18 @@ def run_callback(self, script: str, write_callback: WriteCallback, inputs: Optio @WRITE_CALLBACK def write_cb(_context, buffer, length): try: - return write_callback(ctypes.string_at(buffer, length)) + data = ctypes.string_at(buffer, length) + with _native_callback_scope(): + return write_callback(data) except Exception: return -1 try: raw = self._native.run_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) + except DataWeaveError: + raise except Exception as error: - raise DataWeaveError(f"Failed to execute callback streaming: {error}") + raise DataWeaveError(f"Failed to execute callback streaming: {error}") from error def _stream_worker(self, invoke, cancelled: Event) -> Generator[bytes, None, StreamingResult]: sentinel = object() @@ -238,7 +247,8 @@ def read_cb(_context, buffer, buffer_size): return size if state["done"]: return 0 - chunk = next(iterator, None) + with _native_callback_scope(): + chunk = next(iterator, None) if not chunk: state["done"] = True return 0 @@ -265,7 +275,8 @@ def run_input_output_callback(self, script: str, input_name: str, input_mime_typ @READ_CALLBACK def read_cb(_context, buffer, buffer_size): try: - data = read_callback(buffer_size) + with _native_callback_scope(): + data = read_callback(buffer_size) if not data: return 0 if len(data) > buffer_size: @@ -277,14 +288,18 @@ def read_cb(_context, buffer, buffer_size): @WRITE_CALLBACK def write_cb(_context, buffer, length): try: - return write_callback(ctypes.string_at(buffer, length)) + data = ctypes.string_at(buffer, length) + with _native_callback_scope(): + return write_callback(data) except Exception: return -1 try: raw = self._native.run_input_output_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) + except DataWeaveError: + raise except Exception as error: - raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") + raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") from error def __enter__(self): self.initialize() diff --git a/native-lib/python/tests/integration/test_module_resolver.py b/native-lib/python/tests/integration/test_module_resolver.py index fb305b0b..b1b469fc 100644 --- a/native-lib/python/tests/integration/test_module_resolver.py +++ b/native-lib/python/tests/integration/test_module_resolver.py @@ -321,6 +321,62 @@ def run(): } +@pytest.mark.integration +def test_cross_engine_resolver_reentry_is_rejected_without_terminating_process(): + source_dir = Path(__file__).resolve().parents[2] / "src" + code = f""" +import json + +import dataweave + +script = {IMPORT_LIB_SCRIPT!r} +nested = {{}} +inner = dataweave.DataWeave() + +def resolver(_module_path): + try: + inner.run("40 + 2") + except dataweave.DataWeaveError as error: + nested["type"] = type(error).__name__ + nested["error"] = str(error) + return "%dw 2.0\\nfun answer() = 42" + +outer = dataweave.DataWeave(resolve_module=resolver) +inner.initialize() +outer.initialize() +try: + result = outer.run(script) + response = {{ + "nested_type": nested.get("type"), + "nested_error": nested.get("error"), + "outer": {{"success": result.success, "value": result.get_string()}}, + }} +finally: + outer.cleanup() + inner.cleanup() + +print(json.dumps(response)) +""" + environment = os.environ.copy() + environment["PYTHONPATH"] = str(source_dir) + os.pathsep + environment.get("PYTHONPATH", "") + + completed = subprocess.run( + [sys.executable, "-c", code], + capture_output=True, + check=False, + env=environment, + text=True, + timeout=30, + ) + + assert completed.returncode == 0, completed.stderr + response = json.loads(completed.stdout) + assert response["nested_type"] == "DataWeaveError" + assert "native callback" in response["nested_error"].lower() + assert response["outer"] == {"success": True, "value": "42"} + assert "Fatal error" not in completed.stderr + + @pytest.mark.integration def test_shared_isolate_survives_until_the_last_instance_cleans_up(): from dataweave import native diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index 5587b686..f87e227b 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -1,7 +1,7 @@ from pathlib import Path import ctypes import threading -from threading import Barrier, BrokenBarrierError, current_thread, get_ident, Thread +from threading import Barrier, BrokenBarrierError, current_thread, get_ident, Lock, Thread import pytest @@ -381,6 +381,126 @@ def _capture_error(errors, invoke): errors.append(error) +@pytest.mark.unit +def test_native_callback_scope_rejects_lifecycle_and_execution_on_any_same_thread_instance(monkeypatch): + guarded = native.NativeRuntime.__new__(native.NativeRuntime) + guarded.initialized = True + guarded._operation_lock = Lock() + guarded._execution_owner = None + other = native.NativeRuntime.__new__(native.NativeRuntime) + other._init_lock = Lock() + other._operation_lock = Lock() + other._execution_owner = None + monkeypatch.setattr( + native, + "_acquire_isolate", + lambda _path: (_ for _ in ()).throw(AssertionError("native isolate acquired")), + ) + monkeypatch.setattr( + native, + "_release_isolate", + lambda: (_ for _ in ()).throw(AssertionError("native isolate released")), + ) + + with native._native_callback_scope(): + assert native._isolate_lock.acquire(blocking=False) + native._isolate_lock.release() + assert native._resolver_lock_global.acquire(blocking=False) + native._resolver_lock_global.release() + assert guarded._operation_lock.acquire(blocking=False) + guarded._operation_lock.release() + assert other._init_lock.acquire(blocking=False) + other._init_lock.release() + for invoke in ( + guarded.capture_operation, + other.initialize, + guarded.cleanup, + ): + with pytest.raises( + dataweave.DataWeaveError, + match="DataWeave lifecycle and execution are not allowed from a native callback on the same thread\\.", + ): + invoke() + + +@pytest.mark.unit +def test_native_callback_scope_is_thread_local(): + errors = [] + outcomes = [] + runtime = native.NativeRuntime.__new__(native.NativeRuntime) + runtime.initialized = True + + with native._native_callback_scope(): + worker = Thread( + target=lambda: _capture_error( + errors, lambda: outcomes.append(runtime.capture_operation()) + ) + ) + worker.start() + worker.join(1) + + assert not worker.is_alive() + assert errors == [] + assert outcomes == [None] + with pytest.raises(dataweave.DataWeaveError, match="native callback"): + native._raise_if_native_callback_active() + + +@pytest.mark.unit +def test_native_callback_reentry_preserves_the_exact_public_error(): + runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) + runtime._native = native.NativeRuntime.__new__(native.NativeRuntime) + runtime._native.initialized = True + + with native._native_callback_scope(), pytest.raises(dataweave.DataWeaveError) as error: + runtime.run("1 + 1") + + assert str(error.value) == ( + "DataWeave lifecycle and execution are not allowed from a native callback on the same thread." + ) + + +@pytest.mark.unit +@pytest.mark.parametrize("failure_depth", [1, 2]) +def test_native_callback_scope_restores_depth_after_success_error_and_nesting(failure_depth): + assert not hasattr(native._native_callback_state, "depth") + + with pytest.raises(RuntimeError, match="callback failed"): + with native._native_callback_scope(): + assert native._native_callback_state.depth == 1 + if failure_depth == 1: + raise RuntimeError("callback failed") + with native._native_callback_scope(): + assert native._native_callback_state.depth == 2 + raise RuntimeError("callback failed") + + assert not hasattr(native._native_callback_state, "depth") + native._raise_if_native_callback_active() + + with native._native_callback_scope(): + assert native._native_callback_state.depth == 1 + assert not hasattr(native._native_callback_state, "depth") + + +@pytest.mark.unit +def test_resolver_callback_exception_restores_native_callback_depth(monkeypatch): + library = FakeLibrary() + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + runtime.install_resolver( + lambda _path: (_ for _ in ()).throw(RuntimeError("resolver failed")) + ) + runtime.initialize() + _handle, callback, context = library.created_engines[0] + + with runtime._resolver_scope(): + assert callback(None, context, b"org/test/lib.dwl") is None + + assert not hasattr(native._native_callback_state, "depth") + native._raise_if_native_callback_active() + runtime.cleanup() + + @pytest.mark.unit def test_cleanup_from_worker_uses_current_thread_for_isolate_teardown(monkeypatch): library = FakeLibrary() diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index ee00ad78..1afc31bc 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -89,7 +89,7 @@ def configured_runtime(native): native_runtime._resolver_buffers = [] native_runtime._resolver_active = False native_runtime._resolver_active_ident = None - native_runtime._resolver_lock = Lock() + native_runtime._operation_lock = Lock() native_runtime._execution_owner = None runtime._native = native_runtime return runtime @@ -121,57 +121,57 @@ def test_run_input_output_callback_converts_read_exception_to_abort_result(): @pytest.mark.unit def test_write_callback_reentry_is_translated_to_abort_without_deadlocking(): - completed = Event() - outcomes = [] - native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") - runtime = configured_runtime(native) - - worker = Thread( - target=lambda: ( - outcomes.append( - runtime.run_callback( - "outer", - lambda _chunk: runtime.run_callback("nested", lambda _data: 0), - ) - ), - completed.set(), - ), - daemon=True, + outer_native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + inner_native = FakeNative('{"success": true}') + outer = configured_runtime(outer_native) + inner = configured_runtime(inner_native) + + result = outer.run_callback( + "outer", + lambda _chunk: inner.run_callback("nested", lambda _data: 0), ) - worker.start() - assert completed.wait(1), "write callback re-entry deadlocked" - assert native.write_status == -1 - assert outcomes == [dataweave.StreamingResult(False, "write aborted", None, None, False)] + assert outer_native.write_status == -1 + assert inner_native.attach_count == 0 + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) @pytest.mark.unit def test_read_callback_reentry_is_translated_to_abort_without_deadlocking(): - completed = Event() - outcomes = [] - native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) - runtime = configured_runtime(native) - - worker = Thread( - target=lambda: ( - outcomes.append( - runtime.run_input_output_callback( - "outer", - "payload", - "application/json", - lambda _size: runtime.run("nested").get_bytes(), - lambda _data: 0, - ) - ), - completed.set(), - ), - daemon=True, + outer_native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) + inner_native = FakeNative('{"success": true}') + outer = configured_runtime(outer_native) + inner = configured_runtime(inner_native) + + result = outer.run_input_output_callback( + "outer", + "payload", + "application/json", + lambda _size: inner.run("nested").get_bytes(), + lambda _data: 0, ) - worker.start() - assert completed.wait(1), "read callback re-entry deadlocked" - assert native.read_status == -1 - assert outcomes == [dataweave.StreamingResult(False, "read aborted", None, None, False)] + assert outer_native.read_status == -1 + assert inner_native.attach_count == 0 + assert result == dataweave.StreamingResult(False, "read aborted", None, None, False) + + +@pytest.mark.unit +def test_transform_input_iterator_reentry_is_translated_to_abort_without_native_attach(): + outer_native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) + inner_native = FakeNative('{"success": true}') + outer = configured_runtime(outer_native) + inner = configured_runtime(inner_native) + + def input_stream(): + yield inner.run("nested").get_bytes() + + stream = outer.run_transform("outer", input_stream()) + + assert list(stream) == [] + assert outer_native.read_status == -1 + assert inner_native.attach_count == 0 + assert stream.metadata == dataweave.StreamingResult(False, "read aborted", None, None, False) @pytest.mark.unit From 9e9aca39e6d1531af40e0a0d40be67c428421b45 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 16:10:47 -0300 Subject: [PATCH 12/48] fix(python): reject native callback reentrancy --- native-lib/python/src/dataweave/native.py | 69 ++++++++++++------- native-lib/python/tests/unit/test_native.py | 64 ++++++++++++++++- .../python/tests/unit/test_streaming.py | 37 ++++++++-- 3 files changed, 136 insertions(+), 34 deletions(-) diff --git a/native-lib/python/src/dataweave/native.py b/native-lib/python/src/dataweave/native.py index b958da10..09c68788 100644 --- a/native-lib/python/src/dataweave/native.py +++ b/native-lib/python/src/dataweave/native.py @@ -3,7 +3,7 @@ import os from pathlib import Path import sys -from threading import get_ident, local, Lock +from threading import Condition, get_ident, local, Lock import traceback from typing import Optional @@ -382,7 +382,8 @@ def __init__(self, lib_path: Optional[str] = None): self._resolver_buffers = [] self._resolver_active = False self._resolver_active_ident = None - self._operation_lock = Lock() + self._operation_lock = Condition(Lock()) + self._operation_active = False self._execution_owner = None # Guards this instance's initialize()/cleanup() lifecycle transitions # (the initialized-check -> acquire -> create-engine -> publish @@ -446,6 +447,7 @@ def _create_engine(self) -> int: return handle def attach_thread(self): + _raise_if_native_callback_active() worker_thread = GraalIsolateThreadPointer() try: result = self.lib.graal_attach_thread(self.isolate, ctypes.byref(worker_thread)) @@ -456,6 +458,7 @@ def attach_thread(self): return worker_thread def detach_thread(self, thread) -> None: + _raise_if_native_callback_active() try: result = self.lib.graal_detach_thread(thread) except Exception as error: @@ -481,37 +484,49 @@ def decode_and_free(self, ptr, thread=None) -> str: raise def run_engine_and_decode(self, script: bytes, inputs: bytes) -> str: - with self._serialized_native_operation(): + with self._serialized_native_operation() as operation_lock: with self._current_thread_attachment(self.thread) as thread: with self._resolver_scope(): - return self.decode_and_free( - self.lib.run_script_engine(thread, self.handle, script, inputs), - thread, - ) + try: + operation_lock.release() + return self.decode_and_free( + self.lib.run_script_engine(thread, self.handle, script, inputs), + thread, + ) + finally: + operation_lock.acquire() def run_callback_engine_and_decode(self, thread, script: bytes, inputs: bytes, write_callback) -> str: - with self._serialized_native_operation(): + with self._serialized_native_operation() as operation_lock: with self._current_thread_attachment(thread) as current: - return self.decode_and_free( - self.lib.run_script_callback_engine( - current, self.handle, script, inputs, write_callback, None - ), - current, - ) + try: + operation_lock.release() + return self.decode_and_free( + self.lib.run_script_callback_engine( + current, self.handle, script, inputs, write_callback, None + ), + current, + ) + finally: + operation_lock.acquire() def run_input_output_callback_engine_and_decode( self, thread, script: bytes, inputs: bytes, input_name: bytes, input_mime_type: bytes, input_charset: Optional[bytes], read_callback, write_callback, ) -> str: - with self._serialized_native_operation(): + with self._serialized_native_operation() as operation_lock: with self._current_thread_attachment(thread) as current: - return self.decode_and_free( - self.lib.run_script_input_output_callback_engine( - current, self.handle, script, inputs, input_name, - input_mime_type, input_charset, read_callback, write_callback, None, - ), - current, - ) + try: + operation_lock.release() + return self.decode_and_free( + self.lib.run_script_input_output_callback_engine( + current, self.handle, script, inputs, input_name, + input_mime_type, input_charset, read_callback, write_callback, None, + ), + current, + ) + finally: + operation_lock.acquire() def install_resolver(self, resolver: ModuleResolver) -> None: """Binds a module resolver to this engine. Must be called before initialize().""" @@ -617,13 +632,19 @@ def _serialized_native_operation(self): if getattr(self, "_execution_owner", None) == owner: raise DataWeaveError("Reentrant DataWeave execution is not supported.") if not hasattr(self, "_operation_lock"): - self._operation_lock = Lock() + self._operation_lock = Condition(Lock()) + self._operation_active = False with self._operation_lock: + while getattr(self, "_operation_active", False): + self._operation_lock.wait() + self._operation_active = True self._execution_owner = owner try: - yield + yield self._operation_lock finally: self._execution_owner = None + self._operation_active = False + self._operation_lock.notify() def capture_operation(self) -> None: _raise_if_native_callback_active() diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index f87e227b..e649a45c 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -1,7 +1,7 @@ from pathlib import Path import ctypes import threading -from threading import Barrier, BrokenBarrierError, current_thread, get_ident, Lock, Thread +from threading import Barrier, BrokenBarrierError, Condition, current_thread, get_ident, Lock, Thread import pytest @@ -385,11 +385,13 @@ def _capture_error(errors, invoke): def test_native_callback_scope_rejects_lifecycle_and_execution_on_any_same_thread_instance(monkeypatch): guarded = native.NativeRuntime.__new__(native.NativeRuntime) guarded.initialized = True - guarded._operation_lock = Lock() + guarded._operation_lock = Condition(Lock()) + guarded._operation_active = False guarded._execution_owner = None other = native.NativeRuntime.__new__(native.NativeRuntime) other._init_lock = Lock() - other._operation_lock = Lock() + other._operation_lock = Condition(Lock()) + other._operation_active = False other._execution_owner = None monkeypatch.setattr( native, @@ -446,6 +448,29 @@ def test_native_callback_scope_is_thread_local(): native._raise_if_native_callback_active() +@pytest.mark.unit +def test_native_callback_scope_rejects_direct_thread_attachment(): + runtime = native.NativeRuntime.__new__(native.NativeRuntime) + runtime.lib = type( + "Native", + (), + { + "graal_attach_thread": lambda _self, _isolate, _thread: (_ for _ in ()).throw( + AssertionError("native attach called") + ), + "graal_detach_thread": lambda _self, _thread: (_ for _ in ()).throw( + AssertionError("native detach called") + ), + }, + )() + runtime.isolate = object() + + with native._native_callback_scope(): + for invoke in (runtime.attach_thread, lambda: runtime.detach_thread(object())): + with pytest.raises(dataweave.DataWeaveError, match="native callback"): + invoke() + + @pytest.mark.unit def test_native_callback_reentry_preserves_the_exact_public_error(): runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) @@ -501,6 +526,39 @@ def test_resolver_callback_exception_restores_native_callback_depth(monkeypatch) runtime.cleanup() +@pytest.mark.unit +def test_resolver_callback_runs_without_the_instance_operation_lock(monkeypatch): + library = FakeLibrary() + result_buffer = ctypes.create_string_buffer(b"result") + lock_available = [] + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + + def resolver(_path): + acquired = runtime._operation_lock.acquire(blocking=False) + lock_available.append(acquired) + if acquired: + runtime._operation_lock.release() + return "source" + + runtime.install_resolver(resolver) + runtime.initialize() + _handle, callback, context = library.created_engines[0] + + def run_script(_thread, _handle, _script, _inputs): + source = callback(None, context, b"org/test/lib.dwl") + assert ctypes.string_at(source) == b"source" + return ctypes.addressof(result_buffer) + + library.run_script_engine = CallableFunction(run_script) + library.free_cstring = CallableFunction(lambda _thread, _ptr: None) + + assert runtime.run_engine_and_decode(b"script", b"{}") == "result" + assert lock_available == [True] + + runtime.cleanup() + + @pytest.mark.unit def test_cleanup_from_worker_uses_current_thread_for_isolate_teardown(monkeypatch): library = FakeLibrary() diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index 1afc31bc..32affd93 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -1,6 +1,6 @@ import ctypes from queue import Full, Queue -from threading import Event, Lock, Thread +from threading import Condition, Event, Lock, Thread from time import sleep import pytest @@ -89,7 +89,7 @@ def configured_runtime(native): native_runtime._resolver_buffers = [] native_runtime._resolver_active = False native_runtime._resolver_active_ident = None - native_runtime._operation_lock = Lock() + native_runtime._operation_lock = Condition(Lock()) native_runtime._execution_owner = None runtime._native = native_runtime return runtime @@ -126,13 +126,20 @@ def test_write_callback_reentry_is_translated_to_abort_without_deadlocking(): outer = configured_runtime(outer_native) inner = configured_runtime(inner_native) - result = outer.run_callback( - "outer", - lambda _chunk: inner.run_callback("nested", lambda _data: 0), - ) + lock_available = [] + + def reenter(_chunk): + acquired = outer._native._operation_lock.acquire(blocking=False) + lock_available.append(acquired) + if acquired: + outer._native._operation_lock.release() + return inner.run_callback("nested", lambda _data: 0) + + result = outer.run_callback("outer", reenter) assert outer_native.write_status == -1 assert inner_native.attach_count == 0 + assert lock_available == [True] assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) @@ -143,16 +150,26 @@ def test_read_callback_reentry_is_translated_to_abort_without_deadlocking(): outer = configured_runtime(outer_native) inner = configured_runtime(inner_native) + lock_available = [] + + def reenter(_size): + acquired = outer._native._operation_lock.acquire(blocking=False) + lock_available.append(acquired) + if acquired: + outer._native._operation_lock.release() + return inner.run("nested").get_bytes() + result = outer.run_input_output_callback( "outer", "payload", "application/json", - lambda _size: inner.run("nested").get_bytes(), + reenter, lambda _data: 0, ) assert outer_native.read_status == -1 assert inner_native.attach_count == 0 + assert lock_available == [True] assert result == dataweave.StreamingResult(False, "read aborted", None, None, False) @@ -162,8 +179,13 @@ def test_transform_input_iterator_reentry_is_translated_to_abort_without_native_ inner_native = FakeNative('{"success": true}') outer = configured_runtime(outer_native) inner = configured_runtime(inner_native) + lock_available = [] def input_stream(): + acquired = outer._native._operation_lock.acquire(blocking=False) + lock_available.append(acquired) + if acquired: + outer._native._operation_lock.release() yield inner.run("nested").get_bytes() stream = outer.run_transform("outer", input_stream()) @@ -171,6 +193,7 @@ def input_stream(): assert list(stream) == [] assert outer_native.read_status == -1 assert inner_native.attach_count == 0 + assert lock_available == [True] assert stream.metadata == dataweave.StreamingResult(False, "read aborted", None, None, False) From 0aba97269a355de1946b9a8c9aa019a1dd3a464b Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 16:19:04 -0300 Subject: [PATCH 13/48] fix(python): reject native callback reentrancy --- native-lib/python/src/dataweave/runtime.py | 6 ------ native-lib/python/tests/unit/test_native.py | 14 -------------- 2 files changed, 20 deletions(-) diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index fe9a7f07..1f02b853 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -116,8 +116,6 @@ def run(self, script: str, inputs: Optional[Dict[str, Any]] = None, raise_on_err script.encode("utf-8"), self._inputs_json(inputs) ) result = parse_native_encoded_response(raw) - except DataWeaveError: - raise except Exception as error: raise DataWeaveError(f"Failed to execute script: {error}") from error if raise_on_error and not result.success: @@ -137,8 +135,6 @@ def write_cb(_context, buffer, length): try: raw = self._native.run_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) - except DataWeaveError: - raise except Exception as error: raise DataWeaveError(f"Failed to execute callback streaming: {error}") from error @@ -296,8 +292,6 @@ def write_cb(_context, buffer, length): try: raw = self._native.run_input_output_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) - except DataWeaveError: - raise except Exception as error: raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") from error diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index e649a45c..f637df01 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -471,20 +471,6 @@ def test_native_callback_scope_rejects_direct_thread_attachment(): invoke() -@pytest.mark.unit -def test_native_callback_reentry_preserves_the_exact_public_error(): - runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) - runtime._native = native.NativeRuntime.__new__(native.NativeRuntime) - runtime._native.initialized = True - - with native._native_callback_scope(), pytest.raises(dataweave.DataWeaveError) as error: - runtime.run("1 + 1") - - assert str(error.value) == ( - "DataWeave lifecycle and execution are not allowed from a native callback on the same thread." - ) - - @pytest.mark.unit @pytest.mark.parametrize("failure_depth", [1, 2]) def test_native_callback_scope_restores_depth_after_success_error_and_nesting(failure_depth): From bd6a375b6e962f471219026e9d133ba2e5bbb9a6 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 16:45:21 -0300 Subject: [PATCH 14/48] fix(python): contain callback base exceptions --- native-lib/python/src/dataweave/runtime.py | 14 +-- native-lib/python/tests/unit/test_native.py | 9 +- .../python/tests/unit/test_streaming.py | 93 ++++++++++++++++++- 3 files changed, 106 insertions(+), 10 deletions(-) diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index 1f02b853..4b32a639 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -117,7 +117,7 @@ def run(self, script: str, inputs: Optional[Dict[str, Any]] = None, raise_on_err ) result = parse_native_encoded_response(raw) except Exception as error: - raise DataWeaveError(f"Failed to execute script: {error}") from error + raise DataWeaveError(f"Failed to execute script: {error}") if raise_on_error and not result.success: raise DataWeaveScriptError(result) return result @@ -130,13 +130,13 @@ def write_cb(_context, buffer, length): data = ctypes.string_at(buffer, length) with _native_callback_scope(): return write_callback(data) - except Exception: + except BaseException: return -1 try: raw = self._native.run_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) except Exception as error: - raise DataWeaveError(f"Failed to execute callback streaming: {error}") from error + raise DataWeaveError(f"Failed to execute callback streaming: {error}") def _stream_worker(self, invoke, cancelled: Event) -> Generator[bytes, None, StreamingResult]: sentinel = object() @@ -250,7 +250,7 @@ def read_cb(_context, buffer, buffer_size): return 0 state["chunk"] = chunk state["offset"] = 0 - except Exception: + except BaseException: return -1 return read_cb @@ -279,7 +279,7 @@ def read_cb(_context, buffer, buffer_size): return -1 ctypes.memmove(buffer, data, len(data)) return len(data) - except Exception: + except BaseException: return -1 @WRITE_CALLBACK def write_cb(_context, buffer, length): @@ -287,13 +287,13 @@ def write_cb(_context, buffer, length): data = ctypes.string_at(buffer, length) with _native_callback_scope(): return write_callback(data) - except Exception: + except BaseException: return -1 try: raw = self._native.run_input_output_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) except Exception as error: - raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") from error + raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") def __enter__(self): self.initialize() diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index f637df01..40a01078 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -9,6 +9,10 @@ from dataweave import native +class CallbackBaseException(BaseException): + pass + + class Function: pass @@ -494,12 +498,12 @@ def test_native_callback_scope_restores_depth_after_success_error_and_nesting(fa @pytest.mark.unit -def test_resolver_callback_exception_restores_native_callback_depth(monkeypatch): +def test_resolver_callback_contains_base_exception_and_restores_native_callback_depth(monkeypatch, capsys): library = FakeLibrary() monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) runtime = native.NativeRuntime("/tmp/dwlib") runtime.install_resolver( - lambda _path: (_ for _ in ()).throw(RuntimeError("resolver failed")) + lambda _path: (_ for _ in ()).throw(CallbackBaseException("resolver failed")) ) runtime.initialize() _handle, callback, context = library.created_engines[0] @@ -509,6 +513,7 @@ def test_resolver_callback_exception_restores_native_callback_depth(monkeypatch) assert not hasattr(native._native_callback_state, "depth") native._raise_if_native_callback_active() + assert capsys.readouterr().err == "DataWeave module resolver callback failed.\n" runtime.cleanup() diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index 32affd93..ac6c9da4 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -1,5 +1,6 @@ import ctypes from queue import Full, Queue +import sys from threading import Condition, Event, Lock, Thread from time import sleep @@ -63,13 +64,27 @@ def run_script_input_output_callback_engine( self.read_input = b"".join(read) if self.emit: buffer = ctypes.create_string_buffer(self.emit) - assert write_callback(None, ctypes.addressof(buffer), len(self.emit)) == 0 + self.write_status = write_callback(None, ctypes.addressof(buffer), len(self.emit)) + if self.write_status != 0: + return self._response_pointer() return self._response_pointer() def destroy_engine(self, _thread, _handle): self.destroyed_handle = _handle +class CallbackBaseException(BaseException): + pass + + +class UnraisableRecorder: + def __init__(self): + self.unraisable = [] + + def __call__(self, unraisable): + self.unraisable.append(unraisable) + + def configured_runtime(native): runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) native_runtime = runtime_module.NativeRuntime.__new__(runtime_module.NativeRuntime) @@ -119,6 +134,82 @@ def test_run_input_output_callback_converts_read_exception_to_abort_result(): assert native.read_status == -1 +@pytest.mark.unit +def test_run_callback_contains_write_callback_base_exception(monkeypatch): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = runtime.run_callback( + "script", + lambda _chunk: (_ for _ in ()).throw(CallbackBaseException("stop")), + ) + + assert native.write_status == -1 + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + +@pytest.mark.unit +def test_run_input_output_callback_contains_read_callback_base_exception(monkeypatch): + native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = runtime.run_input_output_callback( + "script", + "payload", + "application/json", + lambda _size: (_ for _ in ()).throw(CallbackBaseException("stop")), + lambda _data: 0, + ) + + assert native.read_status == -1 + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "read aborted", None, None, False) + + +@pytest.mark.unit +def test_run_input_output_callback_contains_write_callback_base_exception(monkeypatch): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk", consume_input=True) + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = runtime.run_input_output_callback( + "script", + "payload", + "application/json", + lambda _size: b"", + lambda _data: (_ for _ in ()).throw(CallbackBaseException("stop")), + ) + + assert native.write_status == -1 + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + +@pytest.mark.unit +def test_transform_contains_input_iterator_base_exception(monkeypatch): + native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + def input_stream(): + raise CallbackBaseException("stop") + yield b"" # pragma: no cover + + stream = runtime.run_transform("script", input_stream()) + + assert list(stream) == [] + assert native.read_status == -1 + assert recorder.unraisable == [] + assert stream.metadata == dataweave.StreamingResult(False, "read aborted", None, None, False) + + @pytest.mark.unit def test_write_callback_reentry_is_translated_to_abort_without_deadlocking(): outer_native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") From 789b711f18bb6014ae87672af6c0b915a49dadad Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 17:34:49 -0300 Subject: [PATCH 15/48] fix(python): harden callback admission --- native-lib/python/src/dataweave/__init__.py | 3 + native-lib/python/src/dataweave/runtime.py | 7 +- native-lib/python/tests/unit/test_facade.py | 39 +++++ .../python/tests/unit/test_streaming.py | 143 ++++++++++++++++++ 4 files changed, 189 insertions(+), 3 deletions(-) diff --git a/native-lib/python/src/dataweave/__init__.py b/native-lib/python/src/dataweave/__init__.py index 5168ab93..91cb802e 100644 --- a/native-lib/python/src/dataweave/__init__.py +++ b/native-lib/python/src/dataweave/__init__.py @@ -24,6 +24,7 @@ ) from .native import candidate_library_paths as _candidate_library_paths from .native import find_library as _find_library +from .native import _raise_if_native_callback_active from .resolver import ( ModuleResolver, compose_resolvers, @@ -40,6 +41,7 @@ def _get_global_instance() -> DataWeave: global _global_instance + _raise_if_native_callback_active() with _global_lock: if _global_instance is None: import atexit @@ -72,6 +74,7 @@ def run_input_output_callback(script: str, input_name: str, input_mime_type: str def cleanup() -> None: global _global_instance + _raise_if_native_callback_active() with _global_lock: if _global_instance is not None: instance, _global_instance = _global_instance, None diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index 4b32a639..2232b3e3 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -129,7 +129,7 @@ def write_cb(_context, buffer, length): try: data = ctypes.string_at(buffer, length) with _native_callback_scope(): - return write_callback(data) + return int(write_callback(data)) except BaseException: return -1 try: @@ -139,6 +139,7 @@ def write_cb(_context, buffer, length): raise DataWeaveError(f"Failed to execute callback streaming: {error}") def _stream_worker(self, invoke, cancelled: Event) -> Generator[bytes, None, StreamingResult]: + _raise_if_native_callback_active() sentinel = object() queue: Queue = Queue(maxsize=_OUTPUT_QUEUE_MAXSIZE) @@ -161,7 +162,7 @@ def write_cb(_context, buffer, length): return -1 queue.put(ctypes.string_at(buffer, length), timeout=_WORKER_TIMEOUT_SECONDS) return 0 - except Exception: + except BaseException: return -1 def worker_main(): @@ -286,7 +287,7 @@ def write_cb(_context, buffer, length): try: data = ctypes.string_at(buffer, length) with _native_callback_scope(): - return write_callback(data) + return int(write_callback(data)) except BaseException: return -1 try: diff --git a/native-lib/python/tests/unit/test_facade.py b/native-lib/python/tests/unit/test_facade.py index 23ec8de6..ee7fbb10 100644 --- a/native-lib/python/tests/unit/test_facade.py +++ b/native-lib/python/tests/unit/test_facade.py @@ -159,6 +159,45 @@ def cleanup(self): assert registered == [dataweave.cleanup, dataweave.cleanup] +@pytest.mark.unit +def test_module_cleanup_from_native_callback_preserves_published_instance(monkeypatch): + instance = configured_runtime() + cleanup_calls = [] + instance.cleanup = lambda: cleanup_calls.append(True) + monkeypatch.setattr(dataweave, "_global_instance", instance) + + with native._native_callback_scope(), pytest.raises(dataweave.DataWeaveError, match="native callback"): + dataweave.cleanup() + + assert dataweave._global_instance is instance + assert cleanup_calls == [] + assert dataweave.run("payload").get_string() == "Hello" + + +@pytest.mark.unit +def test_module_execution_from_native_callback_rejects_before_global_lock_or_native_run(monkeypatch): + instance = configured_runtime() + monkeypatch.setattr(dataweave, "_global_instance", instance) + lock_calls = [] + + class UnexpectedLock: + def __enter__(self): + lock_calls.append("enter") + raise AssertionError("global lock acquired") + + def __exit__(self, _exc_type, _exc_value, _traceback): + pass + + monkeypatch.setattr(dataweave, "_global_lock", UnexpectedLock()) + + with native._native_callback_scope(): + with pytest.raises(dataweave.DataWeaveError, match="native callback"): + dataweave.run("payload") + + assert lock_calls == [] + assert instance._native.calls == [] + + @pytest.mark.unit def test_cleanup_is_noop_without_global_runtime(): dataweave.cleanup() diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index ac6c9da4..ad2c8b88 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -85,6 +85,17 @@ def __call__(self, unraisable): self.unraisable.append(unraisable) +class ReentrantWriteStatus: + def __init__(self, runtime): + self.runtime = runtime + self.int_calls = 0 + + def __int__(self): + self.int_calls += 1 + self.runtime.run("nested") + return 0 + + def configured_runtime(native): runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) native_runtime = runtime_module.NativeRuntime.__new__(runtime_module.NativeRuntime) @@ -151,6 +162,75 @@ def test_run_callback_contains_write_callback_base_exception(monkeypatch): assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) +@pytest.mark.unit +@pytest.mark.parametrize( + "invoke", + [ + lambda runtime, write_callback: runtime.run_callback("script", write_callback), + lambda runtime, write_callback: runtime.run_input_output_callback( + "script", "payload", "application/json", lambda _size: b"", write_callback, + ), + ], +) +def test_public_write_callback_normalizes_invalid_status_to_abort_without_unraisable(monkeypatch, invoke): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = invoke(runtime, lambda _data: None) + + assert native.write_status == -1 + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + +@pytest.mark.unit +@pytest.mark.parametrize( + "invoke", + [ + lambda runtime, write_callback: runtime.run_callback("script", write_callback), + lambda runtime, write_callback: runtime.run_input_output_callback( + "script", "payload", "application/json", lambda _size: b"", write_callback, + ), + ], +) +def test_public_write_callback_preserves_integer_status(invoke): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + + result = invoke(runtime, lambda _data: -7) + + assert native.write_status == -7 + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + +@pytest.mark.unit +@pytest.mark.parametrize( + "invoke", + [ + lambda runtime, write_callback: runtime.run_callback("script", write_callback), + lambda runtime, write_callback: runtime.run_input_output_callback( + "script", "payload", "application/json", lambda _size: b"", write_callback, + ), + ], +) +def test_public_write_callback_converts_custom_status_inside_native_callback_scope(monkeypatch, invoke): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + status = ReentrantWriteStatus(runtime) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = invoke(runtime, lambda _data: status) + + assert native.write_status == -1 + assert status.int_calls == 1 + assert native.attach_count == 1 # Only the outer callback invocation attached. + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + @pytest.mark.unit def test_run_input_output_callback_contains_read_callback_base_exception(monkeypatch): native = FakeNative('{"success": false, "error": "read aborted"}', consume_input=True) @@ -288,6 +368,69 @@ def input_stream(): assert stream.metadata == dataweave.StreamingResult(False, "read aborted", None, None, False) +@pytest.mark.unit +@pytest.mark.parametrize( + "create_stream", + [ + lambda runtime: runtime.run_streaming("script"), + lambda runtime: runtime.run_transform("script", [b"null"]), + ], +) +def test_precreated_stream_rejects_callback_time_first_consumption_before_worker_admission(monkeypatch, create_stream): + native = FakeNative('{"success": true}') + runtime = configured_runtime(native) + stream = create_stream(runtime) + registrations = [] + starts = [] + original_register = runtime._register_stream_worker + + def record_registration(worker): + registrations.append(worker) + return original_register(worker) + + class UnexpectedThread: + def __init__(self, *_args, **_kwargs): + starts.append("constructed") + + monkeypatch.setattr(runtime, "_register_stream_worker", record_registration) + monkeypatch.setattr(runtime_module, "Thread", UnexpectedThread) + + with runtime_module._native_callback_scope(), pytest.raises(dataweave.DataWeaveError, match="native callback"): + next(stream) + + assert registrations == [] + assert starts == [] + assert native.attach_count == 0 + + +@pytest.mark.unit +def test_internal_streaming_write_trampoline_contains_base_exception(monkeypatch): + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + class RaisingCancel(Event): + def __init__(self): + super().__init__() + self.raise_once = True + + def is_set(self): + if self.raise_once: + self.raise_once = False + raise CallbackBaseException("cancel check failed") + return super().is_set() + + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + monkeypatch.setattr(runtime_module, "Event", RaisingCancel) + stream = runtime.run_streaming("script") + + assert list(stream) == [] + + assert native.write_status == -1 + assert recorder.unraisable == [] + assert stream.metadata == dataweave.StreamingResult(False, "write aborted", None, None, False) + + @pytest.mark.unit @pytest.mark.parametrize( "invoke", From efa6f291b548811dec14bd4bdb1d8cf0f469343e Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 4 Sep 2026 19:11:44 -0300 Subject: [PATCH 16/48] fix(python): normalize callback status ABI --- native-lib/python/src/dataweave/runtime.py | 4 +- .../python/tests/unit/test_streaming.py | 83 ++++++++++++++++++- 2 files changed, 81 insertions(+), 6 deletions(-) diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index 2232b3e3..3e1e102b 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -129,7 +129,7 @@ def write_cb(_context, buffer, length): try: data = ctypes.string_at(buffer, length) with _native_callback_scope(): - return int(write_callback(data)) + return ctypes.c_int(write_callback(data)).value except BaseException: return -1 try: @@ -287,7 +287,7 @@ def write_cb(_context, buffer, length): try: data = ctypes.string_at(buffer, length) with _native_callback_scope(): - return int(write_callback(data)) + return ctypes.c_int(write_callback(data)).value except BaseException: return -1 try: diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index ad2c8b88..e733447c 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -88,14 +88,24 @@ def __call__(self, unraisable): class ReentrantWriteStatus: def __init__(self, runtime): self.runtime = runtime - self.int_calls = 0 + self.index_calls = 0 - def __int__(self): - self.int_calls += 1 + def __index__(self): + self.index_calls += 1 self.runtime.run("nested") return 0 +class IndexOnlyWriteStatus: + def __init__(self, value): + self.value = value + self.index_calls = 0 + + def __index__(self): + self.index_calls += 1 + return self.value + + def configured_runtime(native): runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) native_runtime = runtime_module.NativeRuntime.__new__(runtime_module.NativeRuntime) @@ -205,6 +215,71 @@ def test_public_write_callback_preserves_integer_status(invoke): assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) +@pytest.mark.unit +@pytest.mark.parametrize( + "status, expected", + [ + ((1 << 200) + 23, 23), + (-((1 << 200) + 23), -23), + (IndexOnlyWriteStatus((1 << 80) + 23), 23), + ], + ids=["large-positive", "large-negative", "index-only"], +) +@pytest.mark.parametrize( + "invoke", + [ + lambda runtime, write_callback: runtime.run_callback("script", write_callback), + lambda runtime, write_callback: runtime.run_input_output_callback( + "script", "payload", "application/json", lambda _size: b"", write_callback, + ), + ], +) +def test_public_write_callback_uses_c_int_status_semantics_without_unraisable(monkeypatch, invoke, status, expected): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + + result = invoke(runtime, lambda _data: status) + + assert native.write_status == expected + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + +@pytest.mark.unit +@pytest.mark.parametrize( + "status, expected", + [ + ((1 << 200) + 23, 23), + (-((1 << 200) + 23), -23), + (IndexOnlyWriteStatus((1 << 80) + 23), 23), + ], + ids=["large-positive", "large-negative", "index-only"], +) +@pytest.mark.parametrize( + "invoke", + [ + lambda runtime, write_callback: runtime.run_callback("script", write_callback), + lambda runtime, write_callback: runtime.run_input_output_callback( + "script", "payload", "application/json", lambda _size: b"", write_callback, + ), + ], +) +def test_public_write_callback_normalizes_status_before_ctypes_return(monkeypatch, invoke, status, expected): + native = FakeNative('{"success": false, "error": "write aborted"}', emit=b"chunk") + runtime = configured_runtime(native) + recorder = UnraisableRecorder() + monkeypatch.setattr(sys, "unraisablehook", recorder) + monkeypatch.setattr(runtime_module, "WRITE_CALLBACK", lambda callback: callback) + + result = invoke(runtime, lambda _data: status) + + assert native.write_status == expected + assert recorder.unraisable == [] + assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) + + @pytest.mark.unit @pytest.mark.parametrize( "invoke", @@ -225,7 +300,7 @@ def test_public_write_callback_converts_custom_status_inside_native_callback_sco result = invoke(runtime, lambda _data: status) assert native.write_status == -1 - assert status.int_calls == 1 + assert status.index_calls == 1 assert native.attach_count == 1 # Only the outer callback invocation attached. assert recorder.unraisable == [] assert result == dataweave.StreamingResult(False, "write aborted", None, None, False) From aca8a825204ceb5a8684ce3943c99c6ddee5af04 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 10:36:14 -0300 Subject: [PATCH 17/48] fix(python): bind operations to engine generations --- native-lib/python/src/dataweave/native.py | 135 ++++++++------ native-lib/python/src/dataweave/runtime.py | 35 ++-- .../tests/integration/test_streaming.py | 46 +++++ native-lib/python/tests/unit/test_native.py | 166 +++++++++++++++++- .../python/tests/unit/test_streaming.py | 88 +++++++++- 5 files changed, 393 insertions(+), 77 deletions(-) diff --git a/native-lib/python/src/dataweave/native.py b/native-lib/python/src/dataweave/native.py index 09c68788..627e6441 100644 --- a/native-lib/python/src/dataweave/native.py +++ b/native-lib/python/src/dataweave/native.py @@ -1,5 +1,6 @@ import ctypes from contextlib import contextmanager +from dataclasses import dataclass import os from pathlib import Path import sys @@ -362,6 +363,12 @@ def find_library() -> str: ) +@dataclass(frozen=True) +class _EngineOperation: + handle: int + generation: int + + class NativeRuntime: """Owns the native library handle, isolate lifecycle, and ctypes ABI.""" @@ -372,6 +379,8 @@ def __init__(self, lib_path: Optional[str] = None): self.thread = None self.handle = 0 self.initialized = False + self._generation = 0 + self._engine_operation = None # Every engine supports every API now (single unified ABI). self.has_callback_streaming = True self.has_callback_input_output = True @@ -396,35 +405,37 @@ def __init__(self, lib_path: Optional[str] = None): def initialize(self) -> None: _raise_if_native_callback_active() - if self.initialized: - return - with self._init_lock: - if self.initialized: - return - acquired = False - try: - self.lib, self.isolate = _acquire_isolate(self.lib_path) - acquired = True - self.handle = self._create_engine() - except Exception: - # Roll back the ref we just took (if any) so a failed init leaks - # nothing. - self.lib = self.isolate = None - # Finding #2: install_resolver() registered a token BEFORE this call. - # A failed init must unregister it, or it leaks: self.initialized stays - # False, so a later cleanup() returns early and never reaches the pop. - if self._resolver_token: - with _resolver_lock_global: - _resolver_registry.pop(self._resolver_token, None) - self._resolver_token = 0 - # Release the ref only if _acquire_isolate actually incremented it - # (a library-load / isolate-create / bootstrap-detach failure inside - # _acquire_isolate never increments the refcount, so releasing here - # unconditionally would decrement someone else's live reference). - if acquired: - _release_isolate() - raise - self.initialized = True + with self._serialized_native_operation(): + with self._init_lock: + if self.initialized: + return + acquired = False + try: + self.lib, self.isolate = _acquire_isolate(self.lib_path) + acquired = True + handle = self._create_engine() + except Exception: + # Roll back the ref we just took (if any) so a failed init leaks + # nothing. + self.lib = self.isolate = None + # Finding #2: install_resolver() registered a token BEFORE this call. + # A failed init must unregister it, or it leaks: self.initialized stays + # False, so a later cleanup() returns early and never reaches the pop. + if self._resolver_token: + with _resolver_lock_global: + _resolver_registry.pop(self._resolver_token, None) + self._resolver_token = 0 + # Release the ref only if _acquire_isolate actually incremented it + # (a library-load / isolate-create / bootstrap-detach failure inside + # _acquire_isolate never increments the refcount, so releasing here + # unconditionally would decrement someone else's live reference). + if acquired: + _release_isolate() + raise + self.handle = handle + self._generation += 1 + self._engine_operation = _EngineOperation(handle, self._generation) + self.initialized = True def _create_engine(self) -> int: with self._current_thread_attachment(self.thread) as thread: @@ -483,27 +494,29 @@ def decode_and_free(self, ptr, thread=None) -> str: if primary_error is None: raise - def run_engine_and_decode(self, script: bytes, inputs: bytes) -> str: - with self._serialized_native_operation() as operation_lock: + def run_engine_and_decode(self, script: bytes, inputs: bytes, operation: _EngineOperation) -> str: + with self._serialized_native_operation(operation) as operation_lock: with self._current_thread_attachment(self.thread) as thread: with self._resolver_scope(): try: operation_lock.release() return self.decode_and_free( - self.lib.run_script_engine(thread, self.handle, script, inputs), + self.lib.run_script_engine(thread, operation.handle, script, inputs), thread, ) finally: operation_lock.acquire() - def run_callback_engine_and_decode(self, thread, script: bytes, inputs: bytes, write_callback) -> str: - with self._serialized_native_operation() as operation_lock: + def run_callback_engine_and_decode( + self, thread, script: bytes, inputs: bytes, write_callback, operation: _EngineOperation, + ) -> str: + with self._serialized_native_operation(operation) as operation_lock: with self._current_thread_attachment(thread) as current: try: operation_lock.release() return self.decode_and_free( self.lib.run_script_callback_engine( - current, self.handle, script, inputs, write_callback, None + current, operation.handle, script, inputs, write_callback, None ), current, ) @@ -512,15 +525,16 @@ def run_callback_engine_and_decode(self, thread, script: bytes, inputs: bytes, w def run_input_output_callback_engine_and_decode( self, thread, script: bytes, inputs: bytes, input_name: bytes, - input_mime_type: bytes, input_charset: Optional[bytes], read_callback, write_callback, + input_mime_type: bytes, input_charset: Optional[bytes], read_callback, + write_callback, operation: _EngineOperation, ) -> str: - with self._serialized_native_operation() as operation_lock: + with self._serialized_native_operation(operation) as operation_lock: with self._current_thread_attachment(thread) as current: try: operation_lock.release() return self.decode_and_free( self.lib.run_script_input_output_callback_engine( - current, self.handle, script, inputs, input_name, + current, operation.handle, script, inputs, input_name, input_mime_type, input_charset, read_callback, write_callback, None, ), current, @@ -591,12 +605,10 @@ def _resolver_scope(self): def cleanup(self) -> None: _raise_if_native_callback_active() with self._serialized_native_operation(): - # _init_lock is nested INSIDE _operation_lock here (never the - # reverse -- initialize() only ever takes _init_lock alone, and - # never takes _operation_lock), so there is no lock-ordering - # inversion between the two. Only the initialized-clearing flag - # flip needs the lock; the actual destroy/release below stays - # outside it, guarded by _serialized_native_operation as before. + # _init_lock is nested INSIDE _operation_lock in both initialize() + # and cleanup(), so lifecycle transitions use one lock order. Only + # the initialized/token clearing needs _init_lock; destroy/release + # stays guarded by _serialized_native_operation as before. # Mirrors _serialized_native_operation's own hasattr guard below: # some unit tests build a NativeRuntime via __new__ and set # attributes directly, bypassing __init__. @@ -605,11 +617,13 @@ def cleanup(self) -> None: with self._init_lock: if not self.initialized: return + operation = self._engine_operation self.initialized = False + self._engine_operation = None try: - if self.handle: + if operation is not None: with self._current_thread_attachment(self.thread) as thread: - self.lib.destroy_engine(thread, self.handle) + self.lib.destroy_engine(thread, operation.handle) finally: # Release the isolate ref even if destroy_engine throws, so a # throwing destroy cannot strand the isolate. @@ -626,7 +640,7 @@ def cleanup(self) -> None: _release_isolate() @contextmanager - def _serialized_native_operation(self): + def _serialized_native_operation(self, expected: Optional[_EngineOperation] = None): _raise_if_native_callback_active() owner = get_ident() if getattr(self, "_execution_owner", None) == owner: @@ -637,6 +651,8 @@ def _serialized_native_operation(self): with self._operation_lock: while getattr(self, "_operation_active", False): self._operation_lock.wait() + if expected is not None: + self._validate_operation_locked(expected) self._operation_active = True self._execution_owner = owner try: @@ -646,11 +662,28 @@ def _serialized_native_operation(self): self._operation_active = False self._operation_lock.notify() - def capture_operation(self) -> None: + def _validate_operation_locked(self, expected: _EngineOperation) -> None: + if self._engine_operation != expected: + raise DataWeaveError("DataWeave operation belongs to a stale engine generation.") + + def validate_operation(self, expected: _EngineOperation) -> None: + _raise_if_native_callback_active() + if not hasattr(self, "_operation_lock"): + self._operation_lock = Condition(Lock()) + self._operation_active = False + with self._operation_lock: + self._validate_operation_locked(expected) + + def capture_operation(self) -> _EngineOperation: _raise_if_native_callback_active() - if not self.initialized: - raise DataWeaveError("DataWeave runtime not initialized. Call initialize() first.") - return None + if not hasattr(self, "_operation_lock"): + self._operation_lock = Condition(Lock()) + self._operation_active = False + with self._operation_lock: + operation = self._engine_operation + if not self.initialized or operation is None: + raise DataWeaveError("DataWeave runtime not initialized. Call initialize() first.") + return operation @contextmanager def _current_thread_attachment(self, thread): diff --git a/native-lib/python/src/dataweave/runtime.py b/native-lib/python/src/dataweave/runtime.py index 3e1e102b..e23e5be7 100644 --- a/native-lib/python/src/dataweave/runtime.py +++ b/native-lib/python/src/dataweave/runtime.py @@ -86,11 +86,12 @@ def _worker_registry(self): self._stream_workers_lock = Lock() return self._stream_workers, self._stream_workers_lock - def _register_stream_worker(self, worker: Thread) -> None: + def _register_stream_worker(self, worker: Thread, operation) -> None: workers, lock = self._worker_registry() with lock: if getattr(self, "_cleaning_up", False): raise DataWeaveError("Cannot start a streaming worker while the DataWeave runtime is being cleaned up.") + self._native.validate_operation(operation) workers.add(worker) def _unregister_stream_worker(self, worker: Optional[Thread] = None) -> None: @@ -98,22 +99,22 @@ def _unregister_stream_worker(self, worker: Optional[Thread] = None) -> None: with lock: workers.discard(worker or current_thread()) - def _require_initialized(self, supported: bool, api_name: str) -> None: + def _require_initialized(self, supported: bool, api_name: str): _raise_if_native_callback_active() - if not self._native.initialized: - raise DataWeaveError("DataWeave runtime not initialized. Call initialize() first.") + operation = self._native.capture_operation() if not supported: raise DataWeaveError(f"Native library does not support {api_name}.") + return operation @staticmethod def _inputs_json(inputs: Optional[Dict[str, Any]]) -> bytes: return json.dumps({key: normalize_input_value(value) for key, value in (inputs or {}).items()}).encode("utf-8") def run(self, script: str, inputs: Optional[Dict[str, Any]] = None, raise_on_error: bool = False) -> ExecutionResult: - self._require_initialized(True, "script execution") + operation = self._require_initialized(True, "script execution") try: raw = self._native.run_engine_and_decode( - script.encode("utf-8"), self._inputs_json(inputs) + script.encode("utf-8"), self._inputs_json(inputs), operation=operation ) result = parse_native_encoded_response(raw) except Exception as error: @@ -123,7 +124,7 @@ def run(self, script: str, inputs: Optional[Dict[str, Any]] = None, raise_on_err return result def run_callback(self, script: str, write_callback: WriteCallback, inputs: Optional[Dict[str, Any]] = None) -> StreamingResult: - self._require_initialized(self._native.has_callback_streaming, "callback streaming API (run_script_callback not found)") + operation = self._require_initialized(self._native.has_callback_streaming, "callback streaming API (run_script_callback not found)") @WRITE_CALLBACK def write_cb(_context, buffer, length): try: @@ -133,12 +134,12 @@ def write_cb(_context, buffer, length): except BaseException: return -1 try: - raw = self._native.run_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), write_cb) + raw = self._native.run_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), write_cb, operation=operation) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) except Exception as error: raise DataWeaveError(f"Failed to execute callback streaming: {error}") - def _stream_worker(self, invoke, cancelled: Event) -> Generator[bytes, None, StreamingResult]: + def _stream_worker(self, operation, invoke, cancelled: Event) -> Generator[bytes, None, StreamingResult]: _raise_if_native_callback_active() sentinel = object() queue: Queue = Queue(maxsize=_OUTPUT_QUEUE_MAXSIZE) @@ -191,7 +192,7 @@ def worker_main(): # Python cannot cancel a native call. Daemon workers keep an abandoned # call from extending interpreter lifetime after bounded cancellation. worker = Thread(target=worker_main, name="dw-streaming-worker", daemon=True) - self._register_stream_worker(worker) + self._register_stream_worker(worker, operation) try: worker.start() except Exception: @@ -220,10 +221,10 @@ def worker_main(): return parse_streaming_result(metadata or {"success": False, "error": "No metadata received from native call"}) def run_streaming(self, script: str, inputs: Optional[Dict[str, Any]] = None) -> Stream: - self._require_initialized(self._native.has_callback_streaming, "callback streaming API (run_script_callback not found)") + operation = self._require_initialized(self._native.has_callback_streaming, "callback streaming API (run_script_callback not found)") cancelled = Event() encoded_inputs = self._inputs_json(inputs) - stream = Stream(self._stream_worker(lambda thread, write_cb: self._native.run_callback_engine_and_decode(thread, script.encode("utf-8"), encoded_inputs, write_cb), cancelled)) + stream = Stream(self._stream_worker(operation, lambda thread, write_cb: self._native.run_callback_engine_and_decode(thread, script.encode("utf-8"), encoded_inputs, write_cb, operation=operation), cancelled)) stream._on_close = cancelled.set stream._cancelled = cancelled return stream @@ -256,19 +257,19 @@ def read_cb(_context, buffer, buffer_size): return read_cb def run_transform(self, script: str, input_stream: Iterable[bytes], input_name: str = "payload", input_mime_type: str = "application/json", input_charset: Optional[str] = None, inputs: Optional[Dict[str, Any]] = None) -> Stream: - self._require_initialized(self._native.has_callback_input_output, "callback input/output API (run_script_input_output_callback not found)") + operation = self._require_initialized(self._native.has_callback_input_output, "callback input/output API (run_script_input_output_callback not found)") cancelled = Event() read_cb = self._chunk_reader(input_stream) encoded_inputs = self._inputs_json(inputs) def invoke(thread, write_cb): - return self._native.run_input_output_callback_engine_and_decode(thread, script.encode("utf-8"), encoded_inputs, input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb) - stream = Stream(self._stream_worker(invoke, cancelled)) + return self._native.run_input_output_callback_engine_and_decode(thread, script.encode("utf-8"), encoded_inputs, input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb, operation=operation) + stream = Stream(self._stream_worker(operation, invoke, cancelled)) stream._on_close = cancelled.set stream._cancelled = cancelled return stream def run_input_output_callback(self, script: str, input_name: str, input_mime_type: str, read_callback: ReadCallback, write_callback: WriteCallback, input_charset: Optional[str] = None, inputs: Optional[Dict[str, Any]] = None) -> StreamingResult: - self._require_initialized(self._native.has_callback_input_output, "callback input/output API (run_script_input_output_callback not found)") + operation = self._require_initialized(self._native.has_callback_input_output, "callback input/output API (run_script_input_output_callback not found)") @READ_CALLBACK def read_cb(_context, buffer, buffer_size): try: @@ -291,7 +292,7 @@ def write_cb(_context, buffer, length): except BaseException: return -1 try: - raw = self._native.run_input_output_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb) + raw = self._native.run_input_output_callback_engine_and_decode(self._native.thread, script.encode("utf-8"), self._inputs_json(inputs), input_name.encode("utf-8"), input_mime_type.encode("utf-8"), input_charset.encode("utf-8") if input_charset else None, read_cb, write_cb, operation=operation) return parse_streaming_result(json.loads(raw) if raw else {"success": False, "error": "Empty response"}) except Exception as error: raise DataWeaveError(f"Failed to execute callback input/output streaming: {error}") diff --git a/native-lib/python/tests/integration/test_streaming.py b/native-lib/python/tests/integration/test_streaming.py index dc92c272..6a1f2406 100644 --- a/native-lib/python/tests/integration/test_streaming.py +++ b/native-lib/python/tests/integration/test_streaming.py @@ -6,6 +6,52 @@ import dataweave +def _staged_native_library() -> Path: + native_dir = Path(__file__).resolve().parents[2] / "src" / "dataweave" / "native" + return next( + path + for path in ( + native_dir / "dwlib.dylib", + native_dir / "dwlib.so", + native_dir / "dwlib.dll", + ) + if path.is_file() + ) + + +@pytest.mark.integration +@pytest.mark.parametrize( + "create_stream", + [ + lambda runtime: runtime.run_streaming("output application/json --- 1"), + lambda runtime: runtime.run_transform( + "output application/json --- payload", + [b"null"], + ), + ], + ids=["run-streaming", "run-transform"], +) +def test_real_native_precreated_stream_rejects_stale_generation(create_stream): + lib_path = str(_staged_native_library()) + keeper = dataweave.DataWeave(lib_path) + runtime = dataweave.DataWeave(lib_path) + keeper.initialize() + runtime.initialize() + stream = create_stream(runtime) + old_handle = runtime._native.handle + try: + runtime.cleanup() + runtime.initialize() + assert runtime._native.handle != old_handle + + with pytest.raises(dataweave.DataWeaveError, match="stale engine generation"): + next(stream) + finally: + stream.close() + runtime.cleanup() + keeper.cleanup() + + @pytest.mark.integration def test_run_streaming_returns_chunks_and_metadata(collect_stream): output, metadata = collect_stream(dataweave.run_streaming("output application/json --- {a: 1, b: 2}")) diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index 40a01078..cf549453 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -1,7 +1,7 @@ from pathlib import Path import ctypes import threading -from threading import Barrier, BrokenBarrierError, Condition, current_thread, get_ident, Lock, Thread +from threading import Barrier, BrokenBarrierError, Condition, current_thread, Event, get_ident, Lock, Thread import pytest @@ -83,6 +83,158 @@ def _attach_thread(self, _isolate, thread): return 0 +@pytest.mark.unit +def test_run_admission_generation_rejects_operation_after_cleanup_and_reinitialize(monkeypatch): + library = FakeLibrary() + result_buffer = ctypes.create_string_buffer( + b'{"success":true,"result":"","binary":false,"mimeType":"application/json","charset":"UTF-8"}' + ) + run_handles = [] + library.run_script_engine = CallableFunction( + lambda _thread, handle, _script, _inputs: run_handles.append(handle) + or ctypes.addressof(result_buffer) + ) + library.free_cstring = CallableFunction(lambda _thread, _ptr: None) + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + wrapper = dataweave.DataWeave("/tmp/dwlib") + wrapper.initialize() + old_handle = wrapper._native.handle + captured = Event() + resume = Event() + original_require_initialized = wrapper._require_initialized + + def pause_after_capture(supported, api_name): + operation = original_require_initialized(supported, api_name) + captured.set() + assert resume.wait(1) + return operation + + monkeypatch.setattr(wrapper, "_require_initialized", pause_after_capture) + errors = [] + worker = Thread(target=lambda: _capture_error(errors, lambda: wrapper.run("1"))) + worker.start() + assert captured.wait(1) + + wrapper.cleanup() + wrapper.initialize() + replacement_handle = wrapper._native.handle + assert replacement_handle != old_handle + resume.set() + worker.join(1) + + assert not worker.is_alive() + assert len(errors) == 1 + assert "stale engine generation" in str(errors[0]) + assert run_handles == [] + wrapper.cleanup() + + +@pytest.mark.unit +def test_failed_initialize_does_not_publish_or_advance_engine_generation(monkeypatch): + library = FakeLibrary() + library.create_engine = CallableFunction(lambda _thread: 0) + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + + with pytest.raises(dataweave.DataWeaveError, match="null handle"): + runtime.initialize() + + assert runtime._generation == 0 + assert runtime._engine_operation is None + + +@pytest.mark.unit +def test_engine_operation_is_immutable_and_generation_remains_monotonic(monkeypatch): + library = FakeLibrary() + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + runtime.initialize() + first = runtime.capture_operation() + + with pytest.raises(AttributeError): + first.handle = 99 + runtime.cleanup() + runtime.initialize() + second = runtime.capture_operation() + + assert second.generation == first.generation + 1 + runtime.cleanup() + + +@pytest.mark.unit +def test_run_admission_generation_uses_captured_handle_after_mutable_handle_changes(monkeypatch): + library = FakeLibrary() + result_buffer = ctypes.create_string_buffer(b"result") + run_handles = [] + library.run_script_engine = CallableFunction( + lambda _thread, handle, _script, _inputs: run_handles.append(handle) + or ctypes.addressof(result_buffer) + ) + library.free_cstring = CallableFunction(lambda _thread, _ptr: None) + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + runtime.initialize() + operation = runtime.capture_operation() + runtime.handle = operation.handle + 100 + + assert runtime.run_engine_and_decode(b"script", b"{}", operation=operation) == "result" + assert run_handles == [operation.handle] + runtime.cleanup() + + +@pytest.mark.unit +def test_run_admission_generation_keeps_admitted_operation_on_original_engine(monkeypatch): + library = FakeLibrary() + result_buffer = ctypes.create_string_buffer( + b'{"success":true,"result":"","binary":false,"mimeType":"application/json","charset":"UTF-8"}' + ) + admitted = Event() + release_run = Event() + run_handles = [] + destroyed_handles = [] + + def run_script(_thread, handle, _script, _inputs): + run_handles.append(handle) + admitted.set() + assert release_run.wait(1) + return ctypes.addressof(result_buffer) + + library.run_script_engine = CallableFunction(run_script) + library.free_cstring = CallableFunction(lambda _thread, _ptr: None) + library.destroy_engine = CallableFunction( + lambda _thread, handle: destroyed_handles.append(handle) + ) + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + wrapper = dataweave.DataWeave("/tmp/dwlib") + wrapper.initialize() + old_handle = wrapper._native.handle + run_errors = [] + cleanup_errors = [] + run_worker = Thread( + target=lambda: _capture_error(run_errors, lambda: wrapper.run("1")) + ) + run_worker.start() + assert admitted.wait(1) + + cleanup_worker = Thread( + target=lambda: _capture_error(cleanup_errors, wrapper.cleanup) + ) + cleanup_worker.start() + assert cleanup_worker.is_alive() + assert destroyed_handles == [] + + release_run.set() + run_worker.join(1) + cleanup_worker.join(1) + + assert not run_worker.is_alive() + assert not cleanup_worker.is_alive() + assert run_errors == [] + assert cleanup_errors == [] + assert run_handles == [old_handle] + assert destroyed_handles == [old_handle] + + @pytest.mark.unit def test_shared_isolate_is_created_once_and_torn_down_on_last_release(monkeypatch): library = FakeLibrary() @@ -236,7 +388,7 @@ def test_buffered_worker_execution_uses_one_current_thread_attachment_for_run_de worker = Thread( target=lambda: outcomes.append( - (get_ident(), runtime.run_engine_and_decode(b"script", b"{}")) + (get_ident(), runtime.run_engine_and_decode(b"script", b"{}", operation=runtime.capture_operation())) ) ) worker.start() @@ -294,7 +446,7 @@ def test_attach_on_demand_does_not_cache_by_thread_ident(monkeypatch): worker = Thread( target=lambda: ( observed_threads.append(current_thread()), - outcomes.append(runtime.run_engine_and_decode(b"script", b"{}")), + outcomes.append(runtime.run_engine_and_decode(b"script", b"{}", operation=runtime.capture_operation())), ) ) worker.start() @@ -362,7 +514,7 @@ def free_cstring(_thread, _ptr): worker = Thread( target=lambda: _capture_error( errors, - lambda: runtime.run_engine_and_decode(b"script", b"{}"), + lambda: runtime.run_engine_and_decode(b"script", b"{}", operation=runtime.capture_operation()), ) ) worker.start() @@ -435,6 +587,8 @@ def test_native_callback_scope_is_thread_local(): outcomes = [] runtime = native.NativeRuntime.__new__(native.NativeRuntime) runtime.initialized = True + runtime._engine_operation = native._EngineOperation(1, 1) + runtime._operation_lock = Condition(Lock()) with native._native_callback_scope(): worker = Thread( @@ -447,7 +601,7 @@ def test_native_callback_scope_is_thread_local(): assert not worker.is_alive() assert errors == [] - assert outcomes == [None] + assert outcomes == [native._EngineOperation(1, 1)] with pytest.raises(dataweave.DataWeaveError, match="native callback"): native._raise_if_native_callback_active() @@ -544,7 +698,7 @@ def run_script(_thread, _handle, _script, _inputs): library.run_script_engine = CallableFunction(run_script) library.free_cstring = CallableFunction(lambda _thread, _ptr: None) - assert runtime.run_engine_and_decode(b"script", b"{}") == "result" + assert runtime.run_engine_and_decode(b"script", b"{}", operation=runtime.capture_operation()) == "result" assert lock_available == [True] runtime.cleanup() diff --git a/native-lib/python/tests/unit/test_streaming.py b/native-lib/python/tests/unit/test_streaming.py index e733447c..49ace57a 100644 --- a/native-lib/python/tests/unit/test_streaming.py +++ b/native-lib/python/tests/unit/test_streaming.py @@ -7,6 +7,7 @@ import pytest import dataweave +from dataweave import native as native_module from dataweave import runtime as runtime_module @@ -21,6 +22,7 @@ def __init__(self, metadata=None, attach_code=0, emit=b"", consume_input=False): self.freed = [] self._buffers = [] self.detached_event = Event() + self.callback_handles = [] def graal_attach_thread(self, _isolate, _thread): self.attach_count += 1 @@ -42,6 +44,7 @@ def _response_pointer(self): return ctypes.addressof(buffer) def run_script_callback_engine(self, _thread, _handle, _script, _inputs, write_callback, _context): + self.callback_handles.append(_handle) if self.emit: buffer = ctypes.create_string_buffer(self.emit) self.write_status = write_callback(None, ctypes.addressof(buffer), len(self.emit)) @@ -50,6 +53,7 @@ def run_script_callback_engine(self, _thread, _handle, _script, _inputs, write_c def run_script_input_output_callback_engine( self, _thread, _handle, _script, _inputs, _input_name, _mime_type, _charset, read_callback, write_callback, _context, ): + self.callback_handles.append(_handle) if self.consume_input: buffer = ctypes.create_string_buffer(3) read = [] @@ -119,6 +123,8 @@ def configured_runtime(native): # graal_attach_thread/graal_detach_thread. native_runtime.thread = None native_runtime.handle = 1 + native_runtime._generation = 1 + native_runtime._engine_operation = native_module._EngineOperation(1, 1) native_runtime._resolver = None native_runtime._resolver_callback = None native_runtime._resolver_token = 0 @@ -126,11 +132,39 @@ def configured_runtime(native): native_runtime._resolver_active = False native_runtime._resolver_active_ident = None native_runtime._operation_lock = Condition(Lock()) + native_runtime._operation_active = False native_runtime._execution_owner = None runtime._native = native_runtime return runtime +def initialized_runtime(monkeypatch, *, reuse_handle): + native = FakeNative('{"success": true}') + handles = [] + + def acquire(_path): + return native, object() + + def create_engine(_thread): + handle = 1 if reuse_handle else len(handles) + 1 + handles.append(handle) + return handle + + native.create_engine = create_engine + native.create_engine_with_resolver = lambda _thread, _callback, _context: create_engine(_thread) + monkeypatch.setattr(native_module, "_acquire_isolate", acquire) + monkeypatch.setattr(native_module, "_release_isolate", lambda: None) + runtime = dataweave.DataWeave.__new__(dataweave.DataWeave) + runtime._native = runtime_module.NativeRuntime("/tmp/dwlib") + runtime._resolve_module = None + runtime._stream_workers = set() + runtime._stream_workers_lock = Lock() + runtime._cleaning_up = False + runtime._lifecycle_lock = Lock() + runtime.initialize() + return runtime, native + + @pytest.mark.unit def test_run_callback_converts_write_callback_exception_to_abort_result(): native = FakeNative('{"success": false, "error": "callback aborted"}', emit=b"chunk") @@ -459,9 +493,9 @@ def test_precreated_stream_rejects_callback_time_first_consumption_before_worker starts = [] original_register = runtime._register_stream_worker - def record_registration(worker): + def record_registration(worker, operation): registrations.append(worker) - return original_register(worker) + return original_register(worker, operation) class UnexpectedThread: def __init__(self, *_args, **_kwargs): @@ -478,6 +512,53 @@ def __init__(self, *_args, **_kwargs): assert native.attach_count == 0 +@pytest.mark.unit +@pytest.mark.parametrize( + "create_stream", + [ + lambda runtime: runtime.run_streaming("script"), + lambda runtime: runtime.run_transform("script", [b"null"]), + ], + ids=["run-streaming", "run-transform"], +) +@pytest.mark.parametrize("reuse_handle", [False, True], ids=["distinct-handle", "reused-handle"]) +def test_precreated_stream_rejects_stale_generation_before_registration_attach_or_callback( + monkeypatch, create_stream, reuse_handle, +): + runtime, native = initialized_runtime(monkeypatch, reuse_handle=reuse_handle) + stream = create_stream(runtime) + old_handle = runtime._native.handle + runtime.cleanup() + runtime.initialize() + replacement_handle = runtime._native.handle + if reuse_handle: + assert replacement_handle == old_handle + else: + assert replacement_handle != old_handle + registered = [] + original_register = runtime._register_stream_worker + + def record_registration(*args): + result = original_register(*args) + registered.append(args[0]) + return result + + monkeypatch.setattr(runtime, "_register_stream_worker", record_registration) + attach_count = native.attach_count + native.callback_handles.clear() + try: + with pytest.raises(dataweave.DataWeaveError, match="stale engine generation"): + next(stream) + + assert registered == [] + assert runtime._stream_workers == set() + assert native.attach_count == attach_count + assert native.callback_handles == [] + finally: + stream.close() + runtime.cleanup() + + @pytest.mark.unit def test_internal_streaming_write_trampoline_contains_base_exception(monkeypatch): recorder = UnraisableRecorder() @@ -784,8 +865,9 @@ def destroy_engine(self, _thread, _handle): cleanup.start() assert native.cleanup_started.wait(timeout=1) + operation = runtime._native._engine_operation with pytest.raises(dataweave.DataWeaveError, match="being cleaned up"): - runtime._register_stream_worker(Thread()) + runtime._register_stream_worker(Thread(), operation) native.release_cleanup.set() cleanup.join(timeout=1) From 256238c2358fe52243fe6b9bbdc3410be56ddb48 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 10:42:18 -0300 Subject: [PATCH 18/48] test(python): update operation token fakes --- native-lib/python/tests/unit/test_facade.py | 24 ++++++++++++++------ native-lib/python/tests/unit/test_runtime.py | 17 +++++++++++--- 2 files changed, 31 insertions(+), 10 deletions(-) diff --git a/native-lib/python/tests/unit/test_facade.py b/native-lib/python/tests/unit/test_facade.py index ee7fbb10..65a73948 100644 --- a/native-lib/python/tests/unit/test_facade.py +++ b/native-lib/python/tests/unit/test_facade.py @@ -13,10 +13,16 @@ class FakeNativeRuntime: def __init__(self): self.initialized = True self.thread = "thread" + self.operation = native._EngineOperation(handle=7, generation=1) self.calls = [] - def run_engine_and_decode(self, *args): - self.calls.append(("run_engine_and_decode", args)) + def capture_operation(self): + return self.operation + + def run_engine_and_decode(self, script, inputs, *, operation): + assert operation is self.operation + assert operation.handle == 7 + self.calls.append(("run_engine_and_decode", script, inputs, operation)) return self._result() @staticmethod @@ -102,10 +108,9 @@ def test_run_uses_engine_execution_regardless_of_resolver(): assert instance._native.calls == [ ( "run_engine_and_decode", - ( - b"payload", - b'{"value": {"content": "MQ==", "mimeType": "application/json", "charset": "utf-8"}}', - ), + b"payload", + b'{"value": {"content": "MQ==", "mimeType": "application/json", "charset": "utf-8"}}', + instance._native.operation, ) ] @@ -120,7 +125,12 @@ def test_run_without_resolver_routes_through_engine(): True, "SGVsbG8=", None, False, "text/plain", "utf-8" ) assert instance._native.calls == [ - ("run_engine_and_decode", (b"payload", b"{}")) + ( + "run_engine_and_decode", + b"payload", + b"{}", + instance._native.operation, + ) ] diff --git a/native-lib/python/tests/unit/test_runtime.py b/native-lib/python/tests/unit/test_runtime.py index b32ad3a0..4b46c70a 100644 --- a/native-lib/python/tests/unit/test_runtime.py +++ b/native-lib/python/tests/unit/test_runtime.py @@ -10,6 +10,7 @@ def __init__(self, lib_path=None): self.initialized = False self.handle = 0 self.thread = object() + self.operation = None self.has_callback_streaming = True self.has_callback_input_output = True self.cleaned = 0 @@ -24,14 +25,22 @@ def install_resolver(self, resolver): def initialize(self): self.initialized = True self.handle = 7 + self.operation = native._EngineOperation(handle=self.handle, generation=1) - def run_engine_and_decode(self, script, inputs): - self.runs.append((script, inputs)) + def capture_operation(self): + assert self.initialized + return self.operation + + def run_engine_and_decode(self, script, inputs, *, operation): + assert operation is self.operation + assert operation.handle == self.handle + self.runs.append((script, inputs, operation)) return '{"success":true,"result":"","binary":false,"mimeType":"application/json","charset":"UTF-8"}' def cleanup(self): self.cleaned += 1 self.initialized = False + self.operation = None @pytest.mark.unit @@ -65,5 +74,7 @@ def test_run_routes_through_engine(monkeypatch): dw = DataWeave() dw.initialize() dw.run("1 + 1") - assert dw._native.runs == [(b"1 + 1", b"{}")] + operation = dw._native.operation + assert dw._native.runs == [(b"1 + 1", b"{}", operation)] + assert operation == native._EngineOperation(handle=7, generation=1) dw.cleanup() From 1c84b95f62bfe61171b9c6dbb2ad5eb9606b1458 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 11:15:06 -0300 Subject: [PATCH 19/48] fix(python): wake all operation waiters --- native-lib/python/src/dataweave/native.py | 2 +- native-lib/python/tests/unit/test_native.py | 110 ++++++++++++++++++++ 2 files changed, 111 insertions(+), 1 deletion(-) diff --git a/native-lib/python/src/dataweave/native.py b/native-lib/python/src/dataweave/native.py index 627e6441..380fb1ae 100644 --- a/native-lib/python/src/dataweave/native.py +++ b/native-lib/python/src/dataweave/native.py @@ -660,7 +660,7 @@ def _serialized_native_operation(self, expected: Optional[_EngineOperation] = No finally: self._execution_owner = None self._operation_active = False - self._operation_lock.notify() + self._operation_lock.notify_all() def _validate_operation_locked(self, expected: _EngineOperation) -> None: if self._engine_operation != expected: diff --git a/native-lib/python/tests/unit/test_native.py b/native-lib/python/tests/unit/test_native.py index cf549453..402e8edb 100644 --- a/native-lib/python/tests/unit/test_native.py +++ b/native-lib/python/tests/unit/test_native.py @@ -235,6 +235,116 @@ def run_script(_thread, handle, _script, _inputs): assert destroyed_handles == [old_handle] +@pytest.mark.unit +def test_run_admission_generation_wakes_all_stale_waiters_after_cleanup(monkeypatch): + library = FakeLibrary() + result_buffer = ctypes.create_string_buffer( + b'{"success":true,"result":"","binary":false,"mimeType":"application/json","charset":"UTF-8"}' + ) + active_started = Event() + release_active = Event() + + def run_script(_thread, _handle, _script, _inputs): + active_started.set() + assert release_active.wait(1) + return ctypes.addressof(result_buffer) + + library.run_script_engine = CallableFunction(run_script) + library.free_cstring = CallableFunction(lambda _thread, _ptr: None) + monkeypatch.setattr(native.ctypes, "CDLL", lambda _path: library) + runtime = native.NativeRuntime("/tmp/dwlib") + runtime.initialize() + operation = runtime.capture_operation() + active_errors = [] + cleanup_errors = [] + stale_errors = [] + active = Thread( + target=lambda: _capture_error( + active_errors, + lambda: runtime.run_engine_and_decode(b"active", b"{}", operation=operation), + ) + ) + active.start() + assert active_started.wait(1) + + original_wait = runtime._operation_lock.wait + cleanup_done = Event() + waiting = { + "cleanup-waiter": Event(), + "stale-waiter-0": Event(), + "stale-waiter-1": Event(), + } + + def record_wait(timeout=None): + name = current_thread().name + waiting[name].set() + result = original_wait(timeout) + if name.startswith("stale-waiter"): + runtime._operation_lock.release() + try: + assert cleanup_done.wait(1) + finally: + runtime._operation_lock.acquire() + return result + + monkeypatch.setattr(runtime._operation_lock, "wait", record_wait) + + def cleanup_runtime(): + try: + runtime.cleanup() + finally: + cleanup_done.set() + + cleanup = Thread( + target=lambda: _capture_error(cleanup_errors, cleanup_runtime), + name="cleanup-waiter", + ) + cleanup.start() + assert waiting["cleanup-waiter"].wait(1) + with runtime._operation_lock: + pass + stale_waiters = [ + Thread( + target=lambda: _capture_error( + stale_errors, + lambda: runtime.run_engine_and_decode( + b"stale", b"{}", operation=operation + ), + ), + name=f"stale-waiter-{index}", + ) + for index in range(2) + ] + for waiter in stale_waiters: + waiter.start() + assert waiting[waiter.name].wait(1) + with runtime._operation_lock: + pass + release_active.set() + active.join(1) + cleanup.join(1) + for waiter in stale_waiters: + waiter.join(1) + stranded = [waiter.name for waiter in stale_waiters if waiter.is_alive()] + if stranded: + with runtime._operation_lock: + runtime._operation_lock.notify_all() + for waiter in stale_waiters: + waiter.join(1) + + assert not active.is_alive() + assert not cleanup.is_alive() + assert stranded == [] + assert not any(waiter.is_alive() for waiter in stale_waiters) + assert active_errors == [] + assert cleanup_errors == [] + assert len(stale_errors) == 2 + assert all( + str(error) == "DataWeave operation belongs to a stale engine generation." + for error in stale_errors + ) + + @pytest.mark.unit def test_shared_isolate_is_created_once_and_torn_down_on_last_release(monkeypatch): library = FakeLibrary() From ab903ac6606464354d9354d3291973a8647b2b43 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 12:11:16 -0300 Subject: [PATCH 20/48] fix(node): reject native callback reentrancy --- native-lib/node/src/addon.c | 59 ++++++++++++ native-lib/node/src/ffi.ts | 54 +++++++---- .../fixtures/resolver-reentrancy.cjs | 89 +++++++++++++++++++ .../integration/resolver-reentrancy.test.ts | 43 +++++++++ 4 files changed, 229 insertions(+), 16 deletions(-) create mode 100644 native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs create mode 100644 native-lib/node/tests/integration/resolver-reentrancy.test.ts diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 29544806..c9c26aca 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -4,6 +4,7 @@ #include #include #include +#include #ifndef _WIN32 #include #endif @@ -36,6 +37,7 @@ static void* g_thread = NULL; static int g_initialized = 0; static int g_ref_count = 0; static uv_mutex_t g_mutex; +static uv_key_t g_native_callback_depth; // Guards initialization of the process-global g_mutex. Init() runs once per // Worker environment that loads this addon, but g_mutex is process-global — // re-running uv_mutex_init() on an already-initialized mutex from a second @@ -44,6 +46,46 @@ static uv_mutex_t g_mutex; // body runs exactly once per process regardless of how many Workers load us. static uv_once_t g_mutex_once = UV_ONCE_INIT; +#define CALLBACK_REENTRANCY_CODE "ERR_DATAWEAVE_CALLBACK_REENTRANCY" + +static unsigned native_callback_depth(void) { + return (unsigned)(uintptr_t)uv_key_get(&g_native_callback_depth); +} + +static void native_callback_enter(void) { + uv_key_set(&g_native_callback_depth, + (void*)(uintptr_t)(native_callback_depth() + 1)); +} + +static void native_callback_exit(void) { + unsigned depth = native_callback_depth(); + uv_key_set(&g_native_callback_depth, + depth > 1 ? (void*)(uintptr_t)(depth - 1) : NULL); +} + +static bool native_callback_active(void) { + return native_callback_depth() != 0; +} + +static napi_value throw_callback_reentrancy(napi_env env) { + napi_value message; + napi_value error; + napi_value code; + if (napi_create_string_utf8( + env, + "DataWeave native methods cannot be called from a native callback", + NAPI_AUTO_LENGTH, + &message) != napi_ok || + napi_create_error(env, NULL, message, &error) != napi_ok || + napi_create_string_utf8(env, CALLBACK_REENTRANCY_CODE, NAPI_AUTO_LENGTH, &code) != napi_ok || + napi_set_named_property(env, error, "code", code) != napi_ok || + napi_throw(env, error) != napi_ok) { + napi_throw_error(env, CALLBACK_REENTRANCY_CODE, + "DataWeave native methods cannot be called from a native callback"); + } + return NULL; +} + static graal_create_isolate_fn fn_create_isolate = NULL; static graal_attach_thread_fn fn_attach_thread = NULL; static graal_detach_thread_fn fn_detach_thread = NULL; @@ -903,6 +945,7 @@ static void cleanup_thread_fn(void* arg); static void retry_stranded_teardown_locked(void); static napi_value napi_initialize(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); size_t argc = 1; napi_value argv[1]; // Review #10 #5 (svacas P2): check napi_get_cb_info's status too, not just @@ -1210,7 +1253,9 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v napi_value global; napi_get_global(env, &global); + native_callback_enter(); napi_call_function(env, global, js_callback, 1, &buffer, NULL); + native_callback_exit(); free(chunk->buf); free(chunk); @@ -1341,6 +1386,7 @@ static void streaming_thread_fn(void* arg) { } static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); if (!g_initialized) { napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); return NULL; @@ -1592,7 +1638,9 @@ static void call_js_read(napi_env env, napi_value js_callback, void* context, vo napi_get_global(env, &global); napi_value result; + native_callback_enter(); napi_status status = napi_call_function(env, global, js_callback, 1, &buf_size_val, &result); + native_callback_exit(); if (status == napi_ok && result != NULL) { bool is_buffer; @@ -1758,7 +1806,9 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* napi_value global; napi_get_global(env, &global); + native_callback_enter(); napi_call_function(env, global, js_callback, 1, &buffer, NULL); + native_callback_exit(); free(chunk->buf); free(chunk); @@ -1866,6 +1916,7 @@ static void transform_thread_fn(void* arg) { } static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); if (!g_initialized) { napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); return NULL; @@ -2144,7 +2195,9 @@ static char* resolve_module_callback(void* thread, void* ctx, const char* module napi_value undefined, result; napi_get_undefined(env, &undefined); + native_callback_enter(); napi_status status = napi_call_function(env, undefined, js_callback, 1, &module_path_str, &result); + native_callback_exit(); if (status != napi_ok) { // JS resolver threw — clear the pending exception so it doesn't leak // into the next napi call, extract and log its message/stack for @@ -2234,6 +2287,7 @@ static char* resolve_module_callback(void* thread, void* ctx, const char* module // createEngine() -> number static napi_value napi_create_engine(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); (void)info; if (!fn_create_engine) { napi_throw_error(env, NULL, "create_engine not available in native library"); return NULL; } @@ -2367,6 +2421,7 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { // createEngineWithResolver(resolver) -> number static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); if (!fn_create_engine_with_resolver) { napi_throw_error(env, NULL, "create_engine_with_resolver not available in native library"); return NULL; } size_t argc = 1; napi_value argv[1]; napi_get_cb_info(env, info, &argc, argv, NULL, NULL); @@ -2473,6 +2528,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i // destroyEngine(handle) -> void static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); if (!g_initialized) return NULL; size_t argc = 1; napi_value argv[1]; napi_get_cb_info(env, info, &argc, argv, NULL, NULL); @@ -2610,6 +2666,7 @@ static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { // runScriptEngine(handle, script, inputsJson) -> string static napi_value napi_run_script_engine(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); if (!g_initialized) { napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); return NULL; } if (!fn_run_script_engine) { napi_throw_error(env, NULL, "run_script_engine not available in native library"); return NULL; } size_t argc = 3; napi_value argv[3]; @@ -3362,6 +3419,7 @@ static napi_value release_isolate_ref_locked(napi_env env) { } static napi_value napi_cleanup(napi_env env, napi_callback_info info) { + if (native_callback_active()) return throw_callback_reentrancy(env); (void)info; uv_mutex_lock(&g_mutex); return release_isolate_ref_locked(env); // unlocks g_mutex, returns the promise @@ -3372,6 +3430,7 @@ static napi_value napi_cleanup(napi_env env, napi_callback_info info) { static void init_g_mutex(void) { uv_mutex_init(&g_mutex); uv_cond_init(&g_teardown_cond); + uv_key_create(&g_native_callback_depth); } // --- Test-only N-API entrypoints (review #12 #3 / #13) --- diff --git a/native-lib/node/src/ffi.ts b/native-lib/node/src/ffi.ts index 30c3e85b..21e0e287 100644 --- a/native-lib/node/src/ffi.ts +++ b/native-lib/node/src/ffi.ts @@ -1,4 +1,5 @@ import { resolveAddonPath } from "./addon-path"; +import { DataWeaveError } from "./errors"; import type { ModuleResolver } from "./resolver"; interface NativeAddon { @@ -28,6 +29,23 @@ interface NativeAddon { let addon: NativeAddon | null = null; +function callNative(invoke: () => T): T { + try { + return invoke(); + } catch (error) { + if ( + error && + typeof error === "object" && + "code" in error && + error.code === "ERR_DATAWEAVE_CALLBACK_REENTRANCY" + ) { + const message = "message" in error ? String(error.message) : String(error); + throw new DataWeaveError(message); + } + throw error; + } +} + function getAddon(addonPath?: string): NativeAddon { if (!addon) { addon = require(addonPath ?? resolveAddonPath()) as NativeAddon; @@ -36,23 +54,23 @@ function getAddon(addonPath?: string): NativeAddon { } export function initialize(libPath: string, addonPath?: string): void { - getAddon(addonPath).initialize(libPath); + callNative(() => getAddon(addonPath).initialize(libPath)); } export function createEngine(): number { - return getAddon().createEngine(); + return callNative(() => getAddon().createEngine()); } export function createEngineWithResolver(resolver: ModuleResolver): number { - return getAddon().createEngineWithResolver(resolver); + return callNative(() => getAddon().createEngineWithResolver(resolver)); } export function destroyEngine(handle: number): void { - getAddon().destroyEngine(handle); + callNative(() => getAddon().destroyEngine(handle)); } export function runScriptEngine(handle: number, script: string, inputsJson: string): string { - return getAddon().runScriptEngine(handle, script, inputsJson); + return callNative(() => getAddon().runScriptEngine(handle, script, inputsJson)); } export function runScriptStreamingEngine( @@ -61,7 +79,9 @@ export function runScriptStreamingEngine( inputsJson: string, chunkCb: (chunk: Buffer) => void ): Promise { - return getAddon().runScriptStreamingEngine(handle, script, inputsJson, chunkCb); + return callNative(() => + getAddon().runScriptStreamingEngine(handle, script, inputsJson, chunkCb) + ); } export function runScriptTransformEngine( @@ -74,18 +94,20 @@ export function runScriptTransformEngine( readCb: (bufSize: number) => Buffer | null, writeCb: (chunk: Buffer) => void ): Promise { - return getAddon().runScriptTransformEngine( - handle, - script, - inputsJson, - inputName, - inputMimeType, - inputCharset, - readCb, - writeCb + return callNative(() => + getAddon().runScriptTransformEngine( + handle, + script, + inputsJson, + inputName, + inputMimeType, + inputCharset, + readCb, + writeCb + ) ); } export function cleanup(): Promise { - return getAddon().cleanup(); + return callNative(() => getAddon().cleanup()); } diff --git a/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs b/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs new file mode 100644 index 00000000..01ca074c --- /dev/null +++ b/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs @@ -0,0 +1,89 @@ +const path = require("node:path"); + +const ROOT = path.join(__dirname, "..", "..", ".."); +const MODULE_SOURCE = "%dw 2.0\nfun answer() = 42"; +const OUTER_SCRIPT = [ + "%dw 2.0", + "import org::test::reentrant", + "output application/json", + "---", + "reentrant::answer()", +].join("\n"); + +async function runFacade() { + const { DataWeave } = require(path.join(ROOT, "dist", "index.js")); + const inner = new DataWeave(); + let nestedErrorName = null; + const outer = new DataWeave({ + resolveModule: () => { + try { + inner.run("21 * 2"); + } catch (error) { + nestedErrorName = error && error.name; + } + return MODULE_SOURCE; + }, + }); + + try { + inner.initialize(); + outer.initialize(); + const result = outer.run(OUTER_SCRIPT); + console.log(JSON.stringify({ nestedErrorName, outerResult: result.getString() })); + } finally { + try { + await outer.cleanup(); + } finally { + await inner.cleanup(); + } + } +} + +async function runRaw() { + const addon = require(path.join(ROOT, "build", "Release", "dwlib_addon.node")); + const { findLibrary } = require(path.join(ROOT, "dist", "utils.js")); + let innerHandle = null; + let outerHandle = null; + let nestedErrorCode = null; + + addon.initialize(findLibrary()); + try { + innerHandle = addon.createEngine(); + outerHandle = addon.createEngineWithResolver(() => { + try { + addon.runScriptEngine(innerHandle, "21 * 2", "{}"); + } catch (error) { + nestedErrorCode = error && error.code; + } + return MODULE_SOURCE; + }); + const raw = addon.runScriptEngine(outerHandle, OUTER_SCRIPT, "{}"); + const result = JSON.parse(raw); + console.log(JSON.stringify({ + nestedErrorCode, + outerResult: Buffer.from(result.result, "base64").toString("utf-8"), + })); + } finally { + try { + if (outerHandle !== null) addon.destroyEngine(outerHandle); + } finally { + try { + if (innerHandle !== null) addon.destroyEngine(innerHandle); + } finally { + await addon.cleanup(); + } + } + } +} + +const mode = process.argv[2]; +const run = mode === "facade" ? runFacade : mode === "raw" ? runRaw : null; +if (run === null) { + console.error(`unknown mode: ${mode}`); + process.exit(2); +} + +run().catch((error) => { + console.error(error && error.stack ? error.stack : error); + process.exitCode = 99; +}); diff --git a/native-lib/node/tests/integration/resolver-reentrancy.test.ts b/native-lib/node/tests/integration/resolver-reentrancy.test.ts new file mode 100644 index 00000000..f9aecf2c --- /dev/null +++ b/native-lib/node/tests/integration/resolver-reentrancy.test.ts @@ -0,0 +1,43 @@ +import { spawnSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; + +const FIXTURE = join(__dirname, "fixtures", "resolver-reentrancy.cjs"); +const DIST_ENTRY = join(__dirname, "..", "..", "dist", "index.js"); +const ADDON_PATH = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); +const FATAL_GRAAL_ERROR = /Fatal error|Must either be at a safepoint or in native mode/i; + +function runFixture(mode: "facade" | "raw") { + const child = spawnSync(process.execPath, [FIXTURE, mode], { + encoding: "utf-8", + timeout: 30_000, + }); + + expect(child.error, child.error?.message).toBeUndefined(); + expect(child.signal, child.stderr).toBeNull(); + expect(child.status, child.stderr).toBe(0); + expect(child.stderr).not.toMatch(FATAL_GRAAL_ERROR); + + return JSON.parse(child.stdout.trim()) as Record; +} + +describe("resolver callback reentrancy guard", () => { + it("maps nested public DataWeave calls to DataWeaveError and preserves the outer run", () => { + expect(existsSync(DIST_ENTRY), `built entry missing at ${DIST_ENTRY} - run \`npm run build:ts\``).toBe(true); + + expect(runFixture("facade")).toEqual({ + nestedErrorName: "DataWeaveError", + outerResult: "42", + }); + }); + + it("exposes a stable raw-addon error code for nested native admission", () => { + expect(existsSync(ADDON_PATH), `native addon missing at ${ADDON_PATH} - run \`npm run build:addon\``).toBe(true); + + expect(runFixture("raw")).toEqual({ + nestedErrorCode: "ERR_DATAWEAVE_CALLBACK_REENTRANCY", + outerResult: "42", + }); + }); +}); From c3eb13ff6c1777275442ca4c346a785fc8dcd20f Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 12:12:50 -0300 Subject: [PATCH 21/48] test(node): update callback admission regressions --- .../admission-during-teardown.test.ts | 104 +++++++----------- .../tests/integration/run-admission.test.ts | 70 ++++++------ .../integration/teardown-deadlock.test.ts | 43 ++------ 3 files changed, 84 insertions(+), 133 deletions(-) diff --git a/native-lib/node/tests/integration/admission-during-teardown.test.ts b/native-lib/node/tests/integration/admission-during-teardown.test.ts index a87b7f18..6b256020 100644 --- a/native-lib/node/tests/integration/admission-during-teardown.test.ts +++ b/native-lib/node/tests/integration/admission-during-teardown.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from "vitest"; import * as ffi from "../../src/ffi"; +import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; // Round-6 finding #2: napi_run_script_streaming_engine/napi_run_script_transform_engine @@ -45,57 +46,36 @@ import { findLibrary, buildInputsJson } from "../../src/utils"; // mirroring the round-5 teardown-deadlock test's use of a synchronous native // read-callback to force deterministic ordering instead of timers. // -// Real addon, no mocking. -describe("admission rejected while teardown pending (round 6 #2)", () => { - it("a streaming op started on the same handle during pending teardown is rejected, not admitted", async () => { +// Task 6 supersedes this test's callback-based pending-teardown trigger. While +// native code invokes a transform read callback, all isolate-touching methods +// are rejected before lifecycle mutation or worker admission. Use cleanup and +// streaming start together to cover both guards and the TypeScript mapping. +describe("transform read callback admission guard", () => { + it("rejects cleanup and streaming start before native admission", async () => { ffi.initialize(findLibrary()); const handle = ffi.createEngine(); - let cleanupPromise: Promise | undefined; + let cleanupErr: unknown; let admitErr: unknown; let admitted = false; - let secondOpSettled: Promise = Promise.resolve(); let firstRead = true; const readCb = (_bufSize: number): Buffer | null => { if (firstRead) { firstRead = false; - - // Trigger Case 5 of napi_cleanup: last release of the shared library - // ref-count while this transform's worker is attached and - // g_active_ops > 0. Synchronously sets g_teardown_state = - // TEARDOWN_PENDING_WAIT before returning. Not awaited -- the point is - // to observe the state it leaves behind, not its eventual settlement. - cleanupPromise = ffi.cleanup(); - - // Attempt a second admission on the SAME still-live handle/isolate - // while teardown is pending. Fixed code rejects admission with a - // synchronous napi_throw_error (the atomic admission check sees - // g_teardown_state != TEARDOWN_NONE, before any promise is even - // created). Pre-fix code admits it: the unlocked g_initialized check - // passes (the isolate genuinely hasn't been torn down yet -- - // TEARDOWN_PENDING_WAIT hasn't reached physical teardown) and - // g_active_ops is reserved without ever consulting g_teardown_state, - // so the call returns a promise that goes on to resolve successfully. - // - // On rejection, napi_throw_error fires synchronously from this very - // call (admission fails before any promise is created), so it must - // be caught here rather than only via a rejected-promise `.then` -- - // mirroring the round-5 teardown-deadlock test's care not to let a - // thrown exception escape a native read-callback body (it would be - // reinterpreted as a read error, masking the real outcome). try { - secondOpSettled = ffi - .runScriptStreamingEngine( - handle, - "%dw 2.0\noutput application/json\n---\n[1,2,3]", - buildInputsJson({}), - () => {} - ) - .then( - () => { admitted = true; }, - (e) => { admitErr = e; } - ); + ffi.cleanup(); + } catch (e) { + cleanupErr = e; + } + try { + ffi.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[1,2,3]", + buildInputsJson({}), + () => {} + ); + admitted = true; } catch (e) { admitErr = e; } @@ -108,29 +88,25 @@ describe("admission rejected while teardown pending (round 6 #2)", () => { const chunks: Buffer[] = []; const writeCb = (chunk: Buffer) => { chunks.push(chunk); }; - const resultRaw = await ffi.runScriptTransformEngine( - handle, - "output application/json\n---\npayload", - "{}", - "payload", - "application/json", - null, - readCb, - writeCb - ); - const result = JSON.parse(resultRaw); - expect(result.success).toBe(true); - - // Let the second op settle (whichever branch it took) before asserting, - // and drain the pending teardown so the shared native isolate is left in - // a clean, consistent state for sibling test files in this process. - await secondOpSettled; - await cleanupPromise; - - // The second op admitted while teardown was pending must have been - // rejected, not silently admitted against an isolate a concurrent - // teardown could tear down out from under it. - expect(admitErr).toBeTruthy(); - expect(admitted).toBe(false); + try { + const resultRaw = await ffi.runScriptTransformEngine( + handle, + "output application/json\n---\npayload", + "{}", + "payload", + "application/json", + null, + readCb, + writeCb + ); + const result = JSON.parse(resultRaw); + expect(result.success).toBe(true); + expect(cleanupErr).toBeInstanceOf(DataWeaveError); + expect(admitErr).toBeInstanceOf(DataWeaveError); + expect(admitted).toBe(false); + } finally { + ffi.destroyEngine(handle); + await ffi.cleanup(); + } }, 20000); }); diff --git a/native-lib/node/tests/integration/run-admission.test.ts b/native-lib/node/tests/integration/run-admission.test.ts index 88dea6a2..8a6c1cf5 100644 --- a/native-lib/node/tests/integration/run-admission.test.ts +++ b/native-lib/node/tests/integration/run-admission.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from "vitest"; import * as ffi from "../../src/ffi"; +import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; // Round-7 finding #1: the synchronous napi_run_script_engine touched the @@ -28,17 +29,17 @@ import { findLibrary, buildInputsJson } from "../../src/utils"; // returned; the immediately-following ffi.runScriptEngine re-enters native code // synchronously on the same callstack and deterministically observes it. // -// Real addon, no mocking. -describe("run() admission rejected while teardown pending (round 7 #1)", () => { - it("a synchronous run() started during pending teardown throws, not attach to a dead isolate", async () => { +// Task 6 supersedes this test's old callback-based pending-teardown trigger: +// lifecycle and execution entry from a native callback are now rejected before +// cleanup can queue teardown or run can attach to the isolate. Drive the real +// addon through ffi so this also verifies TypeScript error normalization for a +// transform read callback, while the successful outer transform proves callback +// depth is restored afterward. +describe("transform read callback reentrancy guard", () => { + it("rejects cleanup and run before native admission and preserves the outer transform", async () => { ffi.initialize(findLibrary()); const handle = ffi.createEngine(); - - // Keep one op in flight so the ref release becomes Case 5 (pending - // teardown) rather than Case 4 (immediate teardown): use a transform whose - // read callback triggers cleanup() and then attempts a run() on the same - // handle, all on the same synchronous callstack. - let cleanupPromise: Promise | undefined; + let cleanupErr: unknown; let runErr: unknown; let ran = false; @@ -46,14 +47,11 @@ describe("run() admission rejected while teardown pending (round 7 #1)", () => { const readCb = (_bufSize: number): Buffer | null => { if (firstRead) { firstRead = false; - // Case 5: last ref release with g_active_ops > 0 -> TEARDOWN_PENDING_WAIT, - // set synchronously before this returns. Not awaited. - cleanupPromise = ffi.cleanup(); - // Synchronous run() on the same still-live handle while teardown is - // pending. Fixed code rejects admission with a synchronous throw - // (g_teardown_state != TEARDOWN_NONE). Must be caught here -- it is a - // synchronous throw, not a rejected promise. Do not let it escape the - // native read-callback body. + try { + ffi.cleanup(); + } catch (e) { + cleanupErr = e; + } try { ffi.runScriptEngine( handle, @@ -71,23 +69,25 @@ describe("run() admission rejected while teardown pending (round 7 #1)", () => { const writeCb = (_chunk: Buffer) => {}; - const resultRaw = await ffi.runScriptTransformEngine( - handle, - "output application/json\n---\npayload", - "{}", - "payload", - "application/json", - null, - readCb, - writeCb - ); - const result = JSON.parse(resultRaw); - expect(result.success).toBe(true); - - await cleanupPromise; - - // run() started while teardown was pending must have been rejected. - expect(runErr).toBeTruthy(); - expect(ran).toBe(false); + try { + const resultRaw = await ffi.runScriptTransformEngine( + handle, + "output application/json\n---\npayload", + "{}", + "payload", + "application/json", + null, + readCb, + writeCb + ); + const result = JSON.parse(resultRaw); + expect(result.success).toBe(true); + expect(cleanupErr).toBeInstanceOf(DataWeaveError); + expect(runErr).toBeInstanceOf(DataWeaveError); + expect(ran).toBe(false); + } finally { + ffi.destroyEngine(handle); + await ffi.cleanup(); + } }, 20000); }); diff --git a/native-lib/node/tests/integration/teardown-deadlock.test.ts b/native-lib/node/tests/integration/teardown-deadlock.test.ts index efa44cec..642bb4ef 100644 --- a/native-lib/node/tests/integration/teardown-deadlock.test.ts +++ b/native-lib/node/tests/integration/teardown-deadlock.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from "vitest"; import { run, runTransform, cleanup } from "../../src/dataweave"; +import { DataWeaveError } from "../../src/errors"; // Regression test for W-23692110 round 5 (Task 1 fix in native-lib/node/src/addon.c). // @@ -37,20 +38,15 @@ import { run, runTransform, cleanup } from "../../src/dataweave"; // guarantees the ordering "worker attached and mid-read" -> "cleanup() // fired" -> "run() fired", all on the JS thread, before the generator call // returns and the worker can proceed. -describe("re-init during pending teardown (W-23692110, round 5 P1)", () => { - // On the UNFIXED addon.c this deadlocks for real: the JS thread never - // returns from run()'s napi_initialize (blocked waiting for g_active_ops to - // drain), so the background transform worker -- itself blocked waiting for - // the JS thread to service its read callback -- can never proceed either. - // Vitest kills the test at the timeout below, a bounded/deterministic red. - // On the fixed code, napi_initialize adopts the still-live isolate and - // run() returns promptly, letting everything drain normally. +// Task 6 replaces the old callback-triggered pending-teardown scenario with a +// stronger contract: public DataWeave execution is rejected while native code +// is invoking the transform input callback. The real-addon test still proves +// the worker and outer transform drain without a deadlock after that rejection. +describe("public API transform callback reentrancy guard", () => { it( - "module-level cleanup() during an active transform read does not deadlock a concurrent run()", + "rejects a nested module-level run and lets the outer transform drain", async () => { let fired = false; - let cleanupPromise: Promise | undefined; - let runResult: ReturnType | undefined; let runError: unknown; // Large enough that, at the moment of the very first read pull, the @@ -64,20 +60,8 @@ describe("re-init during pending teardown (W-23692110, round 5 P1)", () => { for (let i = 0; i < totalReads; i++) { if (!fired) { fired = true; - // We are executing synchronously inside the native read - // callback (call_js_read in addon.c), on the JS thread, while - // the background transform worker thread is blocked inside - // transform_read_cb waiting for this exact call to return. - // Deliberately do NOT await cleanup() here, and do NOT let an - // assertion throw from inside this generator -- a thrown - // exception here would be caught by the native read-callback - // wrapper and reinterpreted as a read error, silently masking a - // real assertion failure instead of surfacing it as a test - // failure. Capture results and assert on them after the - // generator (and the transform) have fully drained. - cleanupPromise = cleanup(); try { - runResult = run('%dw 2.0\noutput application/json\n---\n1 + 1'); + run('%dw 2.0\noutput application/json\n---\n1 + 1'); } catch (e) { runError = e; } @@ -101,18 +85,9 @@ describe("re-init during pending teardown (W-23692110, round 5 P1)", () => { } expect(fired).toBe(true); - expect(runError).toBeUndefined(); - expect(runResult?.success).toBe(true); - expect(JSON.parse(runResult!.getString()!)).toBe(2); + expect(runError).toBeInstanceOf(DataWeaveError); expect(result.value.success).toBe(true); - // Let both the deferred teardown/cleanup and this test settle cleanly. - // This is essential: the process shares one native isolate across all - // integration test files, so leaving an unresolved cleanup here would - // perturb sibling test files. - await cleanupPromise; - // Idempotent final cleanup: a no-op if the singleton is already fully - // released, leaving the module in a clean state for subsequent tests. await cleanup(); }, 20000 From 3030a1e16ada77677eddf76a22a3504fac2b7b09 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 12:13:55 -0300 Subject: [PATCH 22/48] test(node): clarify callback admission coverage --- .../admission-during-teardown.test.ts | 43 ------------------- .../tests/integration/run-admission.test.ts | 26 ----------- .../integration/teardown-deadlock.test.ts | 36 ---------------- 3 files changed, 105 deletions(-) diff --git a/native-lib/node/tests/integration/admission-during-teardown.test.ts b/native-lib/node/tests/integration/admission-during-teardown.test.ts index 6b256020..4a6dc74a 100644 --- a/native-lib/node/tests/integration/admission-during-teardown.test.ts +++ b/native-lib/node/tests/integration/admission-during-teardown.test.ts @@ -3,49 +3,6 @@ import * as ffi from "../../src/ffi"; import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; -// Round-6 finding #2: napi_run_script_streaming_engine/napi_run_script_transform_engine -// used to read g_initialized outside g_mutex, then reserve g_active_ops in a -// LATER, separate critical section right before spawning the worker thread -- -// with no reference to g_teardown_state at all. The fix folds the lifecycle -// check (including g_teardown_state) and the g_active_ops reservation into one -// atomic critical section, before any work/tsfn/promise/bridge is allocated, -// and rejects admission once a teardown is queued/underway -// (g_teardown_state != TEARDOWN_NONE), not just when the isolate is fully gone. -// -// Why this test drives the addon through the raw `ffi` module instead of the -// module-level `run`/`runStreaming`/`runTransform`/`cleanup` singleton (as the -// original brief sketch does): the module-level `cleanup()` nulls the -// singleton, so a later module-level `runStreaming()`/`runTransform()` call -// re-creates a fresh `DataWeave` instance and calls `initialize()` again. -// `napi_initialize`'s TEARDOWN_PENDING_WAIT branch (round-5's deadlock fix) -// treats that as a legitimate ADOPTION of the still-live isolate: it sets -// g_teardown_cancelled = true and cancels the pending teardown *before* the -// second op's admission check ever runs -- so by the time streaming/transform -// admission is checked, g_teardown_state is already back to TEARDOWN_NONE -// (verified empirically while developing this test: the brief's literal shape -// resolves the second op cleanly on both pre-fix and post-fix code, so it -// cannot distinguish them -- it never reaches the vulnerable window because -// the intervening initialize() call cancels the teardown as a side effect). -// -// To actually observe admission-during-pending-teardown, the second op must -// run against the SAME still-live handle/isolate WITHOUT any intervening -// ffi.initialize() call. Calling `ffi.cleanup()` directly (skipping -// `destroyEngine`) triggers exactly napi_cleanup's Case 5 (last ref release -// with an active op) and sets g_teardown_state = TEARDOWN_PENDING_WAIT -// synchronously, under g_mutex, before napi_cleanup returns its Promise to -// JS -- with no adoption path involved, since nothing calls initialize() -// afterward. -// -// Determinism: `ffi.cleanup()`'s synchronous prefix (native napi_cleanup body) -// runs entirely synchronously up to the point where it returns a Promise; the -// TEARDOWN_PENDING_WAIT transition happens on that same synchronous call, not -// after an await. The immediately-following `ffi.runScriptStreamingEngine` -// call re-enters native code synchronously (it's a plain N-API call), on the -// very same JS callstack, so it deterministically observes -// g_teardown_state == TEARDOWN_PENDING_WAIT with no timing assumptions -- -// mirroring the round-5 teardown-deadlock test's use of a synchronous native -// read-callback to force deterministic ordering instead of timers. -// // Task 6 supersedes this test's callback-based pending-teardown trigger. While // native code invokes a transform read callback, all isolate-touching methods // are rejected before lifecycle mutation or worker admission. Use cleanup and diff --git a/native-lib/node/tests/integration/run-admission.test.ts b/native-lib/node/tests/integration/run-admission.test.ts index 8a6c1cf5..5b851e43 100644 --- a/native-lib/node/tests/integration/run-admission.test.ts +++ b/native-lib/node/tests/integration/run-admission.test.ts @@ -3,32 +3,6 @@ import * as ffi from "../../src/ffi"; import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; -// Round-7 finding #1: the synchronous napi_run_script_engine touched the -// isolate (fn_attach_thread -> fn_run_script_engine -> fn_detach_thread) with -// only a top-of-function !g_initialized fast-path and NO g_active_ops -// reservation under g_mutex. A second Worker's last cleanup() (napi_cleanup -// Case 4) could observe g_active_ops == 0 and tear down g_isolate while this -// op was attaching/executing -- a use-after-free. -// -// The genuine cross-Worker TOCTOU is not reliably forceable from single-thread -// JS (same limitation the round-6 #2 admission-during-teardown test documents: -// re-init would trigger the adoption path and cancel the pending teardown -// before the admission check runs). What we CAN assert deterministically is -// the admission-rejection path the fix introduces: once a teardown is pending -// (g_teardown_state != TEARDOWN_NONE), a freshly started run() is rejected with -// a synchronous throw rather than attaching to an isolate a concurrent teardown -// could pull out from under it. The C-level reasoning -- check-and-reserve is -// now one atomic critical section on the run() path -- is what covers the race -// itself. -// -// We drive the addon through the raw `ffi` module (not the module-level -// singleton) so the second op runs against the SAME still-live handle/isolate -// with no intervening ffi.initialize() call to trigger adoption. Calling -// ffi.cleanup() directly triggers napi_cleanup Case 5 and sets -// g_teardown_state = TEARDOWN_PENDING_WAIT synchronously, before its Promise is -// returned; the immediately-following ffi.runScriptEngine re-enters native code -// synchronously on the same callstack and deterministically observes it. -// // Task 6 supersedes this test's old callback-based pending-teardown trigger: // lifecycle and execution entry from a native callback are now rejected before // cleanup can queue teardown or run can attach to the isolate. Drive the real diff --git a/native-lib/node/tests/integration/teardown-deadlock.test.ts b/native-lib/node/tests/integration/teardown-deadlock.test.ts index 642bb4ef..de0bf82b 100644 --- a/native-lib/node/tests/integration/teardown-deadlock.test.ts +++ b/native-lib/node/tests/integration/teardown-deadlock.test.ts @@ -2,42 +2,6 @@ import { describe, it, expect } from "vitest"; import { run, runTransform, cleanup } from "../../src/dataweave"; import { DataWeaveError } from "../../src/errors"; -// Regression test for W-23692110 round 5 (Task 1 fix in native-lib/node/src/addon.c). -// -// Bug: napi_initialize used to block the JS thread forever whenever it ran -// while a teardown was pending on the shared native isolate and a -// streaming/transform op was still active elsewhere -- because draining that -// active op can need the very same JS thread napi_initialize was blocking. -// The fix makes napi_initialize adopt the still-live isolate instead of -// waiting, in the window before the teardown waiter thread commits to -// physical teardown. -// -// This loads the REAL native addon (no `vi.mock` of ffi) -- the deadlock is -// entirely in C and cannot be reproduced at the mocked-ffi layer. -// -// Why runTransform (not runStreaming) drives this repro: runStreaming's -// output-chunk delivery uses an unbounded napi_threadsafe_function queue, and -// g_active_ops is decremented on the background worker thread right after it -// detaches from the isolate -- independent of whether the JS event loop ever -// turns. So a blocked JS thread does NOT stop a runStreaming() op from -// draining; there is no genuine circular wait on that path (verified -// empirically: the brief's originally-suggested runStreaming shape resolves -// promptly even against pre-Task-1 addon.c, because an earlier round already -// moved that decrement off the JS thread -- see commit ac8d520). -// -// runTransform's INPUT side is different: transform_read_cb (addon.c) calls -// napi_call_threadsafe_function(w->read_tsfn, &req, napi_tsfn_blocking) and -// then genuinely blocks the background worker thread on a condition variable -// until call_js_read runs on the JS thread and signals it. That JS-thread -// callback synchronously invokes our JS read callback (a plain -// Iterable consumed by a sync generator) via napi_call_function -- -// so firing cleanup() and a concurrent run() from *inside* that generator -// deterministically executes them while the background worker is attached -// and blocked waiting for this exact call to return. No timing assumptions -// (no setTimeout/microtask races) are needed: the call graph itself -// guarantees the ordering "worker attached and mid-read" -> "cleanup() -// fired" -> "run() fired", all on the JS thread, before the generator call -// returns and the worker can proceed. // Task 6 replaces the old callback-triggered pending-teardown scenario with a // stronger contract: public DataWeave execution is rejected while native code // is invoking the transform input callback. The real-addon test still proves From ad370a32efba71fd45fbcc7eb80e386a4f226aba Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 12:52:33 -0300 Subject: [PATCH 23/48] test(node): preserve lifecycle regression coverage --- native-lib/node/src/addon.c | 81 ++++++++++++++- .../admission-during-teardown.test.ts | 89 ++++++++++++++++- .../fixtures/resolver-reentrancy.cjs | 76 +++++++++++++- .../integration/resolver-reentrancy.test.ts | 18 +++- .../tests/integration/run-admission.test.ts | 85 +++++++++++++++- .../integration/teardown-deadlock.test.ts | 98 ++++++++++++++----- 6 files changed, 412 insertions(+), 35 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index c9c26aca..d9e9ffd7 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -38,6 +38,7 @@ static int g_initialized = 0; static int g_ref_count = 0; static uv_mutex_t g_mutex; static uv_key_t g_native_callback_depth; +static int g_native_callback_depth_status; // Guards initialization of the process-global g_mutex. Init() runs once per // Worker environment that loads this addon, but g_mutex is process-global — // re-running uv_mutex_init() on an already-initialized mutex from a second @@ -190,10 +191,13 @@ static engine_bridge_t* g_stranded_bridges = NULL; // linked list, guarded by g // post-fix it is kept by its owner-env cleanup hook and the ref is deleted on the // owner thread at env teardown (g_test_resolver_ref_deletes counts those deletes). // g_test_hooks is written once in Init before any reader runs; g_test_force_strand_once -// and g_test_resolver_ref_deletes are accessed only under g_mutex. +// and the remaining test state are accessed only under g_mutex. static bool g_test_hooks = false; static bool g_test_force_strand_once = false; static long long g_test_resolver_ref_deletes = 0; +static bool g_test_hold_next_async_op = false; +static bool g_test_async_op_held = false; +static bool g_test_release_async_op = false; // One record per napi_env that has ever taken an init reference (via // initialize()). init_refs is that env's net initialize()-minus-cleanup() @@ -289,6 +293,28 @@ static bool g_teardown_needed = false; static bool g_isolate_poisoned = false; static uv_cond_t g_teardown_cond; +// Test-only one-shot gate for a real streaming/transform worker. Admission has +// already reserved g_active_ops before the worker reaches this point, so holding +// it here lets tests drive cleanup() into TEARDOWN_PENDING_WAIT without entering +// through a native JS callback. Inert unless DATAWEAVE_TEST_HOOKS is enabled and +// __test_holdNextAsyncOp() armed the gate. +static void test_hold_async_op_if_armed(void) { + if (!g_test_hooks) return; + uv_mutex_lock(&g_mutex); + if (g_test_hold_next_async_op) { + g_test_hold_next_async_op = false; + g_test_async_op_held = true; + uv_cond_broadcast(&g_teardown_cond); + while (!g_test_release_async_op) { + uv_cond_wait(&g_teardown_cond, &g_mutex); + } + g_test_release_async_op = false; + g_test_async_op_held = false; + uv_cond_broadcast(&g_teardown_cond); + } + uv_mutex_unlock(&g_mutex); +} + // teardown+detach double failure (review #17 #1): an exiting worker is stuck // attached to this isolate, so graal_tear_down_isolate can never again get the // sole-attached, current-OS-thread IsolateThread it requires -- retrying is @@ -1285,6 +1311,7 @@ static int streaming_write_cb(void* ctx, const char* buf, int len) { static void streaming_thread_fn(void* arg) { struct streaming_work* w = (struct streaming_work*)arg; + test_hold_async_op_if_armed(); void* worker_thread = NULL; int rc = fn_attach_thread(g_isolate, &worker_thread); @@ -1816,6 +1843,7 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* static void transform_thread_fn(void* arg) { struct transform_work* w = (struct transform_work*)arg; + test_hold_async_op_if_armed(); void* worker_thread = NULL; int rc = fn_attach_thread(g_isolate, &worker_thread); @@ -3430,7 +3458,7 @@ static napi_value napi_cleanup(napi_env env, napi_callback_info info) { static void init_g_mutex(void) { uv_mutex_init(&g_mutex); uv_cond_init(&g_teardown_cond); - uv_key_create(&g_native_callback_depth); + g_native_callback_depth_status = uv_key_create(&g_native_callback_depth); } // --- Test-only N-API entrypoints (review #12 #3 / #13) --- @@ -3466,9 +3494,52 @@ static napi_value napi_test_resolver_ref_delete_count(napi_env env, napi_callbac return out; } +static napi_value napi_test_hold_next_async_op(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + if (g_test_hold_next_async_op || g_test_async_op_held) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "A test async operation gate is already armed"); + return NULL; + } + g_test_hold_next_async_op = true; + g_test_release_async_op = false; + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_async_op_held(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + bool held = g_test_async_op_held; + uv_mutex_unlock(&g_mutex); + napi_value out; + napi_get_boolean(env, held, &out); + return out; +} + +static napi_value napi_test_release_async_op(napi_env env, napi_callback_info info) { + (void)env; + (void)info; + uv_mutex_lock(&g_mutex); + g_test_release_async_op = true; + uv_cond_broadcast(&g_teardown_cond); + uv_mutex_unlock(&g_mutex); + return NULL; +} + static napi_value Init(napi_env env, napi_value exports) { uv_once(&g_mutex_once, init_g_mutex); + if (g_native_callback_depth_status != 0) { + char message[128]; + snprintf(message, sizeof(message), + "Failed to initialize native callback state (libuv error %d)", + g_native_callback_depth_status); + napi_throw_error(env, NULL, message); + return NULL; + } + napi_value fn; napi_create_function(env, "initialize", NAPI_AUTO_LENGTH, napi_initialize, NULL, &fn); @@ -3508,6 +3579,12 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_strandedCount", fn); napi_create_function(env, "__test_resolverRefDeleteCount", NAPI_AUTO_LENGTH, napi_test_resolver_ref_delete_count, NULL, &fn); napi_set_named_property(env, exports, "__test_resolverRefDeleteCount", fn); + napi_create_function(env, "__test_holdNextAsyncOp", NAPI_AUTO_LENGTH, napi_test_hold_next_async_op, NULL, &fn); + napi_set_named_property(env, exports, "__test_holdNextAsyncOp", fn); + napi_create_function(env, "__test_asyncOpHeld", NAPI_AUTO_LENGTH, napi_test_async_op_held, NULL, &fn); + napi_set_named_property(env, exports, "__test_asyncOpHeld", fn); + napi_create_function(env, "__test_releaseAsyncOp", NAPI_AUTO_LENGTH, napi_test_release_async_op, NULL, &fn); + napi_set_named_property(env, exports, "__test_releaseAsyncOp", fn); } return exports; diff --git a/native-lib/node/tests/integration/admission-during-teardown.test.ts b/native-lib/node/tests/integration/admission-during-teardown.test.ts index 4a6dc74a..91bf8513 100644 --- a/native-lib/node/tests/integration/admission-during-teardown.test.ts +++ b/native-lib/node/tests/integration/admission-during-teardown.test.ts @@ -3,10 +3,28 @@ import * as ffi from "../../src/ffi"; import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; -// Task 6 supersedes this test's callback-based pending-teardown trigger. While -// native code invokes a transform read callback, all isolate-touching methods -// are rejected before lifecycle mutation or worker admission. Use cleanup and -// streaming start together to cover both guards and the TypeScript mapping. +interface TestAddon { + __test_holdNextAsyncOp(): void; + __test_asyncOpHeld(): boolean; + __test_releaseAsyncOp(): void; +} + +// eslint-disable-next-line @typescript-eslint/no-var-requires +const testAddon = require("../../build/Release/dwlib_addon.node") as TestAddon; + +async function waitForAsyncOpGate(): Promise { + const deadline = Date.now() + 10000; + while (!testAddon.__test_asyncOpHeld()) { + if (Date.now() >= deadline) throw new Error("async operation did not reach test gate"); + await new Promise((resolve) => setImmediate(resolve)); + } +} + +// Task 6 adds this callback-specific contract alongside the pending-teardown +// lifecycle regression below. While native code invokes a transform read +// callback, all isolate-touching methods are rejected before lifecycle mutation +// or worker admission. Use cleanup and streaming start together to cover both +// guards and the TypeScript mapping. describe("transform read callback admission guard", () => { it("rejects cleanup and streaming start before native admission", async () => { ffi.initialize(findLibrary()); @@ -67,3 +85,66 @@ describe("transform read callback admission guard", () => { } }, 20000); }); + +describe("admission rejected while teardown pending (round 6 #2)", () => { + it("rejects a streaming op on the same handle while teardown is pending", async () => { + ffi.initialize(findLibrary()); + const handle = ffi.createEngine(); + let cleanupPromise: Promise | undefined; + let gateArmed = false; + let gateReleased = false; + let handleDestroyed = false; + const chunks: Buffer[] = []; + + try { + testAddon.__test_holdNextAsyncOp(); + gateArmed = true; + const outerPromise = ffi.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[1,2,3]", + buildInputsJson({}), + (chunk) => chunks.push(chunk) + ); + await waitForAsyncOpGate(); + cleanupPromise = ffi.cleanup(); + + let admitErr: unknown; + let admitted = false; + let secondOpSettled: Promise = Promise.resolve(); + try { + secondOpSettled = ffi.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[1,2,3]", + buildInputsJson({}), + () => {} + ).then( + () => { admitted = true; }, + (error) => { admitErr = error; } + ); + } catch (error) { + admitErr = error; + } + + await secondOpSettled; + expect(admitErr).toBeTruthy(); + expect(admitted).toBe(false); + + ffi.destroyEngine(handle); + handleDestroyed = true; + testAddon.__test_releaseAsyncOp(); + gateReleased = true; + const outerResult = JSON.parse(await outerPromise); + expect(outerResult.success).toBe(true); + expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([1, 2, 3]); + await cleanupPromise; + } finally { + if (gateArmed && !gateReleased) testAddon.__test_releaseAsyncOp(); + try { + if (!handleDestroyed) ffi.destroyEngine(handle); + } finally { + if (cleanupPromise) await cleanupPromise; + await ffi.cleanup(); + } + } + }, 20000); +}); diff --git a/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs b/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs index 01ca074c..649b5970 100644 --- a/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs +++ b/native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs @@ -76,8 +76,82 @@ async function runRaw() { } } +async function runRawOutput(mode) { + const addon = require(path.join(ROOT, "build", "Release", "dwlib_addon.node")); + const { findLibrary } = require(path.join(ROOT, "dist", "utils.js")); + let innerHandle = null; + let outerHandle = null; + let nestedErrorCode = null; + const chunks = []; + + addon.initialize(findLibrary()); + try { + innerHandle = addon.createEngine(); + outerHandle = addon.createEngine(); + const write = (chunk) => { + if (nestedErrorCode === null) { + try { + addon.runScriptEngine(innerHandle, "21 * 2", "{}"); + } catch (error) { + nestedErrorCode = error && error.code; + } + } + chunks.push(chunk); + }; + + let raw; + if (mode === "raw-streaming") { + raw = await addon.runScriptStreamingEngine( + outerHandle, + "%dw 2.0\noutput application/json\n---\n[1,2,3]", + "{}", + write + ); + } else { + let read = false; + raw = await addon.runScriptTransformEngine( + outerHandle, + "output application/json\n---\npayload map ($ * 2)", + "{}", + "payload", + "application/json", + null, + () => { + if (read) return null; + read = true; + return Buffer.from("[1,2,3]"); + }, + write + ); + } + console.log(JSON.stringify({ + nestedErrorCode, + outerSuccess: JSON.parse(raw).success, + outerResult: JSON.parse(Buffer.concat(chunks).toString("utf-8")), + })); + } finally { + try { + if (outerHandle !== null) addon.destroyEngine(outerHandle); + } finally { + try { + if (innerHandle !== null) addon.destroyEngine(innerHandle); + } finally { + await addon.cleanup(); + } + } + } +} + const mode = process.argv[2]; -const run = mode === "facade" ? runFacade : mode === "raw" ? runRaw : null; +const run = mode === "facade" + ? runFacade + : mode === "raw" + ? runRaw + : mode === "raw-streaming" + ? () => runRawOutput(mode) + : mode === "raw-transform" + ? () => runRawOutput(mode) + : null; if (run === null) { console.error(`unknown mode: ${mode}`); process.exit(2); diff --git a/native-lib/node/tests/integration/resolver-reentrancy.test.ts b/native-lib/node/tests/integration/resolver-reentrancy.test.ts index f9aecf2c..54a0dcf4 100644 --- a/native-lib/node/tests/integration/resolver-reentrancy.test.ts +++ b/native-lib/node/tests/integration/resolver-reentrancy.test.ts @@ -8,7 +8,7 @@ const DIST_ENTRY = join(__dirname, "..", "..", "dist", "index.js"); const ADDON_PATH = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); const FATAL_GRAAL_ERROR = /Fatal error|Must either be at a safepoint or in native mode/i; -function runFixture(mode: "facade" | "raw") { +function runFixture(mode: "facade" | "raw" | "raw-streaming" | "raw-transform") { const child = spawnSync(process.execPath, [FIXTURE, mode], { encoding: "utf-8", timeout: 30_000, @@ -40,4 +40,20 @@ describe("resolver callback reentrancy guard", () => { outerResult: "42", }); }); + + it("guards the streaming output callback and preserves the outer operation", () => { + expect(runFixture("raw-streaming")).toEqual({ + nestedErrorCode: "ERR_DATAWEAVE_CALLBACK_REENTRANCY", + outerSuccess: true, + outerResult: [1, 2, 3], + }); + }); + + it("guards the transform output callback and preserves the outer operation", () => { + expect(runFixture("raw-transform")).toEqual({ + nestedErrorCode: "ERR_DATAWEAVE_CALLBACK_REENTRANCY", + outerSuccess: true, + outerResult: [2, 4, 6], + }); + }); }); diff --git a/native-lib/node/tests/integration/run-admission.test.ts b/native-lib/node/tests/integration/run-admission.test.ts index 5b851e43..558515ff 100644 --- a/native-lib/node/tests/integration/run-admission.test.ts +++ b/native-lib/node/tests/integration/run-admission.test.ts @@ -3,11 +3,28 @@ import * as ffi from "../../src/ffi"; import { DataWeaveError } from "../../src/errors"; import { findLibrary, buildInputsJson } from "../../src/utils"; -// Task 6 supersedes this test's old callback-based pending-teardown trigger: -// lifecycle and execution entry from a native callback are now rejected before -// cleanup can queue teardown or run can attach to the isolate. Drive the real -// addon through ffi so this also verifies TypeScript error normalization for a -// transform read callback, while the successful outer transform proves callback +interface TestAddon { + __test_holdNextAsyncOp(): void; + __test_asyncOpHeld(): boolean; + __test_releaseAsyncOp(): void; +} + +// eslint-disable-next-line @typescript-eslint/no-var-requires +const testAddon = require("../../build/Release/dwlib_addon.node") as TestAddon; + +async function waitForAsyncOpGate(): Promise { + const deadline = Date.now() + 10000; + while (!testAddon.__test_asyncOpHeld()) { + if (Date.now() >= deadline) throw new Error("async operation did not reach test gate"); + await new Promise((resolve) => setImmediate(resolve)); + } +} + +// Task 6 adds this callback-specific contract alongside the pending-teardown +// lifecycle regression below: lifecycle and execution entry from a native +// callback are rejected before cleanup can queue teardown or run can attach to +// the isolate. Drive the real addon through ffi so this also verifies TypeScript +// error normalization, while the successful outer transform proves callback // depth is restored afterward. describe("transform read callback reentrancy guard", () => { it("rejects cleanup and run before native admission and preserves the outer transform", async () => { @@ -65,3 +82,61 @@ describe("transform read callback reentrancy guard", () => { } }, 20000); }); + +describe("run() admission rejected while teardown pending (round 7 #1)", () => { + it("rejects a synchronous run instead of attaching to a tearing-down isolate", async () => { + ffi.initialize(findLibrary()); + const handle = ffi.createEngine(); + let cleanupPromise: Promise | undefined; + let gateArmed = false; + let gateReleased = false; + let handleDestroyed = false; + const chunks: Buffer[] = []; + + try { + testAddon.__test_holdNextAsyncOp(); + gateArmed = true; + const outerPromise = ffi.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[1,2,3]", + buildInputsJson({}), + (chunk) => chunks.push(chunk) + ); + await waitForAsyncOpGate(); + cleanupPromise = ffi.cleanup(); + + let runErr: unknown; + let ran = false; + try { + ffi.runScriptEngine( + handle, + "%dw 2.0\noutput application/json\n---\n1 + 1", + buildInputsJson({}) + ); + ran = true; + } catch (error) { + runErr = error; + } + + expect(runErr).toBeTruthy(); + expect(ran).toBe(false); + + ffi.destroyEngine(handle); + handleDestroyed = true; + testAddon.__test_releaseAsyncOp(); + gateReleased = true; + const outerResult = JSON.parse(await outerPromise); + expect(outerResult.success).toBe(true); + expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([1, 2, 3]); + await cleanupPromise; + } finally { + if (gateArmed && !gateReleased) testAddon.__test_releaseAsyncOp(); + try { + if (!handleDestroyed) ffi.destroyEngine(handle); + } finally { + if (cleanupPromise) await cleanupPromise; + await ffi.cleanup(); + } + } + }, 20000); +}); diff --git a/native-lib/node/tests/integration/teardown-deadlock.test.ts b/native-lib/node/tests/integration/teardown-deadlock.test.ts index de0bf82b..cb828711 100644 --- a/native-lib/node/tests/integration/teardown-deadlock.test.ts +++ b/native-lib/node/tests/integration/teardown-deadlock.test.ts @@ -1,11 +1,28 @@ import { describe, it, expect } from "vitest"; -import { run, runTransform, cleanup } from "../../src/dataweave"; +import { run, runStreaming, runTransform, cleanup } from "../../src/dataweave"; import { DataWeaveError } from "../../src/errors"; -// Task 6 replaces the old callback-triggered pending-teardown scenario with a -// stronger contract: public DataWeave execution is rejected while native code -// is invoking the transform input callback. The real-addon test still proves -// the worker and outer transform drain without a deadlock after that rejection. +interface TestAddon { + __test_holdNextAsyncOp(): void; + __test_asyncOpHeld(): boolean; + __test_releaseAsyncOp(): void; +} + +// eslint-disable-next-line @typescript-eslint/no-var-requires +const testAddon = require("../../build/Release/dwlib_addon.node") as TestAddon; + +async function waitForAsyncOpGate(): Promise { + const deadline = Date.now() + 10000; + while (!testAddon.__test_asyncOpHeld()) { + if (Date.now() >= deadline) throw new Error("async operation did not reach test gate"); + await new Promise((resolve) => setImmediate(resolve)); + } +} + +// Task 6 adds this callback-specific contract alongside the adoption regression +// below: public DataWeave execution is rejected while native code is invoking +// the transform input callback. The real-addon test also proves the worker and +// outer transform drain without a deadlock after that rejection. describe("public API transform callback reentrancy guard", () => { it( "rejects a nested module-level run and lets the outer transform drain", @@ -34,26 +51,63 @@ describe("public API transform callback reentrancy guard", () => { } } - const gen = runTransform( - "output application/octet-stream\n---\npayload", - input(), - { mimeType: "application/octet-stream" } - ); - - // Drain the whole transform. On unfixed code, execution never reaches - // here: the trigger inside input() already froze the JS thread - // forever before the first read even returns. - let result = await gen.next(); - while (!result.done) { - result = await gen.next(); - } + try { + const gen = runTransform( + "output application/octet-stream\n---\npayload", + input(), + { mimeType: "application/octet-stream" } + ); - expect(fired).toBe(true); - expect(runError).toBeInstanceOf(DataWeaveError); - expect(result.value.success).toBe(true); + // Drain the whole transform. On unfixed code, execution never reaches + // here: the trigger inside input() already froze the JS thread + // forever before the first read even returns. + let result = await gen.next(); + while (!result.done) { + result = await gen.next(); + } - await cleanup(); + expect(fired).toBe(true); + expect(runError).toBeInstanceOf(DataWeaveError); + expect(result.value.success).toBe(true); + } finally { + await cleanup(); + } }, 20000 ); }); + +describe("re-init during pending teardown (W-23692110, round 5 P1)", () => { + it("adopts the live isolate instead of blocking initialize behind an active op", async () => { + let cleanupPromise: Promise | undefined; + let gateArmed = false; + let gateReleased = false; + + try { + expect(run("%dw 2.0\noutput application/json\n---\n6 * 7").success).toBe(true); + testAddon.__test_holdNextAsyncOp(); + gateArmed = true; + const outer = runStreaming("%dw 2.0\noutput application/json\n---\n[1,2,3]"); + const firstNext = outer.next(); + await waitForAsyncOpGate(); + cleanupPromise = cleanup(); + + const result = run("%dw 2.0\noutput application/json\n---\n1 + 1"); + expect(result.success).toBe(true); + expect(JSON.parse(result.getString()!)).toBe(2); + + testAddon.__test_releaseAsyncOp(); + gateReleased = true; + let outerResult = await firstNext; + while (!outerResult.done) { + outerResult = await outer.next(); + } + expect(outerResult.value.success).toBe(true); + await cleanupPromise; + } finally { + if (gateArmed && !gateReleased) testAddon.__test_releaseAsyncOp(); + if (cleanupPromise) await cleanupPromise; + await cleanup(); + } + }, 20000); +}); From 22c38056b174c884eb55204ac681a765306e2c36 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 13:33:20 -0300 Subject: [PATCH 24/48] fix(node): bind lazy streams to engine generations --- native-lib/node/src/dataweave.ts | 64 +++++++-- .../integration/instance-lifecycle.test.ts | 111 ++++++++++++++-- .../tests/unit/dataweave-initialize.test.ts | 125 ++++++++++++++++++ 3 files changed, 275 insertions(+), 25 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 59f3c4da..089efd15 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -8,6 +8,11 @@ import { DataWeaveError, DataWeaveScriptError } from "./errors"; import type { ExecutionResult, StreamingResult, Inputs, TransformOptions } from "./types"; import type { ModuleResolver } from "./resolver"; +interface EngineOperationToken { + readonly handle: number; + readonly generation: number; +} + /** * Constructor options for {@link DataWeave}. */ @@ -52,6 +57,7 @@ export class DataWeave { private readonly resolveModule?: ModuleResolver; private state: "uninitialized" | "ready" | "cleaning-up" = "uninitialized"; private engineHandle: number | null = null; + private engineGeneration = 0; private cleanupPromise: Promise | null = null; /** @@ -89,9 +95,10 @@ export class DataWeave { try { ffi.initialize(this.libPath, this.addonPath); libRefAcquired = true; - this.engineHandle = this.resolveModule + const engineHandle = this.resolveModule ? ffi.createEngineWithResolver(this.resolveModule) : ffi.createEngine(); + this.engineHandle = engineHandle; } catch (e: unknown) { // If ffi.initialize() already succeeded but engine creation then threw, we // already hold an increment of the native library's ref-counted handle and @@ -136,6 +143,7 @@ export class DataWeave { throw new DataWeaveError(`Failed to initialize: ${e instanceof Error ? e.message : e}`); } this.state = "ready"; + this.engineGeneration++; } /** @@ -251,11 +259,20 @@ export class DataWeave { * @returns An async generator of output chunks, returning the streaming metadata. * @throws DataWeaveError if the runtime is not initialized. */ - async *runStreaming(script: string, inputs?: Inputs): AsyncGenerator { - this.ensureReady(); + runStreaming(script: string, inputs?: Inputs): AsyncGenerator { + const token = this.captureOperationToken(); + return this.runStreamingInternal(token, script, inputs); + } + + private async *runStreamingInternal( + token: EngineOperationToken, + script: string, + inputs?: Inputs + ): AsyncGenerator { const inputsJson = buildInputsJson(inputs ?? {}); + this.assertCurrentOperation(token); return yield* streamFromNative((chunkCb) => - ffi.runScriptStreamingEngine(this.engineHandle!, script, inputsJson, chunkCb) + ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb) ); } @@ -275,12 +292,22 @@ export class DataWeave { * @returns An async generator of output chunks, returning the streaming metadata. * @throws DataWeaveError if the runtime is not initialized. */ - async *runTransform( + runTransform( script: string, input: AsyncIterable | Iterable, opts?: TransformOptions ): AsyncGenerator { - this.ensureReady(); + const token = this.captureOperationToken(); + return this.runTransformInternal(token, script, input, opts); + } + + private async *runTransformInternal( + token: EngineOperationToken, + script: string, + input: AsyncIterable | Iterable, + opts?: TransformOptions + ): AsyncGenerator { + this.assertCurrentOperation(token); const inputName = opts?.inputName ?? "payload"; const inputMimeType = opts?.mimeType ?? "application/json"; @@ -290,17 +317,11 @@ export class DataWeave { const readCb = await createChunkReader(input); - // The instance may have been cleaned up while an async input pre-buffered - // (createChunkReader can await arbitrarily long). Re-check readiness so a - // caller that raced cleanup() gets a synchronous DataWeaveError rather than - // a resolved "Unknown engine handle" envelope. The C admission pin is the - // authoritative memory-safety guard (round 11 #2/#3); this only improves the - // failure ergonomics for a misused instance. (round 12 #4) - this.ensureReady(); + this.assertCurrentOperation(token); return yield* streamFromNative((writeCb) => ffi.runScriptTransformEngine( - this.engineHandle!, + token.handle, script, inputsJson, inputName, @@ -312,6 +333,21 @@ export class DataWeave { ); } + private captureOperationToken(): EngineOperationToken { + this.ensureReady(); + return { handle: this.engineHandle!, generation: this.engineGeneration }; + } + + private assertCurrentOperation(token: EngineOperationToken): void { + if ( + this.state !== "ready" || + this.engineHandle !== token.handle || + this.engineGeneration !== token.generation + ) { + throw new DataWeaveError("DataWeave operation belongs to a stale engine generation."); + } + } + private ensureReady(): void { if (this.state === "ready") return; if (this.state === "cleaning-up") { diff --git a/native-lib/node/tests/integration/instance-lifecycle.test.ts b/native-lib/node/tests/integration/instance-lifecycle.test.ts index e644662e..9df7ab32 100644 --- a/native-lib/node/tests/integration/instance-lifecycle.test.ts +++ b/native-lib/node/tests/integration/instance-lifecycle.test.ts @@ -60,22 +60,25 @@ describe("instance lifecycle during cleanup (round 6)", () => { await closing; }); - // Finding #1, streaming/transform variants: the async generators must reject - // on first pull when started during the cleanup window. - it("runStreaming()/runTransform() during pending cleanup reject on first pull", async () => { + // Finding #1, streaming/transform variants: ordinary public methods capture + // their engine identity at call time, so a call during cleanup rejects before + // it can return a generator or admit native work. + it("runStreaming()/runTransform() during pending cleanup reject at call time", async () => { dw = new DataWeave(); dw.initialize(); const closing = dw.cleanup(); - const sgen = dw.runStreaming("%dw 2.0\noutput application/json\n---\n[1,2,3]"); - await expect(sgen.next()).rejects.toThrow(DataWeaveError); + expect(() => + dw!.runStreaming("%dw 2.0\noutput application/json\n---\n[1,2,3]") + ).toThrow(DataWeaveError); - const tgen = dw.runTransform( - "output application/json\n---\npayload", - [Buffer.from("[1,2,3]")], - { mimeType: "application/json" } - ); - await expect(tgen.next()).rejects.toThrow(DataWeaveError); + expect(() => + dw!.runTransform( + "output application/json\n---\npayload", + [Buffer.from("[1,2,3]")], + { mimeType: "application/json" } + ) + ).toThrow(DataWeaveError); await closing; }); @@ -160,6 +163,92 @@ describe("runTransform re-checks readiness after async input pre-buffering (roun }); }); +describe("lazy streams are bound to their engine generation (Task 7)", () => { + const staleGenerationMessage = "DataWeave operation belongs to a stale engine generation."; + + it("rejects stale runStreaming work and allows a replacement-generation stream", async () => { + const anchor = new DataWeave(); + const target = new DataWeave(); + anchor.initialize(); + + try { + target.initialize(); + const stale = target.runStreaming("output application/json --- [1, 2, 3]"); + + await target.cleanup(); + target.initialize(); + + const stalePull = stale.next(); + await expect(stalePull).rejects.toBeInstanceOf(DataWeaveError); + await expect(stalePull).rejects.toThrow(staleGenerationMessage); + + const current = target.runStreaming("output application/json --- [4, 5, 6]"); + const chunks: Buffer[] = []; + let result = await current.next(); + while (!result.done) { + chunks.push(result.value); + result = await current.next(); + } + expect(result.value.success).toBe(true); + expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([4, 5, 6]); + } finally { + let targetError: unknown; + try { + await target.cleanup(); + } catch (e) { + targetError = e; + } + await anchor.cleanup(); + if (targetError !== undefined) throw targetError; + } + }); + + it("rejects stale runTransform work and allows a replacement-generation transform", async () => { + const anchor = new DataWeave(); + const target = new DataWeave(); + anchor.initialize(); + + try { + target.initialize(); + const stale = target.runTransform( + "output application/json --- payload map ($ * 2)", + [Buffer.from("[1, 2, 3]")], + { mimeType: "application/json" } + ); + + await target.cleanup(); + target.initialize(); + + const stalePull = stale.next(); + await expect(stalePull).rejects.toBeInstanceOf(DataWeaveError); + await expect(stalePull).rejects.toThrow(staleGenerationMessage); + + const current = target.runTransform( + "output application/json --- payload map ($ * 2)", + [Buffer.from("[4, 5, 6]")], + { mimeType: "application/json" } + ); + const chunks: Buffer[] = []; + let result = await current.next(); + while (!result.done) { + chunks.push(result.value); + result = await current.next(); + } + expect(result.value.success).toBe(true); + expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([8, 10, 12]); + } finally { + let targetError: unknown; + try { + await target.cleanup(); + } catch (e) { + targetError = e; + } + await anchor.cleanup(); + if (targetError !== undefined) throw targetError; + } + }); +}); + // Round 12, Task 6: the exported module-level cleanup() nulls globalInstance // synchronously, then awaits instance.cleanup(). A second overlapping // module-level cleanup() call must coalesce onto the SAME in-flight drain diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 51d199f0..68f73a11 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -27,6 +27,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { vi.mocked(ffi.createEngine).mockReset(); vi.mocked(ffi.createEngineWithResolver).mockReset(); vi.mocked(ffi.destroyEngine).mockReset(); + vi.mocked(ffi.runScriptStreamingEngine).mockReset(); + vi.mocked(ffi.runScriptTransformEngine).mockReset(); vi.mocked(ffi.cleanup).mockReset(); }); @@ -397,4 +399,127 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); expect(ffi.destroyEngine).toHaveBeenCalledWith(11); }); + + describe("stale engine generation", () => { + const staleGenerationMessage = "DataWeave operation belongs to a stale engine generation."; + + it("rejects a lazy runStreaming operation after replacement with a different handle before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValueOnce(2).mockReturnValueOnce(3); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- [1, 2, 3]"); + + await dw.cleanup(); + dw.initialize(); + + const firstPull = stream.next(); + await firstPull.catch(() => undefined); + expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); + await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); + await expect(firstPull).rejects.toThrow(staleGenerationMessage); + + await dw.cleanup(); + }); + + it("rejects a lazy runStreaming operation when the replacement reuses its handle before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(2); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- [1, 2, 3]"); + + await dw.cleanup(); + dw.initialize(); + + const firstPull = stream.next(); + await firstPull.catch(() => undefined); + expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); + await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); + await expect(firstPull).rejects.toThrow(staleGenerationMessage); + + await dw.cleanup(); + }); + + it("rejects a lazy runTransform operation after replacement with a different handle before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValueOnce(2).mockReturnValueOnce(3); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform( + "output application/json --- payload", + [Buffer.from("[1, 2, 3]")], + { mimeType: "application/json" } + ); + + await dw.cleanup(); + dw.initialize(); + + const firstPull = transform.next(); + await firstPull.catch(() => undefined); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); + await expect(firstPull).rejects.toThrow(staleGenerationMessage); + + await dw.cleanup(); + }); + + it("rejects a lazy runTransform operation when the replacement reuses its handle before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(2); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform( + "output application/json --- payload", + [Buffer.from("[1, 2, 3]")], + { mimeType: "application/json" } + ); + + await dw.cleanup(); + dw.initialize(); + + const firstPull = transform.next(); + await firstPull.catch(() => undefined); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); + await expect(firstPull).rejects.toThrow(staleGenerationMessage); + + await dw.cleanup(); + }); + + it("revalidates a runTransform operation after async input pre-buffering before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(2); + + let markInputStarted!: () => void; + const inputStarted = new Promise((resolve) => { markInputStarted = resolve; }); + let resumeInput!: () => void; + const inputPaused = new Promise((resolve) => { resumeInput = resolve; }); + async function* slowInput(): AsyncGenerator { + markInputStarted(); + await inputPaused; + yield Buffer.from("[1, 2, 3]"); + } + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform( + "output application/json --- payload", + slowInput(), + { mimeType: "application/json" } + ); + const firstPull = transform.next(); + await inputStarted; + + await dw.cleanup(); + dw.initialize(); + resumeInput(); + + await firstPull.catch(() => undefined); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); + await expect(firstPull).rejects.toThrow(staleGenerationMessage); + + await dw.cleanup(); + }); + }); }); From 6837818a83df133f3c72abf1ac904689f95c2bf3 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 14:56:47 -0300 Subject: [PATCH 25/48] test(node): assert exact stale generation errors --- .../integration/instance-lifecycle.test.ts | 16 +++++++--- .../tests/unit/dataweave-initialize.test.ts | 30 +++++++++---------- 2 files changed, 27 insertions(+), 19 deletions(-) diff --git a/native-lib/node/tests/integration/instance-lifecycle.test.ts b/native-lib/node/tests/integration/instance-lifecycle.test.ts index 9df7ab32..09bd03fb 100644 --- a/native-lib/node/tests/integration/instance-lifecycle.test.ts +++ b/native-lib/node/tests/integration/instance-lifecycle.test.ts @@ -165,6 +165,16 @@ describe("runTransform re-checks readiness after async input pre-buffering (roun describe("lazy streams are bound to their engine generation (Task 7)", () => { const staleGenerationMessage = "DataWeave operation belongs to a stale engine generation."; + const expectStaleGenerationError = async (operation: Promise): Promise => { + let error: unknown; + try { + await operation; + } catch (caught) { + error = caught; + } + expect(error).toBeInstanceOf(DataWeaveError); + expect((error as DataWeaveError).message).toBe(staleGenerationMessage); + }; it("rejects stale runStreaming work and allows a replacement-generation stream", async () => { const anchor = new DataWeave(); @@ -179,8 +189,7 @@ describe("lazy streams are bound to their engine generation (Task 7)", () => { target.initialize(); const stalePull = stale.next(); - await expect(stalePull).rejects.toBeInstanceOf(DataWeaveError); - await expect(stalePull).rejects.toThrow(staleGenerationMessage); + await expectStaleGenerationError(stalePull); const current = target.runStreaming("output application/json --- [4, 5, 6]"); const chunks: Buffer[] = []; @@ -220,8 +229,7 @@ describe("lazy streams are bound to their engine generation (Task 7)", () => { target.initialize(); const stalePull = stale.next(); - await expect(stalePull).rejects.toBeInstanceOf(DataWeaveError); - await expect(stalePull).rejects.toThrow(staleGenerationMessage); + await expectStaleGenerationError(stalePull); const current = target.runTransform( "output application/json --- payload map ($ * 2)", diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 68f73a11..d33a6071 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -402,6 +402,16 @@ describe("DataWeave.initialize() native ref-count safety", () => { describe("stale engine generation", () => { const staleGenerationMessage = "DataWeave operation belongs to a stale engine generation."; + const expectStaleGenerationError = async (operation: Promise): Promise => { + let error: unknown; + try { + await operation; + } catch (caught) { + error = caught; + } + expect(error).toBeInstanceOf(DataWeaveError); + expect((error as DataWeaveError).message).toBe(staleGenerationMessage); + }; it("rejects a lazy runStreaming operation after replacement with a different handle before native admission", async () => { vi.mocked(ffi.createEngine).mockReturnValueOnce(2).mockReturnValueOnce(3); @@ -414,10 +424,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); const firstPull = stream.next(); - await firstPull.catch(() => undefined); + await expectStaleGenerationError(firstPull); expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); - await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); - await expect(firstPull).rejects.toThrow(staleGenerationMessage); await dw.cleanup(); }); @@ -433,10 +441,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); const firstPull = stream.next(); - await firstPull.catch(() => undefined); + await expectStaleGenerationError(firstPull); expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); - await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); - await expect(firstPull).rejects.toThrow(staleGenerationMessage); await dw.cleanup(); }); @@ -456,10 +462,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); const firstPull = transform.next(); - await firstPull.catch(() => undefined); + await expectStaleGenerationError(firstPull); expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); - await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); - await expect(firstPull).rejects.toThrow(staleGenerationMessage); await dw.cleanup(); }); @@ -479,10 +483,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); const firstPull = transform.next(); - await firstPull.catch(() => undefined); + await expectStaleGenerationError(firstPull); expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); - await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); - await expect(firstPull).rejects.toThrow(staleGenerationMessage); await dw.cleanup(); }); @@ -514,10 +516,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); resumeInput(); - await firstPull.catch(() => undefined); + await expectStaleGenerationError(firstPull); expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); - await expect(firstPull).rejects.toBeInstanceOf(DataWeaveError); - await expect(firstPull).rejects.toThrow(staleGenerationMessage); await dw.cleanup(); }); From 6cdfa998bfccc6a4bcff619658f6fd1fa8bb0fab Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 15:06:24 -0300 Subject: [PATCH 26/48] fix(node): validate streams around serialization --- native-lib/node/src/dataweave.ts | 1 + .../integration/instance-lifecycle.test.ts | 46 ++++++++++------- .../tests/unit/dataweave-initialize.test.ts | 49 +++++++++++++++++++ 3 files changed, 78 insertions(+), 18 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 089efd15..c667f753 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -269,6 +269,7 @@ export class DataWeave { script: string, inputs?: Inputs ): AsyncGenerator { + this.assertCurrentOperation(token); const inputsJson = buildInputsJson(inputs ?? {}); this.assertCurrentOperation(token); return yield* streamFromNative((chunkCb) => diff --git a/native-lib/node/tests/integration/instance-lifecycle.test.ts b/native-lib/node/tests/integration/instance-lifecycle.test.ts index 09bd03fb..e0032fd6 100644 --- a/native-lib/node/tests/integration/instance-lifecycle.test.ts +++ b/native-lib/node/tests/integration/instance-lifecycle.test.ts @@ -4,6 +4,26 @@ import { DataWeaveError } from "../../src/errors"; import * as ffi from "../../src/ffi"; import { findLibrary, buildInputsJson } from "../../src/utils"; +async function cleanupTask7Instances( + target: DataWeave, + anchor: DataWeave, + bodySucceeded: boolean +): Promise { + let cleanupError: unknown; + try { + await target.cleanup(); + } catch (error) { + cleanupError = error; + } + try { + await anchor.cleanup(); + } catch (error) { + if (cleanupError === undefined) cleanupError = error; + } + // Cleanup must not replace a more actionable failure from the test body. + if (bodySucceeded && cleanupError !== undefined) throw cleanupError; +} + // Same-instance lifecycle regression tests (round 6, W-23692110). Round 5's // coverage used a second instance; the same-instance cleanup window is exactly // what findings #1 and #3 exploit. Real addon, no mocking. @@ -179,9 +199,10 @@ describe("lazy streams are bound to their engine generation (Task 7)", () => { it("rejects stale runStreaming work and allows a replacement-generation stream", async () => { const anchor = new DataWeave(); const target = new DataWeave(); - anchor.initialize(); + let bodySucceeded = false; try { + anchor.initialize(); target.initialize(); const stale = target.runStreaming("output application/json --- [1, 2, 3]"); @@ -200,24 +221,19 @@ describe("lazy streams are bound to their engine generation (Task 7)", () => { } expect(result.value.success).toBe(true); expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([4, 5, 6]); + bodySucceeded = true; } finally { - let targetError: unknown; - try { - await target.cleanup(); - } catch (e) { - targetError = e; - } - await anchor.cleanup(); - if (targetError !== undefined) throw targetError; + await cleanupTask7Instances(target, anchor, bodySucceeded); } }); it("rejects stale runTransform work and allows a replacement-generation transform", async () => { const anchor = new DataWeave(); const target = new DataWeave(); - anchor.initialize(); + let bodySucceeded = false; try { + anchor.initialize(); target.initialize(); const stale = target.runTransform( "output application/json --- payload map ($ * 2)", @@ -244,15 +260,9 @@ describe("lazy streams are bound to their engine generation (Task 7)", () => { } expect(result.value.success).toBe(true); expect(JSON.parse(Buffer.concat(chunks).toString("utf-8"))).toEqual([8, 10, 12]); + bodySucceeded = true; } finally { - let targetError: unknown; - try { - await target.cleanup(); - } catch (e) { - targetError = e; - } - await anchor.cleanup(); - if (targetError !== undefined) throw targetError; + await cleanupTask7Instances(target, anchor, bodySucceeded); } }); }); diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index d33a6071..420c5caf 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -447,6 +447,55 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("rejects stale runStreaming work before input serialization can run", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(2); + let inputEnumerations = 0; + const inputs = new Proxy({ payload: 1 }, { + ownKeys() { + inputEnumerations++; + throw new Error("stale inputs must not be enumerated"); + }, + }); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- payload", inputs); + + await dw.cleanup(); + dw.initialize(); + + await expectStaleGenerationError(stream.next()); + expect(inputEnumerations).toBe(0); + expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("revalidates runStreaming after input serialization before native admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(2); + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + + let inputReads = 0; + let invalidatingCleanup: Promise | undefined; + const inputs = {}; + Object.defineProperty(inputs, "payload", { + enumerable: true, + get() { + inputReads++; + invalidatingCleanup = dw.cleanup(); + return 1; + }, + }); + + const stream = dw.runStreaming("output application/json --- payload", inputs); + await expectStaleGenerationError(stream.next()); + + expect(inputReads).toBe(1); + expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); + await invalidatingCleanup; + }); + it("rejects a lazy runTransform operation after replacement with a different handle before native admission", async () => { vi.mocked(ffi.createEngine).mockReturnValueOnce(2).mockReturnValueOnce(3); From 1ea48a6b608765861371a0ce54539fd2d6e060e6 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 15:28:39 -0300 Subject: [PATCH 27/48] fix(node): propagate streaming consumer credits --- native-lib/node/src/dataweave.ts | 49 +++- native-lib/node/src/ffi.ts | 15 +- native-lib/node/src/stream.ts | 172 ++++++++++--- .../tests/unit/dataweave-initialize.test.ts | 166 ++++++++++++ native-lib/node/tests/unit/stream.test.ts | 241 ++++++++++++++++-- 5 files changed, 572 insertions(+), 71 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index c667f753..fac3bc20 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -5,6 +5,7 @@ import { parseNativeResponse } from "./result"; import { createChunkReader } from "./reader"; import { streamFromNative } from "./stream"; import { DataWeaveError, DataWeaveScriptError } from "./errors"; +import type { NativeStreamingOperation } from "./ffi"; import type { ExecutionResult, StreamingResult, Inputs, TransformOptions } from "./types"; import type { ModuleResolver } from "./resolver"; @@ -58,6 +59,7 @@ export class DataWeave { private state: "uninitialized" | "ready" | "cleaning-up" = "uninitialized"; private engineHandle: number | null = null; private engineGeneration = 0; + private readonly activeStreams = new Set(); private cleanupPromise: Promise | null = null; /** @@ -196,6 +198,22 @@ export class DataWeave { // during the async teardown window are rejected deterministically rather // than seeing a stale "ready" state with a null engineHandle (round-6 #1/#3). this.state = "cleaning-up"; + const activeStreams = [...this.activeStreams]; + for (const operation of activeStreams) { + try { + operation.cancel(); + } catch { + // A controller failure must not prevent the remaining streams from + // being canceled or the engine reference from being released. + } + } + // Cancellation may reject completion as its normal terminal signal. Wait + // for every snapshotted operation, but do not let those rejections mask an + // engine destruction or native cleanup failure. + if (activeStreams.length > 0) { + await Promise.allSettled(activeStreams.map((operation) => operation.completion)); + } + let destroyError: unknown; try { if (this.engineHandle !== null) { @@ -272,8 +290,10 @@ export class DataWeave { this.assertCurrentOperation(token); const inputsJson = buildInputsJson(inputs ?? {}); this.assertCurrentOperation(token); - return yield* streamFromNative((chunkCb) => - ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb) + return yield* streamFromNative( + (chunkCb) => ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb), + (operation) => { this.activeStreams.add(operation); }, + (operation) => { this.activeStreams.delete(operation); } ); } @@ -320,17 +340,20 @@ export class DataWeave { this.assertCurrentOperation(token); - return yield* streamFromNative((writeCb) => - ffi.runScriptTransformEngine( - token.handle, - script, - inputsJson, - inputName, - inputMimeType, - inputCharset, - readCb, - writeCb - ) + return yield* streamFromNative( + (writeCb) => + ffi.runScriptTransformEngine( + token.handle, + script, + inputsJson, + inputName, + inputMimeType, + inputCharset, + readCb, + writeCb + ), + (operation) => { this.activeStreams.add(operation); }, + (operation) => { this.activeStreams.delete(operation); } ); } diff --git a/native-lib/node/src/ffi.ts b/native-lib/node/src/ffi.ts index 21e0e287..21b1cce6 100644 --- a/native-lib/node/src/ffi.ts +++ b/native-lib/node/src/ffi.ts @@ -2,6 +2,13 @@ import { resolveAddonPath } from "./addon-path"; import { DataWeaveError } from "./errors"; import type { ModuleResolver } from "./resolver"; +export interface NativeStreamingOperation { + readonly completion: Promise; + acknowledge(bytes: number): void; + cancel(): void; + close(): void; +} + interface NativeAddon { initialize(libPath: string): void; createEngine(): number; @@ -13,7 +20,7 @@ interface NativeAddon { script: string, inputsJson: string, chunkCb: (chunk: Buffer) => void - ): Promise; + ): NativeStreamingOperation; runScriptTransformEngine( handle: number, script: string, @@ -23,7 +30,7 @@ interface NativeAddon { inputCharset: string | null, readCb: (bufSize: number) => Buffer | null, writeCb: (chunk: Buffer) => void - ): Promise; + ): NativeStreamingOperation; cleanup(): Promise; } @@ -78,7 +85,7 @@ export function runScriptStreamingEngine( script: string, inputsJson: string, chunkCb: (chunk: Buffer) => void -): Promise { +): NativeStreamingOperation { return callNative(() => getAddon().runScriptStreamingEngine(handle, script, inputsJson, chunkCb) ); @@ -93,7 +100,7 @@ export function runScriptTransformEngine( inputCharset: string | null, readCb: (bufSize: number) => Buffer | null, writeCb: (chunk: Buffer) => void -): Promise { +): NativeStreamingOperation { return callNative(() => getAddon().runScriptTransformEngine( handle, diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index 032d5a68..4cd516be 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -1,11 +1,12 @@ import { parseStreamingResult } from "./result"; +import type { NativeStreamingOperation } from "./ffi"; import type { StreamingResult } from "./types"; /** * Starts a native streaming call, wiring its chunk callback to `chunkCb` and - * resolving to the raw trailing metadata JSON once the stream completes. + * returning its controller once native admission succeeds. */ -export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => Promise; +export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => NativeStreamingOperation; /** * Bridges a native push-based streaming call into a pull-based async generator. @@ -13,22 +14,41 @@ export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => Promise void, + onClose?: (operation: NativeStreamingOperation) => void ): AsyncGenerator { const chunks: Buffer[] = []; const pendingResolves: Array<() => void> = []; - let done = false; + let operation: NativeStreamingOperation | undefined; + let nativeSettled = false; + let finalized = false; + let cancellationRequested = false; let metaRaw: string | null = null; + let settlementCloseError: unknown; const chunkCb = (chunk: Buffer) => { - chunks.push(chunk); + const copy = Buffer.from(chunk); + if ((finalized || cancellationRequested) && operation) { + try { + operation.acknowledge(copy.length); + } catch { + // Late callback credit is best-effort after the consumer has abandoned + // the stream; cleanup must still be able to cancel and close it. + } + return; + } + chunks.push(copy); // Resolve one waiting consumer if any const resolve = pendingResolves.shift(); if (resolve) { @@ -44,38 +64,114 @@ export async function* streamFromNative( if (resolve) resolve(); } }; + const acknowledgeBufferedChunks = () => { + while (chunks.length > 0) { + operation!.acknowledge(chunks.shift()!.length); + } + }; - // Handle BOTH settlement branches. Without the rejection handler, a rejected - // start() leaves `done` false forever: a consumer parked in next() below is - // never woken and the generator hangs, and the rejection is unhandled - // (review #6 #2). On rejection we record the error, flip startRejected, mark - // completion, and wake every waiter; the error is re-thrown (by settlement - // state, not by value -- see below) after draining any chunks that arrived - // before the rejection. Because we handle rejection here, metaPromise itself - // always fulfills -- `await metaPromise` below never throws. - const metaPromise = start(chunkCb).then( - (raw) => { metaRaw = raw; done = true; wakeAll(); }, - (err) => { startError = err; startRejected = true; done = true; wakeAll(); } - ); + try { + const nativeOperation = start(chunkCb); + let cancelCalled = false; + let closeCalled = false; + let registered = false; + operation = { + completion: nativeOperation.completion, + acknowledge: (bytes) => nativeOperation.acknowledge(bytes), + cancel: () => { + if (cancelCalled) return; + cancelCalled = true; + cancellationRequested = true; + try { + acknowledgeBufferedChunks(); + } finally { + wakeAll(); + if (!nativeSettled) nativeOperation.cancel(); + else operation!.close(); + } + }, + close: () => { + if (closeCalled) return; + closeCalled = true; + try { + nativeOperation.close(); + } finally { + if (registered) onClose?.(operation!); + } + }, + }; - while (true) { - if (chunks.length > 0) { - yield chunks.shift()!; - continue; + let completionHandled: Promise; + try { + // Handle both settlement branches so rejected native completion cannot + // become unhandled or leave a parked consumer asleep. Chunks already in + // the JS queue still drain before the rejection is surfaced. + completionHandled = operation.completion.then( + (raw) => { + metaRaw = raw; + nativeSettled = true; + wakeAll(); + if (cancellationRequested) { + try { + operation!.close(); + } catch (error) { + settlementCloseError = error; + } + } + }, + (error) => { + startError = error; + startRejected = true; + nativeSettled = true; + wakeAll(); + if (cancellationRequested) { + try { + operation!.close(); + } catch (closeError) { + settlementCloseError = closeError; + } + } + } + ); + + onStart?.(operation); + registered = true; + } catch (error) { + operation.cancel(); + await Promise.allSettled([operation.completion]); + throw error; } - if (done) break; - await new Promise((resolve) => { pendingResolves.push(resolve); }); - } - // Drain remaining chunks buffered before completion/rejection. - while (chunks.length > 0) { - yield chunks.shift()!; - } + while (true) { + if (chunks.length > 0) { + const chunk = chunks.shift()!; + operation.acknowledge(chunk.length); + yield chunk; + continue; + } + if (nativeSettled) break; + await new Promise((resolve) => { pendingResolves.push(resolve); }); + } - await metaPromise; - // Track rejection by settlement STATE, not by the rejected value: Promise.reject(undefined) - // is valid JS, so a value sentinel (startError !== undefined) would swallow it as an empty - // result. startRejected is only ever set in the rejection handler above (review #7 #6). - if (startRejected) throw startError; - return parseStreamingResult(metaRaw ?? ""); -} \ No newline at end of file + await completionHandled; + // Track rejection by settlement state, not value: Promise.reject(undefined) + // is valid and must not be mistaken for a successful empty response. + if (startRejected) throw startError; + if (settlementCloseError !== undefined) throw settlementCloseError; + return parseStreamingResult(metaRaw ?? ""); + } finally { + finalized = true; + if (operation) { + try { + acknowledgeBufferedChunks(); + } finally { + try { + if (!nativeSettled) operation.cancel(); + } finally { + operation.close(); + } + } + } + wakeAll(); + } +} diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 420c5caf..752ea564 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -20,6 +20,30 @@ vi.mock("../../src/ffi", () => ({ import * as ffi from "../../src/ffi"; import { DataWeave, run, cleanup } from "../../src/dataweave"; import { DataWeaveError } from "../../src/errors"; +import type { NativeStreamingOperation } from "../../src/ffi"; + +function deferred() { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((res, rej) => { resolve = res; reject = rej; }); + return { promise, resolve, reject }; +} + +function operation(completion: Promise): NativeStreamingOperation { + return { + completion, + acknowledge: vi.fn(), + cancel: vi.fn(), + close: vi.fn(), + }; +} + +const okStreamingMeta = () => JSON.stringify({ + success: true, + mimeType: "application/json", + charset: "utf-8", + binary: false, +}); describe("DataWeave.initialize() native ref-count safety", () => { beforeEach(() => { @@ -400,6 +424,148 @@ describe("DataWeave.initialize() native ref-count safety", () => { expect(ffi.destroyEngine).toHaveBeenCalledWith(11); }); + describe("active streaming cleanup", () => { + it("starts and registers native work only on first iteration", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(12); + const nativeOperation = operation(Promise.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- [1]"); + + expect(ffi.runScriptStreamingEngine).not.toHaveBeenCalled(); + const result = await stream.next(); + expect(result.done).toBe(true); + expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + }); + + it("cancels active operations and awaits their settlement before destroying the engine", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(13); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- [1]"); + const firstPull = stream.next(); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1)); + + const cleanupPromise = dw.cleanup(); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(ffi.destroyEngine).not.toHaveBeenCalled(); + + completion.resolve(okStreamingMeta()); + await Promise.all([firstPull, cleanupPromise]); + + expect(ffi.destroyEngine).toHaveBeenCalledWith(13); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("acknowledges buffered chunks abandoned by cleanup", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(16); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(ffi.runScriptStreamingEngine).mockImplementation((_handle, _script, _inputs, cb) => { + cb(Buffer.from("x")); + cb(Buffer.from("yy")); + return nativeOperation; + }); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const stream = dw.runStreaming("output application/json --- [1]"); + const first = await stream.next(); + expect(first.value?.toString()).toBe("x"); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + + const cleanupPromise = dw.cleanup(); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + + completion.resolve(okStreamingMeta()); + await cleanupPromise; + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("tracks and cancels transform operations after lazy input preparation", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(17); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(ffi.runScriptTransformEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform( + "output application/json --- payload", + [Buffer.from("[1]")] + ); + + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + + const cleanupPromise = dw.cleanup(); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + completion.resolve(okStreamingMeta()); + await Promise.all([firstPull, cleanupPromise]); + + expect(ffi.destroyEngine).toHaveBeenCalledWith(17); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("does not fail cleanup when a canceled operation rejects", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(14); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPull = dw.runStreaming("output application/json --- [1]").next(); + const firstPullOutcome = firstPull.catch((error) => error); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1)); + + const cleanupPromise = dw.cleanup(); + completion.reject(new Error("operation canceled")); + + await expect(cleanupPromise).resolves.toBeUndefined(); + await expect(firstPullOutcome).resolves.toEqual(new Error("operation canceled")); + expect(ffi.destroyEngine).toHaveBeenCalledWith(14); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("does not let operation rejection mask a primary destroy error", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(15); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + vi.mocked(ffi.destroyEngine).mockImplementation(() => { + throw new Error("destroy boom"); + }); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPull = dw.runStreaming("output application/json --- [1]").next(); + const firstPullOutcome = firstPull.catch((error) => error); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1)); + + const cleanupPromise = dw.cleanup(); + completion.reject(new Error("operation canceled")); + + await expect(cleanupPromise).rejects.toThrow("destroy boom"); + await firstPullOutcome; + expect(ffi.cleanup).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + }); + describe("stale engine generation", () => { const staleGenerationMessage = "DataWeave operation belongs to a stale engine generation."; const expectStaleGenerationError = async (operation: Promise): Promise => { diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index ab6e3580..8d96a473 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -1,5 +1,6 @@ -import { describe, it, expect } from "vitest"; +import { describe, it, expect, vi } from "vitest"; import { streamFromNative } from "../../src/stream"; +import type { NativeStreamingOperation } from "../../src/ffi"; import type { StreamingResult } from "../../src/types"; const okMeta = (extra: Record = {}) => @@ -25,42 +26,66 @@ function deferred() { return { promise, resolve, reject }; } +function operation(completion: Promise): NativeStreamingOperation { + return { + completion, + acknowledge: vi.fn(), + cancel: vi.fn(), + close: vi.fn(), + }; +} + describe("streamFromNative", () => { it("yields chunks pushed before completion, in order", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); const { chunks, result } = await collect( streamFromNative((cb) => { cb(Buffer.from("a")); cb(Buffer.from("b")); cb(Buffer.from("c")); - return Promise.resolve(okMeta()); + return nativeOperation; }) ); expect(chunks.map((c) => c.toString())).toEqual(["a", "b", "c"]); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(3, 1); expect(result.success).toBe(true); expect(result.mimeType).toBe("application/json"); + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); it("returns success metadata with no chunks", async () => { - const { chunks, result } = await collect(streamFromNative(() => Promise.resolve(okMeta()))); + const nativeOperation = operation(Promise.resolve(okMeta())); + const { chunks, result } = await collect(streamFromNative(() => nativeOperation)); expect(chunks).toEqual([]); expect(result.success).toBe(true); + expect(nativeOperation.acknowledge).not.toHaveBeenCalled(); + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); - it("parks the consumer until a chunk arrives, then wakes it (backpressure)", async () => { + it("parks the consumer until a chunk arrives, then wakes it", async () => { const meta = deferred(); + const nativeOperation = operation(meta.promise); let push!: (chunk: Buffer) => void; const gen = streamFromNative((cb) => { push = cb; - return meta.promise; + return nativeOperation; }); // First pull starts the generator and parks — no chunk is ready yet. const pending = gen.next(); // Producing a chunk should wake the parked consumer. push(Buffer.from("late")); + // Callback enqueue alone does not return native credit. + expect(nativeOperation.acknowledge).not.toHaveBeenCalled(); const first = await pending; expect(first.done).toBe(false); expect(first.value!.toString()).toBe("late"); + expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(1); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(4); // Completing the stream ends the generator with the parsed metadata. meta.resolve(okMeta({ mimeType: "text/plain" })); @@ -69,22 +94,44 @@ describe("streamFromNative", () => { expect((last.value as StreamingResult).mimeType).toBe("text/plain"); }); + it("copies callback chunks before enqueueing them", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); + const source = Buffer.from("original"); + const gen = streamFromNative((cb) => { + cb(source); + source.fill(0); + return nativeOperation; + }); + + const first = await gen.next(); + expect(first.value?.toString()).toBe("original"); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(8); + await gen.next(); + }); + it("drains chunks that arrive together with completion", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); const { chunks, result } = await collect( streamFromNative((cb) => { // Chunks buffered but not yet consumed when the native call resolves. cb(Buffer.from("x")); - cb(Buffer.from("y")); - return Promise.resolve(okMeta()); + cb(Buffer.from("yy")); + return nativeOperation; }) ); - expect(chunks.map((c) => c.toString())).toEqual(["x", "y"]); + expect(chunks.map((c) => c.toString())).toEqual(["x", "yy"]); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); expect(result.success).toBe(true); }); it("propagates a failure envelope as the terminal result", async () => { + const nativeOperation = operation( + Promise.resolve(JSON.stringify({ success: false, error: "stream boom" })) + ); const { chunks, result } = await collect( - streamFromNative(() => Promise.resolve(JSON.stringify({ success: false, error: "stream boom" }))) + streamFromNative(() => nativeOperation) ); expect(chunks).toEqual([]); expect(result.success).toBe(false); @@ -92,14 +139,15 @@ describe("streamFromNative", () => { }); it("treats empty terminal metadata as a failure", async () => { - const { result } = await collect(streamFromNative(() => Promise.resolve(""))); + const { result } = await collect(streamFromNative(() => operation(Promise.resolve("")))); expect(result.success).toBe(false); expect(result.error).toBe("Empty response"); }); it("rejects a parked consumer when native start() rejects (no hang)", async () => { const startGate = deferred(); - const gen = streamFromNative(() => startGate.promise); + const nativeOperation = operation(startGate.promise); + const gen = streamFromNative(() => nativeOperation); // Park a consumer in next() BEFORE the start promise settles: no chunk is // ready and done is false, so next() awaits on pendingResolves. @@ -110,29 +158,36 @@ describe("streamFromNative", () => { startGate.reject(new Error("native start boom")); await expect(pending).rejects.toThrow("native start boom"); + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); it("drains buffered chunks, then throws, when start() rejects after pushing chunks", async () => { + const nativeOperation = operation(Promise.reject(new Error("late boom"))); const gen = streamFromNative((cb) => { cb(Buffer.from("x")); - cb(Buffer.from("y")); - return Promise.reject(new Error("late boom")); + cb(Buffer.from("yy")); + return nativeOperation; }); // Buffered chunks yield first... const a = await gen.next(); const b = await gen.next(); - expect([a.value?.toString(), b.value?.toString()]).toEqual(["x", "y"]); + expect([a.value?.toString(), b.value?.toString()]).toEqual(["x", "yy"]); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); // ...then the drained generator surfaces the start error. await expect(gen.next()).rejects.toThrow("late boom"); + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); it("propagates a native start() rejection of undefined instead of returning empty metadata", async () => { // Promise.reject(undefined) is valid JS. The old value-sentinel // (startError !== undefined) treated it as 'never rejected' and returned the // normal empty-metadata result; a settlement-state flag must propagate it (review #7 #6). - const gen = streamFromNative(() => Promise.reject(undefined)); + const gen = streamFromNative(() => operation(Promise.reject(undefined))); await expect( (async () => { // Drain fully: iterate to completion so the post-drain re-throw runs. @@ -140,4 +195,158 @@ describe("streamFromNative", () => { })() ).rejects.toBeUndefined(); }); -}); \ No newline at end of file + + it("acknowledges abandoned buffered chunks, cancels, and closes on generator return", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + cb(Buffer.from("yy")); + return nativeOperation; + }); + + const first = await gen.next(); + expect(first.value?.toString()).toBe("x"); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + + await expect(gen.return(undefined)).resolves.toEqual({ done: true, value: undefined }); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("keeps cancel and close idempotent when cancellation settles native completion", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + await gen.return(undefined); + await gen.return(undefined); + await gen.next(); + + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("closes once when registration finalization competes with generator finalization", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + const gen = streamFromNative( + (cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }, + (managedOperation) => managedOperation.cancel() + ); + + const pending = gen.next(); + completion.resolve(okMeta()); + await expect(pending).resolves.toEqual({ done: true, value: expect.any(Object) }); + + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("does not cancel when native completion wins before finalization", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + completion.resolve(okMeta()); + await Promise.resolve(); + await gen.return(undefined); + + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("still closes when cancel throws during early return", async () => { + const nativeOperation = operation(new Promise(() => {})); + nativeOperation.cancel = vi.fn(() => { + throw new Error("cancel boom"); + }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + await expect(gen.return(undefined)).rejects.toThrow("cancel boom"); + + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("still cancels and closes when abandoned-chunk acknowledgement throws", async () => { + const nativeOperation = operation(new Promise(() => {})); + vi.mocked(nativeOperation.acknowledge) + .mockImplementationOnce(() => {}) + .mockImplementationOnce(() => { + throw new Error("ack boom"); + }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + cb(Buffer.from("y")); + return nativeOperation; + }); + + await gen.next(); + await expect(gen.return(undefined)).rejects.toThrow("ack boom"); + + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("still reaches native cancellation when external cancellation credit return throws", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + vi.mocked(nativeOperation.acknowledge).mockImplementation(() => { + throw new Error("ack boom"); + }); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + let managedOperation!: NativeStreamingOperation; + let push!: (chunk: Buffer) => void; + const gen = streamFromNative( + (cb) => { + push = cb; + return nativeOperation; + }, + (started) => { managedOperation = started; } + ); + const firstPull = gen.next(); + await vi.waitFor(() => expect(managedOperation).toBeDefined()); + push(Buffer.from("x")); + + expect(() => managedOperation.cancel()).toThrow("ack boom"); + + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + await expect(firstPull).resolves.toEqual({ done: true, value: expect.any(Object) }); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("cancels and closes when operation registration throws", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + const gen = streamFromNative( + () => nativeOperation, + () => { throw new Error("registration boom"); } + ); + + await expect(gen.next()).rejects.toThrow("registration boom"); + + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); +}); From 8a1e5117c6911367901b799004f551d8383d6ec0 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 15:55:09 -0300 Subject: [PATCH 28/48] fix(node): avoid duplicate stream chunk copies --- native-lib/node/src/stream.ts | 5 ++--- native-lib/node/tests/unit/stream.test.ts | 15 --------------- 2 files changed, 2 insertions(+), 18 deletions(-) diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index 4cd516be..b8f7182b 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -38,17 +38,16 @@ export async function* streamFromNative( let settlementCloseError: unknown; const chunkCb = (chunk: Buffer) => { - const copy = Buffer.from(chunk); if ((finalized || cancellationRequested) && operation) { try { - operation.acknowledge(copy.length); + operation.acknowledge(chunk.length); } catch { // Late callback credit is best-effort after the consumer has abandoned // the stream; cleanup must still be able to cancel and close it. } return; } - chunks.push(copy); + chunks.push(chunk); // Resolve one waiting consumer if any const resolve = pendingResolves.shift(); if (resolve) { diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index 8d96a473..d3fe5915 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -94,21 +94,6 @@ describe("streamFromNative", () => { expect((last.value as StreamingResult).mimeType).toBe("text/plain"); }); - it("copies callback chunks before enqueueing them", async () => { - const nativeOperation = operation(Promise.resolve(okMeta())); - const source = Buffer.from("original"); - const gen = streamFromNative((cb) => { - cb(source); - source.fill(0); - return nativeOperation; - }); - - const first = await gen.next(); - expect(first.value?.toString()).toBe("original"); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(8); - await gen.next(); - }); - it("drains chunks that arrive together with completion", async () => { const nativeOperation = operation(Promise.resolve(okMeta())); const { chunks, result } = await collect( From f985d4ade7615b72dee1f05a08bdfe4e528b83f0 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 16:29:07 -0300 Subject: [PATCH 29/48] fix(node): harden streaming finalization --- native-lib/node/src/dataweave.ts | 118 ++++--- native-lib/node/src/stream.ts | 292 +++++++++++------- .../tests/unit/dataweave-initialize.test.ts | 89 +++++- native-lib/node/tests/unit/stream.test.ts | 171 +++++++++- 4 files changed, 507 insertions(+), 163 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index fac3bc20..97ba3591 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -180,9 +180,9 @@ export class DataWeave { // with an in-flight doCleanup() awaits that SAME promise, so the native // teardown (ffi.destroyEngine/ffi.cleanup) still happens exactly once. if (this.cleanupPromise) return this.cleanupPromise; - // Not coalescing with an in-flight cleanup: nothing to do unless we're - // "ready" (covers both never-initialized and already-settled cleanup). - if (this.state !== "ready") return; + // A lifecycle failure leaves the instance in "cleaning-up" with no live + // cleanupPromise so a later call can retry without admitting new work. + if (this.state === "uninitialized") return; this.cleanupPromise = this.doCleanup(); try { await this.cleanupPromise; @@ -199,21 +199,32 @@ export class DataWeave { // than seeing a stale "ready" state with a null engineHandle (round-6 #1/#3). this.state = "cleaning-up"; const activeStreams = [...this.activeStreams]; + let lifecycleError: unknown; for (const operation of activeStreams) { try { operation.cancel(); - } catch { - // A controller failure must not prevent the remaining streams from - // being canceled or the engine reference from being released. + } catch (error) { + lifecycleError ??= error; } } - // Cancellation may reject completion as its normal terminal signal. Wait - // for every snapshotted operation, but do not let those rejections mask an - // engine destruction or native cleanup failure. + // A failed synchronous cancellation cannot guarantee that completion will + // ever settle. Abort before waiting or destroying the engine and keep the + // instance in cleaning-up state so a later cleanup() can retry. + if (lifecycleError !== undefined) throw lifecycleError; + if (activeStreams.length > 0) { await Promise.allSettled(activeStreams.map((operation) => operation.completion)); } + for (const operation of activeStreams) { + try { + operation.close(); + } catch (error) { + lifecycleError ??= error; + } + } + if (lifecycleError !== undefined) throw lifecycleError; + let destroyError: unknown; try { if (this.engineHandle !== null) { @@ -282,16 +293,18 @@ export class DataWeave { return this.runStreamingInternal(token, script, inputs); } - private async *runStreamingInternal( + private runStreamingInternal( token: EngineOperationToken, script: string, inputs?: Inputs ): AsyncGenerator { - this.assertCurrentOperation(token); - const inputsJson = buildInputsJson(inputs ?? {}); - this.assertCurrentOperation(token); - return yield* streamFromNative( - (chunkCb) => ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb), + return streamFromNative( + (chunkCb) => { + this.assertCurrentOperation(token); + const inputsJson = buildInputsJson(inputs ?? {}); + this.assertCurrentOperation(token); + return ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb); + }, (operation) => { this.activeStreams.add(operation); }, (operation) => { this.activeStreams.delete(operation); } ); @@ -322,39 +335,58 @@ export class DataWeave { return this.runTransformInternal(token, script, input, opts); } - private async *runTransformInternal( + private runTransformInternal( token: EngineOperationToken, script: string, input: AsyncIterable | Iterable, opts?: TransformOptions ): AsyncGenerator { - this.assertCurrentOperation(token); - - const inputName = opts?.inputName ?? "payload"; - const inputMimeType = opts?.mimeType ?? "application/json"; - const inputCharset = opts?.charset ?? null; - const extraInputs = opts?.inputs ?? {}; - const inputsJson = Object.keys(extraInputs).length > 0 ? buildInputsJson(extraInputs) : "{}"; - - const readCb = await createChunkReader(input); - - this.assertCurrentOperation(token); - - return yield* streamFromNative( - (writeCb) => - ffi.runScriptTransformEngine( - token.handle, - script, - inputsJson, - inputName, - inputMimeType, - inputCharset, - readCb, - writeCb - ), - (operation) => { this.activeStreams.add(operation); }, - (operation) => { this.activeStreams.delete(operation); } - ); + let streamPromise: Promise> | null = null; + const getStream = () => { + if (!streamPromise) { + streamPromise = (async () => { + this.assertCurrentOperation(token); + + const inputName = opts?.inputName ?? "payload"; + const inputMimeType = opts?.mimeType ?? "application/json"; + const inputCharset = opts?.charset ?? null; + const extraInputs = opts?.inputs ?? {}; + const inputsJson = Object.keys(extraInputs).length > 0 ? buildInputsJson(extraInputs) : "{}"; + const readCb = await createChunkReader(input); + + this.assertCurrentOperation(token); + return streamFromNative( + (writeCb) => + ffi.runScriptTransformEngine( + token.handle, + script, + inputsJson, + inputName, + inputMimeType, + inputCharset, + readCb, + writeCb + ), + (operation) => { this.activeStreams.add(operation); }, + (operation) => { this.activeStreams.delete(operation); } + ); + })(); + } + return streamPromise; + }; + + return { + next: (...args: [] | [undefined]) => getStream().then((stream) => stream.next(...args)), + return: (value) => streamPromise + ? streamPromise.then((stream) => stream.return(value)) + : Promise.resolve({ done: true, value } as IteratorReturnResult), + throw: (error?: unknown) => streamPromise + ? streamPromise.then((stream) => stream.throw(error)) + : Promise.reject(error), + [Symbol.asyncIterator]() { + return this; + }, + }; } private captureOperationToken(): EngineOperationToken { diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index b8f7182b..6705356a 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -11,19 +11,16 @@ export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => NativeStreami /** * Bridges a native push-based streaming call into a pull-based async generator. * - * The native side pushes output chunks through the callback while - * {@link StartStreaming} runs; this generator buffers them and yields in order, - * parking the consumer when no chunk is ready and waking it on the next push or - * on completion. Consumer dequeues return native byte credits before yielding - * each chunk. After all chunks drain, it returns the parsed - * {@link StreamingResult}. + * Native async generators serialize `return()` behind an outstanding `next()`. + * This wrapper intercepts `return()` and `throw()` so they can request native + * cancellation and wake a parked pull before delegating generator finalization. * * @param start - Launches the native call and returns its operation controller. * @param onStart - Called once after native admission with the managed operation. - * @param onClose - Called once when the managed operation closes. - * @returns An async generator of output chunks whose return value is the terminal metadata. + * @param onClose - Called once after the native controller closes successfully. + * @returns An async generator of output chunks whose return value is terminal metadata. */ -export async function* streamFromNative( +export function streamFromNative( start: StartStreaming, onStart?: (operation: NativeStreamingOperation) => void, onClose?: (operation: NativeStreamingOperation) => void @@ -32,145 +29,210 @@ export async function* streamFromNative( const pendingResolves: Array<() => void> = []; let operation: NativeStreamingOperation | undefined; let nativeSettled = false; - let finalized = false; - let cancellationRequested = false; + let nativeSettlementHandled: Promise | undefined; + let nativeRejected = false; + let nativeError: unknown; let metaRaw: string | null = null; - let settlementCloseError: unknown; - - const chunkCb = (chunk: Buffer) => { - if ((finalized || cancellationRequested) && operation) { - try { - operation.acknowledge(chunk.length); - } catch { - // Late callback credit is best-effort after the consumer has abandoned - // the stream; cleanup must still be able to cancel and close it. - } - return; - } - chunks.push(chunk); - // Resolve one waiting consumer if any - const resolve = pendingResolves.shift(); - if (resolve) { - resolve(); - } - }; + let cancellationRequested = false; + let cancelSucceeded = false; + let closeSucceeded = false; + let registered = false; + let finalized = false; + let finalizationError: unknown; + let nativeOperation: NativeStreamingOperation | undefined; + let cancelInProgress = false; - let startError: unknown; - let startRejected = false; const wakeAll = () => { while (pendingResolves.length > 0) { - const resolve = pendingResolves.shift(); - if (resolve) resolve(); + pendingResolves.shift()!(); } }; + const acknowledgeBufferedChunks = () => { while (chunks.length > 0) { operation!.acknowledge(chunks.shift()!.length); } }; - try { - const nativeOperation = start(chunkCb); - let cancelCalled = false; - let closeCalled = false; - let registered = false; - operation = { - completion: nativeOperation.completion, - acknowledge: (bytes) => nativeOperation.acknowledge(bytes), - cancel: () => { - if (cancelCalled) return; - cancelCalled = true; - cancellationRequested = true; - try { - acknowledgeBufferedChunks(); - } finally { - wakeAll(); - if (!nativeSettled) nativeOperation.cancel(); - else operation!.close(); - } - }, - close: () => { - if (closeCalled) return; - closeCalled = true; - try { - nativeOperation.close(); - } finally { - if (registered) onClose?.(operation!); - } - }, - }; + const close = () => { + if (!nativeOperation || closeSucceeded) return; + nativeOperation.close(); + if (registered) onClose?.(operation!); + closeSucceeded = true; + }; - let completionHandled: Promise; + const cancel = () => { + cancellationRequested = true; + let lifecycleError: unknown; try { - // Handle both settlement branches so rejected native completion cannot - // become unhandled or leave a parked consumer asleep. Chunks already in - // the JS queue still drain before the rejection is surfaced. - completionHandled = operation.completion.then( + acknowledgeBufferedChunks(); + } catch (error) { + lifecycleError = error; + } finally { + wakeAll(); + } + + if (nativeOperation && !nativeSettled && !cancelSucceeded && !cancelInProgress) { + cancelInProgress = true; + try { + nativeOperation.cancel(); + cancelSucceeded = true; + } catch (error) { + lifecycleError ??= error; + } finally { + cancelInProgress = false; + } + } + + if (nativeOperation && (nativeSettled || cancelSucceeded)) { + try { + close(); + } catch (error) { + lifecycleError ??= error; + } + } + + if (lifecycleError !== undefined) throw lifecycleError; + }; + + const chunkCb = (chunk: Buffer) => { + if (finalized || cancellationRequested) { + try { + operation?.acknowledge(chunk.length); + } catch { + // No consumer remains to observe a late callback failure. Ownership is + // retained unless close succeeds, so DataWeave cleanup can still retry. + } + return; + } + chunks.push(chunk); + pendingResolves.shift()?.(); + }; + + const generator = (async function* (): AsyncGenerator { + let primaryError = false; + try { + nativeOperation = start(chunkCb); + const startedOperation = nativeOperation; + operation = { + completion: startedOperation.completion, + acknowledge: (bytes) => startedOperation.acknowledge(bytes), + cancel, + close, + }; + + nativeSettlementHandled = operation.completion.then( (raw) => { metaRaw = raw; nativeSettled = true; wakeAll(); - if (cancellationRequested) { - try { - operation!.close(); - } catch (error) { - settlementCloseError = error; - } - } }, (error) => { - startError = error; - startRejected = true; + nativeError = error; + nativeRejected = true; nativeSettled = true; wakeAll(); - if (cancellationRequested) { - try { - operation!.close(); - } catch (closeError) { - settlementCloseError = closeError; - } - } } ); onStart?.(operation); registered = true; - } catch (error) { - operation.cancel(); - await Promise.allSettled([operation.completion]); - throw error; - } - while (true) { - if (chunks.length > 0) { - const chunk = chunks.shift()!; - operation.acknowledge(chunk.length); - yield chunk; - continue; + while (true) { + if (!cancellationRequested && chunks.length > 0) { + const chunk = chunks.shift()!; + operation.acknowledge(chunk.length); + yield chunk; + continue; + } + if (nativeSettled) break; + if (cancellationRequested) { + return undefined as unknown as StreamingResult; + } + await new Promise((resolve) => { pendingResolves.push(resolve); }); } - if (nativeSettled) break; - await new Promise((resolve) => { pendingResolves.push(resolve); }); - } - await completionHandled; - // Track rejection by settlement state, not value: Promise.reject(undefined) - // is valid and must not be mistaken for a successful empty response. - if (startRejected) throw startError; - if (settlementCloseError !== undefined) throw settlementCloseError; - return parseStreamingResult(metaRaw ?? ""); - } finally { - finalized = true; - if (operation) { - try { - acknowledgeBufferedChunks(); - } finally { + await nativeSettlementHandled; + if (nativeRejected) throw nativeError; + if (cancellationRequested) return undefined as unknown as StreamingResult; + return parseStreamingResult(metaRaw ?? ""); + } catch (error) { + primaryError = true; + if (operation && !registered) { try { - if (!nativeSettled) operation.cancel(); - } finally { - operation.close(); + cancel(); + } catch { + // Preserve the registration failure as primary. } } + throw error; + } finally { + finalized = true; + let lifecycleError: unknown; + if (operation && registered) { + if (!nativeSettled && !cancelSucceeded && registered) { + try { + cancel(); + } catch (error) { + lifecycleError = error; + } + } + if ((nativeSettled || cancelSucceeded) && !closeSucceeded) { + try { + close(); + } catch (error) { + lifecycleError ??= error; + } + } + } + wakeAll(); + if (!primaryError && lifecycleError !== undefined) { + finalizationError = lifecycleError; + throw lifecycleError; + } } + })(); + + const requestCancellation = (): unknown => { + cancellationRequested = true; wakeAll(); - } + try { + cancel(); + return undefined; + } catch (error) { + return error; + } + }; + + const iterator: AsyncGenerator = { + next(...args: [] | [undefined]) { + return generator.next(...args); + }, + return(value) { + const lifecycleError = requestCancellation(); + return generator.return(value).then( + (result) => { + if (lifecycleError !== undefined) throw lifecycleError; + if (finalizationError !== undefined) throw finalizationError; + return result; + }, + (error) => { throw error; } + ); + }, + throw(error?: unknown) { + const lifecycleError = requestCancellation(); + return generator.throw(error).then( + (result) => { + if (lifecycleError !== undefined) throw lifecycleError; + if (finalizationError !== undefined) throw finalizationError; + return result; + }, + (primary) => { throw primary; } + ); + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + return iterator; } diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 752ea564..a93eb742 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -467,6 +467,46 @@ describe("DataWeave.initialize() native ref-count safety", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + it("does not wait on unresolved completion and permits cleanup retry when cancellation fails", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(19); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn() + .mockImplementationOnce(() => { throw new Error("cancel boom"); }) + .mockImplementationOnce(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPull = dw.runStreaming("output application/json --- [1]").next(); + const firstPullOutcome = firstPull.catch((error) => error); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1)); + + const firstCleanupOutcome = dw.cleanup().then( + () => undefined, + (error: unknown) => error + ); + const stillPending = Symbol("still pending"); + const observed = await Promise.race([ + firstCleanupOutcome, + new Promise((resolve) => setImmediate(() => resolve(stillPending))), + ]); + + // Release the pre-fix cleanup after capturing that it hung on completion. + if (observed === stillPending) completion.resolve(okStreamingMeta()); + await firstCleanupOutcome; + await firstPullOutcome; + + expect(observed).toEqual(new Error("cancel boom")); + expect(ffi.destroyEngine).not.toHaveBeenCalled(); + expect(() => dw.run("output application/json --- 1")).toThrow(/cleaning up/i); + + await expect(dw.cleanup()).resolves.toBeUndefined(); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(2); + expect(ffi.destroyEngine).toHaveBeenCalledWith(19); + expect(ffi.cleanup).toHaveBeenCalledTimes(1); + }); + it("acknowledges buffered chunks abandoned by cleanup", async () => { vi.mocked(ffi.createEngine).mockReturnValue(16); const completion = deferred(); @@ -520,6 +560,33 @@ describe("DataWeave.initialize() native ref-count safety", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + it("return cancels a transform after native admission while its first pull is pending", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(21); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform( + "output application/json --- payload", + [Buffer.from("[1]")] + ); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + + const returned = transform.return(undefined); + await expect(Promise.all([firstPull, returned])).resolves.toEqual([ + { done: true, value: undefined }, + { done: true, value: undefined }, + ]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + it("does not fail cleanup when a canceled operation rejects", async () => { vi.mocked(ffi.createEngine).mockReturnValue(14); const completion = deferred(); @@ -536,7 +603,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { completion.reject(new Error("operation canceled")); await expect(cleanupPromise).resolves.toBeUndefined(); - await expect(firstPullOutcome).resolves.toEqual(new Error("operation canceled")); + await expect(firstPullOutcome).resolves.toEqual({ done: true, value: undefined }); expect(ffi.destroyEngine).toHaveBeenCalledWith(14); expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); @@ -564,6 +631,26 @@ describe("DataWeave.initialize() native ref-count safety", () => { expect(ffi.cleanup).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + + it("retains active ownership and retries close after close failure", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(20); + const nativeOperation = operation(Promise.resolve(okStreamingMeta())); + nativeOperation.close = vi.fn() + .mockImplementationOnce(() => { throw new Error("close boom"); }) + .mockImplementationOnce(() => {}); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPull = dw.runStreaming("output application/json --- [1]").next(); + + await expect(firstPull).rejects.toThrow("close boom"); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await expect(dw.cleanup()).resolves.toBeUndefined(); + expect(nativeOperation.close).toHaveBeenCalledTimes(2); + expect(ffi.destroyEngine).toHaveBeenCalledWith(20); + }); }); describe("stale engine generation", () => { diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index d3fe5915..8824cbd4 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -66,6 +66,22 @@ describe("streamFromNative", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + it("propagates a synchronous start failure without registering an operation", async () => { + const start = vi.fn((): NativeStreamingOperation => { + throw new Error("start boom"); + }); + const onStart = vi.fn(); + const gen = streamFromNative(start, onStart); + + expect(start).not.toHaveBeenCalled(); + await expect(gen.next()).rejects.toThrow("start boom"); + + expect(start).toHaveBeenCalledTimes(1); + expect(onStart).not.toHaveBeenCalled(); + await expect(gen.return(undefined)).resolves.toEqual({ done: true, value: undefined }); + expect(start).toHaveBeenCalledTimes(1); + }); + it("parks the consumer until a chunk arrives, then wakes it", async () => { const meta = deferred(); const nativeOperation = operation(meta.promise); @@ -94,6 +110,34 @@ describe("streamFromNative", () => { expect((last.value as StreamingResult).mimeType).toBe("text/plain"); }); + it("return cancels and settles a pending first pull", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + const gen = streamFromNative(() => nativeOperation); + + let firstPullSettled = false; + const firstPull = gen.next().finally(() => { firstPullSettled = true; }); + await vi.waitFor(() => expect(nativeOperation.cancel).not.toHaveBeenCalled()); + expect(firstPullSettled).toBe(false); + + const returned = gen.return(undefined); + await Promise.resolve(); + const cancelCallsAfterReturn = vi.mocked(nativeOperation.cancel).mock.calls.length; + // Let the pre-fix serialized async generator settle so RED does not leave + // dangling promises after recording that return() could not cancel it. + if (cancelCallsAfterReturn === 0) completion.resolve(okMeta()); + + const [firstResult, returnResult] = await Promise.allSettled([firstPull, returned]); + expect(cancelCallsAfterReturn).toBe(1); + expect(firstResult.status).toBe("fulfilled"); + expect(returnResult).toEqual({ + status: "fulfilled", + value: { done: true, value: undefined }, + }); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + it("drains chunks that arrive together with completion", async () => { const nativeOperation = operation(Promise.resolve(okMeta())); const { chunks, result } = await collect( @@ -111,6 +155,22 @@ describe("streamFromNative", () => { expect(result.success).toBe(true); }); + it("acknowledges a late callback after completion and close", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); + let push!: (chunk: Buffer) => void; + const gen = streamFromNative((cb) => { + push = cb; + return nativeOperation; + }); + + await collect(gen); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + push(Buffer.from("late")); + expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(1); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(4); + }); + it("propagates a failure envelope as the terminal result", async () => { const nativeOperation = operation( Promise.resolve(JSON.stringify({ success: false, error: "stream boom" })) @@ -168,6 +228,45 @@ describe("streamFromNative", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + it("preserves native rejection when close also throws", async () => { + const nativeOperation = operation(Promise.reject(new Error("native boom"))); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative(() => nativeOperation); + + await expect(gen.next()).rejects.toThrow("native boom"); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("surfaces close failure when completion has no primary error", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative(() => nativeOperation); + + await expect(gen.next()).rejects.toThrow("close boom"); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("retries close when close finalization tracking fails", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); + const onClose = vi.fn() + .mockImplementationOnce(() => { throw new Error("unregister boom"); }) + .mockImplementationOnce(() => {}); + let managedOperation!: NativeStreamingOperation; + const gen = streamFromNative( + () => nativeOperation, + (started) => { managedOperation = started; }, + onClose + ); + + await expect(gen.next()).rejects.toThrow("unregister boom"); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + expect(onClose).toHaveBeenCalledTimes(1); + + managedOperation.close(); + expect(nativeOperation.close).toHaveBeenCalledTimes(2); + expect(onClose).toHaveBeenCalledTimes(2); + }); + it("propagates a native start() rejection of undefined instead of returning empty metadata", async () => { // Promise.reject(undefined) is valid JS. The old value-sentinel // (startError !== undefined) treated it as 'never rejected' and returned the @@ -232,7 +331,7 @@ describe("streamFromNative", () => { const pending = gen.next(); completion.resolve(okMeta()); - await expect(pending).resolves.toEqual({ done: true, value: expect.any(Object) }); + await expect(pending).resolves.toEqual({ done: true, value: undefined }); expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); @@ -256,7 +355,7 @@ describe("streamFromNative", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); - it("still closes when cancel throws during early return", async () => { + it("retries cancel once without closing an operation whose cancellation failed", async () => { const nativeOperation = operation(new Promise(() => {})); nativeOperation.cancel = vi.fn(() => { throw new Error("cancel boom"); @@ -269,8 +368,24 @@ describe("streamFromNative", () => { await gen.next(); await expect(gen.return(undefined)).rejects.toThrow("cancel boom"); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(2); + expect(nativeOperation.close).not.toHaveBeenCalled(); + }); + + it("preserves a consumer error when close also throws", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + await expect(gen.throw(new Error("consumer boom"))).rejects.toThrow("consumer boom"); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); - expect(nativeOperation.close).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(2); }); it("still cancels and closes when abandoned-chunk acknowledgement throws", async () => { @@ -316,7 +431,7 @@ describe("streamFromNative", () => { expect(() => managedOperation.cancel()).toThrow("ack boom"); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); - await expect(firstPull).resolves.toEqual({ done: true, value: expect.any(Object) }); + await expect(firstPull).resolves.toEqual({ done: true, value: undefined }); expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); @@ -334,4 +449,52 @@ describe("streamFromNative", () => { expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + + it("preserves registration failure when cancel and close also throw", async () => { + const nativeOperation = operation(new Promise(() => {})); + nativeOperation.cancel = vi.fn(() => { throw new Error("cancel boom"); }); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative( + () => nativeOperation, + () => { throw new Error("registration boom"); } + ); + + await expect(gen.next()).rejects.toThrow("registration boom"); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).not.toHaveBeenCalled(); + }); + + it("preserves registration failure when successful cancel is followed by close failure", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative( + () => nativeOperation, + () => { throw new Error("registration boom"); } + ); + + await expect(gen.next()).rejects.toThrow("registration boom"); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("preserves a for-await body error when close also throws", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await expect((async () => { + for await (const _chunk of gen) { + throw new Error("body boom"); + } + })()).rejects.toThrow("body boom"); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(2); + }); }); From bc24ed0b692db70e9e44404cb91bb00245601d71 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 16:49:35 -0300 Subject: [PATCH 30/48] fix(node): close lazy streaming iterators safely --- native-lib/node/src/dataweave.ts | 49 +++++--- native-lib/node/src/stream.ts | 54 +++++---- .../tests/unit/dataweave-initialize.test.ts | 106 +++++++++++++++++- native-lib/node/tests/unit/stream.test.ts | 82 +++++++++++++- 4 files changed, 247 insertions(+), 44 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 97ba3591..59c73caa 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -199,18 +199,20 @@ export class DataWeave { // than seeing a stale "ready" state with a null engineHandle (round-6 #1/#3). this.state = "cleaning-up"; const activeStreams = [...this.activeStreams]; - let lifecycleError: unknown; + let lifecycleError: { readonly hasError: false } | { readonly hasError: true; readonly error: unknown } = { + hasError: false, + }; for (const operation of activeStreams) { try { operation.cancel(); } catch (error) { - lifecycleError ??= error; + if (!lifecycleError.hasError) lifecycleError = { hasError: true, error }; } } // A failed synchronous cancellation cannot guarantee that completion will // ever settle. Abort before waiting or destroying the engine and keep the // instance in cleaning-up state so a later cleanup() can retry. - if (lifecycleError !== undefined) throw lifecycleError; + if (lifecycleError.hasError) throw lifecycleError.error; if (activeStreams.length > 0) { await Promise.allSettled(activeStreams.map((operation) => operation.completion)); @@ -220,12 +222,14 @@ export class DataWeave { try { operation.close(); } catch (error) { - lifecycleError ??= error; + if (!lifecycleError.hasError) lifecycleError = { hasError: true, error }; } } - if (lifecycleError !== undefined) throw lifecycleError; + if (lifecycleError.hasError) throw lifecycleError.error; - let destroyError: unknown; + let destroyError: { readonly hasError: false } | { readonly hasError: true; readonly error: unknown } = { + hasError: false, + }; try { if (this.engineHandle !== null) { try { @@ -236,7 +240,7 @@ export class DataWeave { // env's native init reference and block isolate teardown. Capture the // primary error, clear the handle so a retry does not double-destroy, // and fall through to release the reference below. - destroyError = e; + destroyError = { hasError: true, error: e }; } finally { this.engineHandle = null; } @@ -249,7 +253,7 @@ export class DataWeave { // ffi.cleanup() itself rejected, its error already propagated from the await // (the more actionable reference-release failure wins; the destroy error is // then suppressed). - if (destroyError !== undefined) throw destroyError; + if (destroyError.hasError) throw destroyError.error; } /** @@ -342,6 +346,7 @@ export class DataWeave { opts?: TransformOptions ): AsyncGenerator { let streamPromise: Promise> | null = null; + let closed = false; const getStream = () => { if (!streamPromise) { streamPromise = (async () => { @@ -376,13 +381,27 @@ export class DataWeave { }; return { - next: (...args: [] | [undefined]) => getStream().then((stream) => stream.next(...args)), - return: (value) => streamPromise - ? streamPromise.then((stream) => stream.return(value)) - : Promise.resolve({ done: true, value } as IteratorReturnResult), - throw: (error?: unknown) => streamPromise - ? streamPromise.then((stream) => stream.throw(error)) - : Promise.reject(error), + next: (...args: [] | [undefined]) => closed + ? Promise.resolve( + { done: true, value: undefined } as unknown as IteratorReturnResult + ) + : getStream().then((stream) => stream.next(...args)), + return: async (value) => { + if (closed) { + return { done: true, value: await value } as IteratorReturnResult; + } + closed = true; + return streamPromise + ? streamPromise.then((stream) => stream.return(value)) + : { done: true, value: await value } as IteratorReturnResult; + }, + throw: (error?: unknown) => { + if (closed) return Promise.reject(error); + closed = true; + return streamPromise + ? streamPromise.then((stream) => stream.throw(error)) + : Promise.reject(error); + }, [Symbol.asyncIterator]() { return this; }, diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index 6705356a..7eb99b48 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -8,6 +8,10 @@ import type { StreamingResult } from "./types"; */ export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => NativeStreamingOperation; +type ErrorState = { readonly hasError: false } | { readonly hasError: true; readonly error: unknown }; + +const NO_ERROR: ErrorState = { hasError: false }; + /** * Bridges a native push-based streaming call into a pull-based async generator. * @@ -35,10 +39,11 @@ export function streamFromNative( let metaRaw: string | null = null; let cancellationRequested = false; let cancelSucceeded = false; - let closeSucceeded = false; + let nativeCloseSucceeded = false; + let closeFinalized = false; let registered = false; let finalized = false; - let finalizationError: unknown; + let finalizationError: ErrorState = NO_ERROR; let nativeOperation: NativeStreamingOperation | undefined; let cancelInProgress = false; @@ -55,19 +60,22 @@ export function streamFromNative( }; const close = () => { - if (!nativeOperation || closeSucceeded) return; - nativeOperation.close(); + if (!nativeOperation || closeFinalized) return; + if (!nativeCloseSucceeded) { + nativeOperation.close(); + nativeCloseSucceeded = true; + } if (registered) onClose?.(operation!); - closeSucceeded = true; + closeFinalized = true; }; const cancel = () => { cancellationRequested = true; - let lifecycleError: unknown; + let lifecycleError: ErrorState = NO_ERROR; try { acknowledgeBufferedChunks(); } catch (error) { - lifecycleError = error; + lifecycleError = { hasError: true, error }; } finally { wakeAll(); } @@ -78,7 +86,7 @@ export function streamFromNative( nativeOperation.cancel(); cancelSucceeded = true; } catch (error) { - lifecycleError ??= error; + if (!lifecycleError.hasError) lifecycleError = { hasError: true, error }; } finally { cancelInProgress = false; } @@ -88,11 +96,11 @@ export function streamFromNative( try { close(); } catch (error) { - lifecycleError ??= error; + if (!lifecycleError.hasError) lifecycleError = { hasError: true, error }; } } - if (lifecycleError !== undefined) throw lifecycleError; + if (lifecycleError.hasError) throw lifecycleError.error; }; const chunkCb = (chunk: Buffer) => { @@ -168,39 +176,39 @@ export function streamFromNative( throw error; } finally { finalized = true; - let lifecycleError: unknown; + let lifecycleError: ErrorState = NO_ERROR; if (operation && registered) { if (!nativeSettled && !cancelSucceeded && registered) { try { cancel(); } catch (error) { - lifecycleError = error; + lifecycleError = { hasError: true, error }; } } - if ((nativeSettled || cancelSucceeded) && !closeSucceeded) { + if ((nativeSettled || cancelSucceeded) && !closeFinalized) { try { close(); } catch (error) { - lifecycleError ??= error; + if (!lifecycleError.hasError) lifecycleError = { hasError: true, error }; } } } wakeAll(); - if (!primaryError && lifecycleError !== undefined) { + if (!primaryError && lifecycleError.hasError) { finalizationError = lifecycleError; - throw lifecycleError; + throw lifecycleError.error; } } })(); - const requestCancellation = (): unknown => { + const requestCancellation = (): ErrorState => { cancellationRequested = true; wakeAll(); try { cancel(); - return undefined; + return NO_ERROR; } catch (error) { - return error; + return { hasError: true, error }; } }; @@ -212,8 +220,8 @@ export function streamFromNative( const lifecycleError = requestCancellation(); return generator.return(value).then( (result) => { - if (lifecycleError !== undefined) throw lifecycleError; - if (finalizationError !== undefined) throw finalizationError; + if (lifecycleError.hasError) throw lifecycleError.error; + if (finalizationError.hasError) throw finalizationError.error; return result; }, (error) => { throw error; } @@ -223,8 +231,8 @@ export function streamFromNative( const lifecycleError = requestCancellation(); return generator.throw(error).then( (result) => { - if (lifecycleError !== undefined) throw lifecycleError; - if (finalizationError !== undefined) throw finalizationError; + if (lifecycleError.hasError) throw lifecycleError.error; + if (finalizationError.hasError) throw finalizationError.error; return result; }, (primary) => { throw primary; } diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index a93eb742..0dcd1d71 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -587,6 +587,60 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("return before first transform pull closes permanently without preparing or admitting work", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(22); + let inputReads = 0; + const input = { + *[Symbol.iterator]() { + inputReads++; + yield Buffer.from("[1]"); + }, + }; + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", input); + + await expect(transform.return(undefined)).resolves.toEqual({ done: true, value: undefined }); + await expect(transform.next()).resolves.toEqual({ done: true, value: undefined }); + expect(inputReads).toBe(0); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("assimilates a promised return value before first transform pull", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(23); + const promisedResult = Promise.resolve({ success: true } as StreamingResult); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + + const returned = await (transform.return as (value: unknown) => Promise>)(promisedResult); + expect(returned).toEqual({ done: true, value: { success: true } }); + expect(returned.value).not.toBe(promisedResult); + await expect(transform.next()).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("throw before first transform pull closes permanently without admission", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(24); + const thrown = new Error("early throw"); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + + await expect(transform.throw(thrown)).rejects.toBe(thrown); + await expect(transform.next()).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + it("does not fail cleanup when a canceled operation rejects", async () => { vi.mocked(ffi.createEngine).mockReturnValue(14); const completion = deferred(); @@ -636,7 +690,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { vi.mocked(ffi.createEngine).mockReturnValue(20); const nativeOperation = operation(Promise.resolve(okStreamingMeta())); nativeOperation.close = vi.fn() - .mockImplementationOnce(() => { throw new Error("close boom"); }) + .mockImplementationOnce(() => { throw undefined; }) .mockImplementationOnce(() => {}); vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); @@ -644,13 +698,61 @@ describe("DataWeave.initialize() native ref-count safety", () => { dw.initialize(); const firstPull = dw.runStreaming("output application/json --- [1]").next(); - await expect(firstPull).rejects.toThrow("close boom"); + const firstPullResult = await firstPull.then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + expect(firstPullResult).toEqual({ status: "rejected", reason: undefined }); expect(nativeOperation.close).toHaveBeenCalledTimes(1); await expect(dw.cleanup()).resolves.toBeUndefined(); expect(nativeOperation.close).toHaveBeenCalledTimes(2); expect(ffi.destroyEngine).toHaveBeenCalledWith(20); }); + + it("does not wait on unresolved completion when cancellation throws undefined", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(25); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => { throw undefined; }); + vi.mocked(ffi.runScriptStreamingEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPull = dw.runStreaming("output application/json --- [1]").next(); + firstPull.catch(() => {}); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(1)); + + const cleanupOutcome = dw.cleanup().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + const stillPending = Symbol("still pending"); + const observed = await Promise.race([ + cleanupOutcome, + new Promise((resolve) => setImmediate(() => resolve(stillPending))), + ]); + + if (observed === stillPending) completion.resolve(okStreamingMeta()); + expect(observed).toEqual({ status: "rejected", reason: undefined }); + expect(ffi.destroyEngine).not.toHaveBeenCalled(); + }); + + it("surfaces destroyEngine throwing undefined after releasing the native reference", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(26); + vi.mocked(ffi.destroyEngine).mockImplementation(() => { throw undefined; }); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + + const cleanupResult = await dw.cleanup().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(cleanupResult).toEqual({ status: "rejected", reason: undefined }); + expect(ffi.cleanup).toHaveBeenCalledTimes(1); + }); }); describe("stale engine generation", () => { diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index 8824cbd4..18bdad1a 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -246,10 +246,10 @@ describe("streamFromNative", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); - it("retries close when close finalization tracking fails", async () => { + it("retries only close finalization tracking after native close succeeds", async () => { const nativeOperation = operation(Promise.resolve(okMeta())); const onClose = vi.fn() - .mockImplementationOnce(() => { throw new Error("unregister boom"); }) + .mockImplementationOnce(() => { throw undefined; }) .mockImplementationOnce(() => {}); let managedOperation!: NativeStreamingOperation; const gen = streamFromNative( @@ -258,15 +258,69 @@ describe("streamFromNative", () => { onClose ); - await expect(gen.next()).rejects.toThrow("unregister boom"); + const next = await gen.next().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + expect(next).toEqual({ status: "rejected", reason: undefined }); expect(nativeOperation.close).toHaveBeenCalledTimes(1); expect(onClose).toHaveBeenCalledTimes(1); managedOperation.close(); - expect(nativeOperation.close).toHaveBeenCalledTimes(2); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); expect(onClose).toHaveBeenCalledTimes(2); }); + it.each([ + { label: "undefined", thrown: undefined }, + { label: "null", thrown: null }, + ])("treats cancel throwing $label as a retryable lifecycle failure", async ({ thrown }) => { + const nativeOperation = operation(new Promise(() => {})); + nativeOperation.cancel = vi.fn(() => { throw thrown; }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + const returned = await gen.return(undefined).then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(returned).toEqual({ status: "rejected", reason: thrown }); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(2); + expect(nativeOperation.close).not.toHaveBeenCalled(); + }); + + it("treats close throwing undefined as a lifecycle failure without a primary error", async () => { + const nativeOperation = operation(Promise.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw undefined; }); + const gen = streamFromNative(() => nativeOperation); + + const next = await gen.next().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(next).toEqual({ status: "rejected", reason: undefined }); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + + it("preserves null native rejection when close throws undefined", async () => { + const nativeOperation = operation(Promise.reject(null)); + nativeOperation.close = vi.fn(() => { throw undefined; }); + const gen = streamFromNative(() => nativeOperation); + + const next = await gen.next().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(next).toEqual({ status: "rejected", reason: null }); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + it("propagates a native start() rejection of undefined instead of returning empty metadata", async () => { // Promise.reject(undefined) is valid JS. The old value-sentinel // (startError !== undefined) treated it as 'never rejected' and returned the @@ -479,6 +533,26 @@ describe("streamFromNative", () => { expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); + it("preserves undefined registration failure over a null close failure", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + nativeOperation.close = vi.fn(() => { throw null; }); + const gen = streamFromNative( + () => nativeOperation, + () => { throw undefined; } + ); + + const next = await gen.next().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(next).toEqual({ status: "rejected", reason: undefined }); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + }); + it("preserves a for-await body error when close also throws", async () => { const completion = deferred(); const nativeOperation = operation(completion.promise); From 7733da8204b398ec0639d3aa1123d5678462e024 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Mon, 7 Sep 2026 17:21:25 -0300 Subject: [PATCH 31/48] fix(node): serialize transform setup cleanup --- native-lib/node/src/dataweave.ts | 143 ++++++++++++++---- .../tests/unit/dataweave-initialize.test.ts | 126 +++++++++++++++ 2 files changed, 241 insertions(+), 28 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 59c73caa..3b37c3c7 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -14,6 +14,11 @@ interface EngineOperationToken { readonly generation: number; } +interface ActiveStreamState { + completionSettled: boolean; + closeFinalized: boolean; +} + /** * Constructor options for {@link DataWeave}. */ @@ -60,6 +65,7 @@ export class DataWeave { private engineHandle: number | null = null; private engineGeneration = 0; private readonly activeStreams = new Set(); + private readonly activeStreamStates = new Map(); private cleanupPromise: Promise | null = null; /** @@ -219,6 +225,8 @@ export class DataWeave { } for (const operation of activeStreams) { + const streamState = this.activeStreamStates.get(operation); + if (!streamState || streamState.closeFinalized) continue; try { operation.close(); } catch (error) { @@ -309,8 +317,8 @@ export class DataWeave { this.assertCurrentOperation(token); return ffi.runScriptStreamingEngine(token.handle, script, inputsJson, chunkCb); }, - (operation) => { this.activeStreams.add(operation); }, - (operation) => { this.activeStreams.delete(operation); } + (operation) => { this.registerActiveStream(operation); }, + (operation) => { this.markActiveStreamClosed(operation); } ); } @@ -345,11 +353,26 @@ export class DataWeave { input: AsyncIterable | Iterable, opts?: TransformOptions ): AsyncGenerator { - let streamPromise: Promise> | null = null; - let closed = false; - const getStream = () => { - if (!streamPromise) { - streamPromise = (async () => { + type TransformState = + | { readonly kind: "open" } + | { readonly kind: "returning" } + | { readonly kind: "throwing" } + | { readonly kind: "closed" }; + + let state: TransformState = { kind: "open" }; + let stream: AsyncGenerator | null = null; + let setupPromise: Promise | null> | null = null; + let requestTail: Promise = Promise.resolve(); + + const enqueue = (request: () => Promise): Promise => { + const result = requestTail.then(request, request); + requestTail = result.then(() => undefined, () => undefined); + return result; + }; + + const setup = (): Promise | null> => { + if (!setupPromise) { + setupPromise = (async () => { this.assertCurrentOperation(token); const inputName = opts?.inputName ?? "payload"; @@ -359,8 +382,9 @@ export class DataWeave { const inputsJson = Object.keys(extraInputs).length > 0 ? buildInputsJson(extraInputs) : "{}"; const readCb = await createChunkReader(input); + if (state.kind !== "open") return null; this.assertCurrentOperation(token); - return streamFromNative( + stream = streamFromNative( (writeCb) => ffi.runScriptTransformEngine( token.handle, @@ -372,35 +396,66 @@ export class DataWeave { readCb, writeCb ), - (operation) => { this.activeStreams.add(operation); }, - (operation) => { this.activeStreams.delete(operation); } + (operation) => { this.registerActiveStream(operation); }, + (operation) => { this.markActiveStreamClosed(operation); } ); + return stream; })(); + // The first queued next observes this rejection. This second branch + // prevents a concurrent return/throw from creating an unhandled promise. + setupPromise.catch(() => {}); } - return streamPromise; + return setupPromise; }; return { - next: (...args: [] | [undefined]) => closed - ? Promise.resolve( - { done: true, value: undefined } as unknown as IteratorReturnResult - ) - : getStream().then((stream) => stream.next(...args)), - return: async (value) => { - if (closed) { - return { done: true, value: await value } as IteratorReturnResult; + next: (...args: [] | [undefined]) => enqueue(async () => { + if (state.kind !== "open") { + return { done: true, value: undefined } as unknown as IteratorReturnResult; } - closed = true; - return streamPromise - ? streamPromise.then((stream) => stream.return(value)) - : { done: true, value: await value } as IteratorReturnResult; + let activeStream: AsyncGenerator | null; + try { + activeStream = await setup(); + } catch (error) { + if (state.kind !== "open") { + return { done: true, value: undefined } as unknown as IteratorReturnResult; + } + state = { kind: "closed" }; + throw error; + } + if (!activeStream || state.kind !== "open") { + return { done: true, value: undefined } as unknown as IteratorReturnResult; + } + const result = await activeStream.next(...args); + if (result.done) state = { kind: "closed" }; + return result; + }), + return: (value) => { + if (state.kind === "open") state = { kind: "returning" }; + const proactiveReturn = stream?.return(value); + return enqueue(async () => { + try { + if (proactiveReturn) return await proactiveReturn; + return { + done: true, + value: await value, + } as IteratorReturnResult; + } finally { + state = { kind: "closed" }; + } + }); }, throw: (error?: unknown) => { - if (closed) return Promise.reject(error); - closed = true; - return streamPromise - ? streamPromise.then((stream) => stream.throw(error)) - : Promise.reject(error); + if (state.kind === "open") state = { kind: "throwing" }; + const proactiveThrow = stream?.throw(error); + return enqueue(async () => { + try { + if (proactiveThrow) return await proactiveThrow; + throw error; + } finally { + state = { kind: "closed" }; + } + }); }, [Symbol.asyncIterator]() { return this; @@ -408,6 +463,38 @@ export class DataWeave { }; } + private registerActiveStream(operation: NativeStreamingOperation): void { + const state: ActiveStreamState = { completionSettled: false, closeFinalized: false }; + this.activeStreams.add(operation); + this.activeStreamStates.set(operation, state); + operation.completion.then( + () => { + state.completionSettled = true; + this.releaseActiveStream(operation, state); + }, + () => { + state.completionSettled = true; + this.releaseActiveStream(operation, state); + } + ); + } + + private markActiveStreamClosed(operation: NativeStreamingOperation): void { + const state = this.activeStreamStates.get(operation); + if (!state) return; + state.closeFinalized = true; + this.releaseActiveStream(operation, state); + } + + private releaseActiveStream( + operation: NativeStreamingOperation, + state: ActiveStreamState + ): void { + if (!state.completionSettled || !state.closeFinalized) return; + this.activeStreams.delete(operation); + this.activeStreamStates.delete(operation); + } + private captureOperationToken(): EngineOperationToken { this.ensureReady(); return { handle: this.engineHandle!, generation: this.engineGeneration }; diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 0dcd1d71..2362a3b9 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -16,10 +16,15 @@ vi.mock("../../src/ffi", () => ({ runScriptTransformEngine: vi.fn(), cleanup: vi.fn(), })); +vi.mock("../../src/reader", async () => { + const actual = await vi.importActual("../../src/reader"); + return { ...actual, createChunkReader: vi.fn(actual.createChunkReader) }; +}); import * as ffi from "../../src/ffi"; import { DataWeave, run, cleanup } from "../../src/dataweave"; import { DataWeaveError } from "../../src/errors"; +import { createChunkReader } from "../../src/reader"; import type { NativeStreamingOperation } from "../../src/ffi"; function deferred() { @@ -54,6 +59,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { vi.mocked(ffi.runScriptStreamingEngine).mockReset(); vi.mocked(ffi.runScriptTransformEngine).mockReset(); vi.mocked(ffi.cleanup).mockReset(); + vi.mocked(createChunkReader).mockClear(); }); it("releases the native library ref-count if engine creation fails after ffi.initialize() succeeded", () => { @@ -507,6 +513,53 @@ describe("DataWeave.initialize() native ref-count safety", () => { expect(ffi.cleanup).toHaveBeenCalledTimes(1); }); + it("retry cleanup still awaits an earlier successfully canceled operation", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(27); + const completionA = deferred(); + const operationA = operation(completionA.promise); + const completionB = deferred(); + const operationB = operation(completionB.promise); + operationB.cancel = vi.fn() + .mockImplementationOnce(() => { throw undefined; }) + .mockImplementationOnce(() => completionB.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptStreamingEngine) + .mockReturnValueOnce(operationA) + .mockReturnValueOnce(operationB); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const firstPullA = dw.runStreaming("output application/json --- [1]").next(); + const firstPullB = dw.runStreaming("output application/json --- [2]").next(); + const firstPullAOutcome = firstPullA.catch((error) => error); + const firstPullBOutcome = firstPullB.catch((error) => error); + await vi.waitFor(() => expect(ffi.runScriptStreamingEngine).toHaveBeenCalledTimes(2)); + + const firstCleanup = await dw.cleanup().then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + expect(firstCleanup).toEqual({ status: "rejected", reason: undefined }); + expect(operationA.cancel).toHaveBeenCalledTimes(1); + expect(operationA.close).toHaveBeenCalledTimes(1); + expect(operationB.cancel).toHaveBeenCalledTimes(2); + expect(ffi.destroyEngine).not.toHaveBeenCalled(); + + const retry = dw.cleanup(); + expect(operationA.cancel).toHaveBeenCalledTimes(1); + expect(operationB.cancel).toHaveBeenCalledTimes(2); + const retryState = Symbol("retry pending"); + expect(await Promise.race([ + retry.then(() => "settled"), + Promise.resolve(retryState), + ])).toBe(retryState); + expect(ffi.destroyEngine).not.toHaveBeenCalled(); + + completionA.resolve(okStreamingMeta()); + await retry; + await Promise.all([firstPullAOutcome, firstPullBOutcome]); + expect(ffi.destroyEngine).toHaveBeenCalledWith(27); + }); + it("acknowledges buffered chunks abandoned by cleanup", async () => { vi.mocked(ffi.createEngine).mockReturnValue(16); const completion = deferred(); @@ -641,6 +694,79 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("return during pending transform setup wins over setup rejection", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(28); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const returned = transform.return(undefined); + readerGate.reject(new Error("setup boom")); + + await expect(firstPull).resolves.toEqual({ done: true, value: undefined }); + await expect(returned).resolves.toEqual({ done: true, value: undefined }); + await expect(transform.next()).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("throw during pending transform setup wins over setup rejection", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(29); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown); + readerGate.reject(new Error("setup boom")); + + await expect(firstPull).resolves.toEqual({ done: true, value: undefined }); + await expect(throwing).rejects.toBe(thrown); + await expect(transform.next()).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("serializes next behind promised return assimilation during transform setup", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(30); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + const returnGate = deferred(); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const returned = (transform.return as (value: unknown) => Promise>)(returnGate.promise); + const laterNext = transform.next(); + + readerGate.resolve(() => null); + await expect(firstPull).resolves.toEqual({ done: true, value: undefined }); + const stillPending = Symbol("still pending"); + expect(await Promise.race([ + laterNext.then(() => "settled"), + new Promise((resolve) => setImmediate(() => resolve(stillPending))), + ])).toBe(stillPending); + + returnGate.resolve({ success: true } as StreamingResult); + await expect(returned).resolves.toEqual({ done: true, value: { success: true } }); + await expect(laterNext).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + it("does not fail cleanup when a canceled operation rejects", async () => { vi.mocked(ffi.createEngine).mockReturnValue(14); const completion = deferred(); From 8f932af5950f397761940a77549d584725079b35 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Tue, 8 Sep 2026 10:15:09 -0300 Subject: [PATCH 32/48] fix(node): order transform cancellation safely --- native-lib/node/src/dataweave.ts | 155 ++++++++++----- native-lib/node/src/stream.ts | 7 +- .../tests/unit/dataweave-initialize.test.ts | 185 ++++++++++++++++++ native-lib/node/tests/unit/stream.test.ts | 21 ++ 4 files changed, 315 insertions(+), 53 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 3b37c3c7..f3eb3fbd 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -353,26 +353,57 @@ export class DataWeave { input: AsyncIterable | Iterable, opts?: TransformOptions ): AsyncGenerator { - type TransformState = - | { readonly kind: "open" } - | { readonly kind: "returning" } - | { readonly kind: "throwing" } - | { readonly kind: "closed" }; + type QueuedRequest = { + readonly run: () => Promise; + readonly resolve: (result: T) => void; + readonly reject: (error: unknown) => void; + }; - let state: TransformState = { kind: "open" }; + let closed = false; + let controlPending = false; let stream: AsyncGenerator | null = null; - let setupPromise: Promise | null> | null = null; - let requestTail: Promise = Promise.resolve(); + let setupPromise: Promise | null>; + let setupAbandoned = false; + let abandonSetup = () => { setupAbandoned = true; }; + const requestQueue: Array> = []; + let requestRunning = false; const enqueue = (request: () => Promise): Promise => { - const result = requestTail.then(request, request); - requestTail = result.then(() => undefined, () => undefined); + const result = new Promise((resolve, reject) => { + requestQueue.push({ run: request, resolve, reject } as QueuedRequest); + }); + drainRequests(); return result; }; - const setup = (): Promise | null> => { - if (!setupPromise) { - setupPromise = (async () => { + const drainRequests = (): void => { + if (requestRunning) return; + const request = requestQueue.shift(); + if (!request) return; + requestRunning = true; + let result: Promise; + try { + result = request.run(); + } catch (error) { + result = Promise.reject(error); + } + result.then(request.resolve, request.reject).finally(() => { + requestRunning = false; + drainRequests(); + }); + }; + + const setup = (): Promise | null> => + setupPromise ??= new Promise((resolve, reject) => { + abandonSetup = () => { + setupAbandoned = true; + resolve(null); + }; + try { + if (controlPending) { + abandonSetup(); + return; + } this.assertCurrentOperation(token); const inputName = opts?.inputName ?? "payload"; @@ -380,80 +411,100 @@ export class DataWeave { const inputCharset = opts?.charset ?? null; const extraInputs = opts?.inputs ?? {}; const inputsJson = Object.keys(extraInputs).length > 0 ? buildInputsJson(extraInputs) : "{}"; - const readCb = await createChunkReader(input); - - if (state.kind !== "open") return null; - this.assertCurrentOperation(token); - stream = streamFromNative( - (writeCb) => - ffi.runScriptTransformEngine( - token.handle, - script, - inputsJson, - inputName, - inputMimeType, - inputCharset, - readCb, - writeCb - ), - (operation) => { this.registerActiveStream(operation); }, - (operation) => { this.markActiveStreamClosed(operation); } + createChunkReader(input).then( + (readCb) => { + if (setupAbandoned || controlPending) return; + try { + this.assertCurrentOperation(token); + stream = streamFromNative( + (writeCb) => { + this.assertCurrentOperation(token); + return ffi.runScriptTransformEngine( + token.handle, + script, + inputsJson, + inputName, + inputMimeType, + inputCharset, + readCb, + writeCb + ); + }, + (operation) => { this.registerActiveStream(operation); }, + (operation) => { this.markActiveStreamClosed(operation); } + ); + resolve(stream); + } catch (error) { + reject(error); + } + }, + (error) => { + if (!setupAbandoned) reject(error); + } ); - return stream; - })(); - // The first queued next observes this rejection. This second branch - // prevents a concurrent return/throw from creating an unhandled promise. - setupPromise.catch(() => {}); - } - return setupPromise; - }; + } catch (error) { + reject(error); + } + }); return { next: (...args: [] | [undefined]) => enqueue(async () => { - if (state.kind !== "open") { + if (closed) { return { done: true, value: undefined } as unknown as IteratorReturnResult; } let activeStream: AsyncGenerator | null; try { activeStream = await setup(); } catch (error) { - if (state.kind !== "open") { - return { done: true, value: undefined } as unknown as IteratorReturnResult; - } - state = { kind: "closed" }; + closed = true; throw error; } - if (!activeStream || state.kind !== "open") { + if (!activeStream) { return { done: true, value: undefined } as unknown as IteratorReturnResult; } const result = await activeStream.next(...args); - if (result.done) state = { kind: "closed" }; + if (result.done) closed = true; return result; }), return: (value) => { - if (state.kind === "open") state = { kind: "returning" }; - const proactiveReturn = stream?.return(value); + const isPrimaryControl = !closed && !controlPending; + if (isPrimaryControl) controlPending = true; + // Let an earlier delegated next() consume an already-buffered chunk + // before cancellation wakes it if it is genuinely parked. + const proactiveReturn = isPrimaryControl && stream + ? Promise.resolve().then(() => stream!.return(value)) + : null; + proactiveReturn?.catch(() => {}); + if (isPrimaryControl && !stream) abandonSetup(); return enqueue(async () => { try { if (proactiveReturn) return await proactiveReturn; + if (stream && !closed) return await stream.return(value); return { done: true, value: await value, } as IteratorReturnResult; } finally { - state = { kind: "closed" }; + closed = true; } }); }, throw: (error?: unknown) => { - if (state.kind === "open") state = { kind: "throwing" }; - const proactiveThrow = stream?.throw(error); + const isPrimaryControl = !closed && !controlPending; + if (isPrimaryControl) controlPending = true; + // Preserve the same request-ordering turn as return(). + const proactiveThrow = isPrimaryControl && stream + ? Promise.resolve().then(() => stream!.throw(error)) + : null; + proactiveThrow?.catch(() => {}); + if (isPrimaryControl && !stream) abandonSetup(); return enqueue(async () => { try { if (proactiveThrow) return await proactiveThrow; + if (stream && !closed) return await stream.throw(error); throw error; } finally { - state = { kind: "closed" }; + closed = true; } }); }, diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index 7eb99b48..e631c4f3 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -224,7 +224,12 @@ export function streamFromNative( if (finalizationError.hasError) throw finalizationError.error; return result; }, - (error) => { throw error; } + (error) => { + // A retry failure from generator finalization cannot replace the + // lifecycle failure observed by this cancellation request. + if (lifecycleError.hasError) throw lifecycleError.error; + throw error; + } ); }, throw(error?: unknown) { diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 2362a3b9..6fd54032 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -662,6 +662,29 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("keeps next before return ordered when transform setup has not started", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(36); + const nativeOperation = operation(Promise.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("x")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + const returned = transform.return(undefined); + + await expect(firstPull).resolves.toEqual({ done: true, value: undefined }); + await expect(returned).resolves.toEqual({ done: true, value: undefined }); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + it("assimilates a promised return value before first transform pull", async () => { vi.mocked(ffi.createEngine).mockReturnValue(23); const promisedResult = Promise.resolve({ success: true } as StreamingResult); @@ -715,6 +738,145 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("return abandons unresolved async transform setup without waiting for input", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(31); + const inputNext = deferred>(); + const inputStarted = deferred(); + const input = { + [Symbol.asyncIterator]() { + return { + next() { + inputStarted.resolve(); + return inputNext.promise; + }, + }; + }, + }; + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", input); + const firstPull = transform.next(); + await inputStarted.promise; + + const returned = transform.return(undefined); + const pending = Symbol("pending setup"); + const observed = await Promise.race([ + Promise.allSettled([firstPull, returned]), + new Promise((resolve) => setImmediate(() => resolve(pending))), + ]); + + inputNext.resolve({ done: true, value: undefined }); + await Promise.allSettled([firstPull, returned]); + await dw.cleanup(); + + expect(observed).toEqual([ + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "fulfilled", value: { done: true, value: undefined } }, + ]); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + }); + + it("throw abandons unresolved async transform setup without waiting for input", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(32); + const inputNext = deferred>(); + const inputStarted = deferred(); + const input = { + [Symbol.asyncIterator]() { + return { + next() { + inputStarted.resolve(); + return inputNext.promise; + }, + }; + }, + }; + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", input); + const firstPull = transform.next(); + await inputStarted.promise; + + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown); + const pending = Symbol("pending setup"); + const observed = await Promise.race([ + Promise.allSettled([firstPull, throwing]), + new Promise((resolve) => setImmediate(() => resolve(pending))), + ]); + + inputNext.resolve({ done: true, value: undefined }); + await Promise.allSettled([firstPull, throwing]); + await dw.cleanup(); + + expect(observed).toEqual([ + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "rejected", reason: thrown }, + ]); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + }); + + it("does not admit transform work when setup resolves after return", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(35); + const inputNext = deferred>(); + const inputStarted = deferred(); + const input = { + [Symbol.asyncIterator]() { + return { + next() { + inputStarted.resolve(); + return inputNext.promise; + }, + }; + }, + }; + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", input); + const firstPull = transform.next(); + await inputStarted.promise; + + await transform.return(undefined); + await firstPull; + inputNext.resolve({ done: true, value: undefined }); + await new Promise((resolve) => setImmediate(resolve)); + + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + await dw.cleanup(); + }); + + it("delivers a buffered transform chunk to an earlier next before return", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(33); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("a")); + writeCb(Buffer.from("b")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + await expect(transform.next()).resolves.toEqual({ done: false, value: Buffer.from("a") }); + + const secondPull = transform.next(); + const returned = transform.return(undefined); + + await expect(secondPull).resolves.toEqual({ done: false, value: Buffer.from("b") }); + await expect(returned).resolves.toEqual({ done: true, value: undefined }); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 1); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + it("throw during pending transform setup wins over setup rejection", async () => { vi.mocked(ffi.createEngine).mockReturnValue(29); const readerGate = deferred>>(); @@ -1051,5 +1213,28 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + + it("revalidates runTransform at its actual lazy native admission boundary", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(34); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + vi.mocked(ffi.runScriptTransformEngine).mockReturnValue( + operation(Promise.resolve(okStreamingMeta())) + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + + let boundaryCleanup: Promise | undefined; + readerGate.resolve(() => null); + queueMicrotask(() => { boundaryCleanup = dw.cleanup(); }); + + await expectStaleGenerationError(firstPull); + await boundaryCleanup; + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + }); }); }); diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index 18bdad1a..9a8ebf53 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -426,6 +426,27 @@ describe("streamFromNative", () => { expect(nativeOperation.close).not.toHaveBeenCalled(); }); + it("preserves the first cancellation failure when the finalization retry also fails", async () => { + const nativeOperation = operation(new Promise(() => {})); + nativeOperation.cancel = vi.fn() + .mockImplementationOnce(() => { throw undefined; }) + .mockImplementationOnce(() => { throw null; }); + const gen = streamFromNative((cb) => { + cb(Buffer.from("x")); + return nativeOperation; + }); + + await gen.next(); + const returned = await gen.return(undefined).then( + () => ({ status: "fulfilled" as const }), + (reason: unknown) => ({ status: "rejected" as const, reason }) + ); + + expect(returned).toEqual({ status: "rejected", reason: undefined }); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(2); + expect(nativeOperation.close).not.toHaveBeenCalled(); + }); + it("preserves a consumer error when close also throws", async () => { const completion = deferred(); const nativeOperation = operation(completion.promise); From 1ba8302ac34858ef5bb2983c4f0ae55a7eea9ad4 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Tue, 8 Sep 2026 10:44:48 -0300 Subject: [PATCH 33/48] fix(node): preserve transform request FIFO --- native-lib/node/src/dataweave.ts | 62 ++-- native-lib/node/src/stream.ts | 81 ++++- .../tests/unit/dataweave-initialize.test.ts | 284 ++++++++++++++++++ 3 files changed, 387 insertions(+), 40 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index f3eb3fbd..73945e5f 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -3,7 +3,7 @@ import { resolveAddonPath } from "./addon-path"; import { findLibrary, buildInputsJson } from "./utils"; import { parseNativeResponse } from "./result"; import { createChunkReader } from "./reader"; -import { streamFromNative } from "./stream"; +import { interruptNativeStreamIfParked, nativeStreamParked, streamFromNative } from "./stream"; import { DataWeaveError, DataWeaveScriptError } from "./errors"; import type { NativeStreamingOperation } from "./ffi"; import type { ExecutionResult, StreamingResult, Inputs, TransformOptions } from "./types"; @@ -354,7 +354,8 @@ export class DataWeave { opts?: TransformOptions ): AsyncGenerator { type QueuedRequest = { - readonly run: () => Promise; + readonly kind: "next" | "control"; + readonly run: (onParked: () => void) => Promise; readonly resolve: (result: T) => void; readonly reject: (error: unknown) => void; }; @@ -368,22 +369,27 @@ export class DataWeave { const requestQueue: Array> = []; let requestRunning = false; - const enqueue = (request: () => Promise): Promise => { + const enqueue = ( + kind: QueuedRequest["kind"], + request: QueuedRequest["run"] + ): Promise => { const result = new Promise((resolve, reject) => { - requestQueue.push({ run: request, resolve, reject } as QueuedRequest); + requestQueue.push({ kind, run: request, resolve, reject } as QueuedRequest); }); drainRequests(); return result; }; - const drainRequests = (): void => { + function drainRequests(): void { if (requestRunning) return; const request = requestQueue.shift(); if (!request) return; requestRunning = true; let result: Promise; try { - result = request.run(); + result = request.run(() => { + if (controlPending) interruptAfterQueuedPulls(); + }); } catch (error) { result = Promise.reject(error); } @@ -391,7 +397,18 @@ export class DataWeave { requestRunning = false; drainRequests(); }); - }; + } + + function interruptHeadPull(): void { + if (stream && interruptNativeStreamIfParked(stream)) return; + if (!stream) abandonSetup(); + } + + function interruptAfterQueuedPulls(): void { + const controlIndex = requestQueue.findIndex((request) => request.kind === "control"); + if (controlIndex > 0 && requestQueue.slice(0, controlIndex).some((request) => request.kind === "next")) return; + interruptHeadPull(); + } const setup = (): Promise | null> => setupPromise ??= new Promise((resolve, reject) => { @@ -448,7 +465,7 @@ export class DataWeave { }); return { - next: (...args: [] | [undefined]) => enqueue(async () => { + next: (...args: [] | [undefined]) => enqueue("next", async (onParked) => { if (closed) { return { done: true, value: undefined } as unknown as IteratorReturnResult; } @@ -462,23 +479,19 @@ export class DataWeave { if (!activeStream) { return { done: true, value: undefined } as unknown as IteratorReturnResult; } - const result = await activeStream.next(...args); + const nextPromise = activeStream.next(...args); + nativeStreamParked(activeStream).then((parked) => { + if (parked) onParked(); + }); + const result = await nextPromise; if (result.done) closed = true; return result; }), return: (value) => { const isPrimaryControl = !closed && !controlPending; if (isPrimaryControl) controlPending = true; - // Let an earlier delegated next() consume an already-buffered chunk - // before cancellation wakes it if it is genuinely parked. - const proactiveReturn = isPrimaryControl && stream - ? Promise.resolve().then(() => stream!.return(value)) - : null; - proactiveReturn?.catch(() => {}); - if (isPrimaryControl && !stream) abandonSetup(); - return enqueue(async () => { + const result = enqueue("control", async (_onParked) => { try { - if (proactiveReturn) return await proactiveReturn; if (stream && !closed) return await stream.return(value); return { done: true, @@ -488,25 +501,22 @@ export class DataWeave { closed = true; } }); + if (isPrimaryControl) interruptAfterQueuedPulls(); + return result; }, throw: (error?: unknown) => { const isPrimaryControl = !closed && !controlPending; if (isPrimaryControl) controlPending = true; - // Preserve the same request-ordering turn as return(). - const proactiveThrow = isPrimaryControl && stream - ? Promise.resolve().then(() => stream!.throw(error)) - : null; - proactiveThrow?.catch(() => {}); - if (isPrimaryControl && !stream) abandonSetup(); - return enqueue(async () => { + const result = enqueue("control", async (_onParked) => { try { - if (proactiveThrow) return await proactiveThrow; if (stream && !closed) return await stream.throw(error); throw error; } finally { closed = true; } }); + if (isPrimaryControl) interruptAfterQueuedPulls(); + return result; }, [Symbol.asyncIterator]() { return this; diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index e631c4f3..2a22bed1 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -8,6 +8,27 @@ import type { StreamingResult } from "./types"; */ export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => NativeStreamingOperation; +interface InterruptibleAsyncGenerator extends AsyncGenerator { + readonly parked: Promise; + interruptIfParked(): boolean; +} + +type NativeStreamIterator = InterruptibleAsyncGenerator; + +/** Interrupts a delegated pull only when it has no buffered result to consume. */ +export function interruptNativeStreamIfParked( + iterator: AsyncGenerator +): boolean { + return (iterator as NativeStreamIterator).interruptIfParked(); +} + +/** Reports whether the currently delegated pull had to wait for native output. */ +export function nativeStreamParked( + iterator: AsyncGenerator +): Promise { + return (iterator as NativeStreamIterator).parked; +} + type ErrorState = { readonly hasError: false } | { readonly hasError: true; readonly error: unknown }; const NO_ERROR: ErrorState = { hasError: false }; @@ -46,6 +67,10 @@ export function streamFromNative( let finalizationError: ErrorState = NO_ERROR; let nativeOperation: NativeStreamingOperation | undefined; let cancelInProgress = false; + let pullParked = false; + let interruptionError: ErrorState = NO_ERROR; + let parked = Promise.resolve(false); + let resolveParked: ((value: boolean) => void) | undefined; const wakeAll = () => { while (pendingResolves.length > 0) { @@ -117,6 +142,22 @@ export function streamFromNative( pendingResolves.shift()?.(); }; + function requestCancellation(): ErrorState { + cancellationRequested = true; + wakeAll(); + try { + cancel(); + return NO_ERROR; + } catch (error) { + return { hasError: true, error }; + } + } + + function interruptPull(): void { + const lifecycleError = requestCancellation(); + if (lifecycleError.hasError) interruptionError = lifecycleError; + } + const generator = (async function* (): AsyncGenerator { let primaryError = false; try { @@ -157,7 +198,12 @@ export function streamFromNative( if (cancellationRequested) { return undefined as unknown as StreamingResult; } - await new Promise((resolve) => { pendingResolves.push(resolve); }); + const wake = new Promise((resolve) => { pendingResolves.push(resolve); }); + pullParked = true; + resolveParked?.(true); + await wake; + pullParked = false; + if (interruptionError.hasError) throw interruptionError.error; } await nativeSettlementHandled; @@ -201,20 +247,22 @@ export function streamFromNative( } })(); - const requestCancellation = (): ErrorState => { - cancellationRequested = true; - wakeAll(); - try { - cancel(); - return NO_ERROR; - } catch (error) { - return { hasError: true, error }; - } - }; - - const iterator: AsyncGenerator = { + const iterator: NativeStreamIterator = { + get parked() { + return parked; + }, next(...args: [] | [undefined]) { - return generator.next(...args); + parked = new Promise((resolve) => { resolveParked = resolve; }); + return generator.next(...args).then( + (result) => { + resolveParked?.(false); + return result; + }, + (error) => { + resolveParked?.(false); + throw error; + } + ); }, return(value) { const lifecycleError = requestCancellation(); @@ -243,6 +291,11 @@ export function streamFromNative( (primary) => { throw primary; } ); }, + interruptIfParked() { + if (!pullParked || chunks.length > 0 || nativeSettled || cancellationRequested) return false; + interruptPull(); + return true; + }, [Symbol.asyncIterator]() { return this; }, diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 6fd54032..a9f38fc6 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -877,6 +877,290 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("delivers two buffered chunks to two earlier pulls before return", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(37); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + let push!: (chunk: Buffer) => void; + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + push = writeCb; + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const admissionPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + push(Buffer.from("admitted")); + await expect(admissionPull).resolves.toEqual({ done: false, value: Buffer.from("admitted") }); + push(Buffer.from("a")); + push(Buffer.from("bb")); + const firstPull = transform.next(); + const secondPull = transform.next(); + const returned = transform.return(undefined); + + await expect(Promise.all([firstPull, secondPull, returned])).resolves.toEqual([ + { done: false, value: Buffer.from("a") }, + { done: false, value: Buffer.from("bb") }, + { done: true, value: undefined }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[8], [1], [2]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("delivers two buffered chunks to two earlier pulls before throw", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(38); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + let push!: (chunk: Buffer) => void; + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + push = writeCb; + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const admissionPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + push(Buffer.from("admitted")); + await expect(admissionPull).resolves.toEqual({ done: false, value: Buffer.from("admitted") }); + push(Buffer.from("a")); + push(Buffer.from("bb")); + const firstPull = transform.next(); + const secondPull = transform.next(); + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown); + + const outcomes = await Promise.allSettled([firstPull, secondPull, throwing]); + expect(outcomes).toEqual([ + { status: "fulfilled", value: { done: false, value: Buffer.from("a") } }, + { status: "fulfilled", value: { done: false, value: Buffer.from("bb") } }, + { status: "rejected", reason: thrown }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[8], [1], [2]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("settles four earlier pulls exactly once before return", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(39); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("a")); + writeCb(Buffer.from("bb")); + writeCb(Buffer.from("ccc")); + writeCb(Buffer.from("dddd")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const settlements = [0, 0, 0, 0, 0]; + const firstPull = transform.next().finally(() => { settlements[0]++; }); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const secondPull = transform.next().finally(() => { settlements[1]++; }); + const thirdPull = transform.next().finally(() => { settlements[2]++; }); + const fourthPull = transform.next().finally(() => { settlements[3]++; }); + const returned = transform.return(undefined).finally(() => { settlements[4]++; }); + + await expect(Promise.all([firstPull, secondPull, thirdPull, fourthPull, returned])).resolves.toEqual([ + { done: false, value: Buffer.from("a") }, + { done: false, value: Buffer.from("bb") }, + { done: false, value: Buffer.from("ccc") }, + { done: false, value: Buffer.from("dddd") }, + { done: true, value: undefined }, + ]); + expect(settlements).toEqual([1, 1, 1, 1, 1]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2], [3], [4]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("wakes only a genuinely parked tail after earlier pulls drain buffered chunks", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(40); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("a")); + writeCb(Buffer.from("bb")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const secondPull = transform.next(); + const parkedPull = transform.next(); + await secondPull; + await Promise.resolve(); + const returned = transform.return(undefined); + + await expect(Promise.all([firstPull, secondPull, parkedPull, returned])).resolves.toEqual([ + { done: false, value: Buffer.from("a") }, + { done: false, value: Buffer.from("bb") }, + { done: true, value: undefined }, + { done: true, value: undefined }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("throws after waking only a genuinely parked tail behind earlier buffered pulls", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(41); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("a")); + writeCb(Buffer.from("bb")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const secondPull = transform.next(); + const parkedPull = transform.next(); + await secondPull; + await Promise.resolve(); + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown); + + const outcomes = await Promise.allSettled([firstPull, secondPull, parkedPull, throwing]); + expect(outcomes).toEqual([ + { status: "fulfilled", value: { done: false, value: Buffer.from("a") } }, + { status: "fulfilled", value: { done: false, value: Buffer.from("bb") } }, + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "rejected", reason: thrown }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("surfaces cancellation failure after interrupting a parked transform pull", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(42); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => { throw new Error("cancel boom"); }); + vi.mocked(ffi.runScriptTransformEngine).mockReturnValue(nativeOperation); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const parkedPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const returned = transform.return(undefined); + + await expect(parkedPull).rejects.toThrow("cancel boom"); + await expect(returned).rejects.toThrow("cancel boom"); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(3); + expect(nativeOperation.close).not.toHaveBeenCalled(); + }); + + it("does not let a pull after return block interruption of an earlier parked pull", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(43); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + writeCb(Buffer.from("a")); + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const parkedPull = transform.next(); + const returned = transform.return(undefined); + const laterPull = transform.next(); + + await vi.waitFor(() => expect(nativeOperation.cancel).toHaveBeenCalledTimes(1), { timeout: 250 }); + await expect(Promise.all([firstPull, parkedPull, returned, laterPull])).resolves.toEqual([ + { done: false, value: Buffer.from("a") }, + { done: true, value: undefined }, + { done: true, value: undefined }, + { done: true, value: undefined }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1]]); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + + it("does not interrupt a parked pull while another earlier pull remains queued", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(44); + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); + let push!: (chunk: Buffer) => void; + vi.mocked(ffi.runScriptTransformEngine).mockImplementation( + (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { + push = writeCb; + return nativeOperation; + } + ); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(ffi.runScriptTransformEngine).toHaveBeenCalledTimes(1)); + const secondPull = transform.next(); + const returned = transform.return(undefined); + + expect(nativeOperation.cancel).not.toHaveBeenCalled(); + push(Buffer.from("a")); + + await expect(Promise.all([firstPull, secondPull, returned])).resolves.toEqual([ + { done: false, value: Buffer.from("a") }, + { done: true, value: undefined }, + { done: true, value: undefined }, + ]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1]]); + expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); + expect(nativeOperation.close).toHaveBeenCalledTimes(1); + + await dw.cleanup(); + }); + it("throw during pending transform setup wins over setup rejection", async () => { vi.mocked(ffi.createEngine).mockReturnValue(29); const readerGate = deferred>>(); From 215df9de8b0c7cde351cab96561714a6722023f3 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Tue, 8 Sep 2026 10:54:37 -0300 Subject: [PATCH 34/48] fix(node): abandon pending transform setup --- native-lib/node/src/dataweave.ts | 43 +++-- .../tests/unit/dataweave-initialize.test.ts | 164 ++++++++++++++++++ 2 files changed, 190 insertions(+), 17 deletions(-) diff --git a/native-lib/node/src/dataweave.ts b/native-lib/node/src/dataweave.ts index 73945e5f..fd9791cb 100644 --- a/native-lib/node/src/dataweave.ts +++ b/native-lib/node/src/dataweave.ts @@ -364,8 +364,8 @@ export class DataWeave { let controlPending = false; let stream: AsyncGenerator | null = null; let setupPromise: Promise | null>; - let setupAbandoned = false; - let abandonSetup = () => { setupAbandoned = true; }; + let admissionState: "pending" | "abandoned" | "admitted" = "pending"; + let abandonSetup = () => { admissionState = "abandoned"; }; const requestQueue: Array> = []; let requestRunning = false; @@ -388,7 +388,7 @@ export class DataWeave { let result: Promise; try { result = request.run(() => { - if (controlPending) interruptAfterQueuedPulls(); + if (controlPending) interruptForControl(); }); } catch (error) { result = Promise.reject(error); @@ -399,21 +399,27 @@ export class DataWeave { }); } - function interruptHeadPull(): void { - if (stream && interruptNativeStreamIfParked(stream)) return; - if (!stream) abandonSetup(); - } - - function interruptAfterQueuedPulls(): void { + function interruptAdmittedPullForControl(): void { const controlIndex = requestQueue.findIndex((request) => request.kind === "control"); if (controlIndex > 0 && requestQueue.slice(0, controlIndex).some((request) => request.kind === "next")) return; - interruptHeadPull(); + if (stream) interruptNativeStreamIfParked(stream); + } + + function interruptForControl(): void { + // Setup has no native pull to preserve: settle every queued pre-control + // next() as done immediately. Admitted streams instead retain FIFO until + // the last earlier pull is genuinely parked. + if (admissionState !== "admitted") { + abandonSetup(); + return; + } + interruptAdmittedPullForControl(); } const setup = (): Promise | null> => setupPromise ??= new Promise((resolve, reject) => { abandonSetup = () => { - setupAbandoned = true; + admissionState = "abandoned"; resolve(null); }; try { @@ -430,7 +436,7 @@ export class DataWeave { const inputsJson = Object.keys(extraInputs).length > 0 ? buildInputsJson(extraInputs) : "{}"; createChunkReader(input).then( (readCb) => { - if (setupAbandoned || controlPending) return; + if (admissionState === "abandoned" || controlPending) return; try { this.assertCurrentOperation(token); stream = streamFromNative( @@ -447,7 +453,10 @@ export class DataWeave { writeCb ); }, - (operation) => { this.registerActiveStream(operation); }, + (operation) => { + admissionState = "admitted"; + this.registerActiveStream(operation); + }, (operation) => { this.markActiveStreamClosed(operation); } ); resolve(stream); @@ -456,7 +465,7 @@ export class DataWeave { } }, (error) => { - if (!setupAbandoned) reject(error); + if (admissionState !== "abandoned") reject(error); } ); } catch (error) { @@ -476,7 +485,7 @@ export class DataWeave { closed = true; throw error; } - if (!activeStream) { + if (!activeStream || admissionState === "abandoned") { return { done: true, value: undefined } as unknown as IteratorReturnResult; } const nextPromise = activeStream.next(...args); @@ -501,7 +510,7 @@ export class DataWeave { closed = true; } }); - if (isPrimaryControl) interruptAfterQueuedPulls(); + if (isPrimaryControl) interruptForControl(); return result; }, throw: (error?: unknown) => { @@ -515,7 +524,7 @@ export class DataWeave { closed = true; } }); - if (isPrimaryControl) interruptAfterQueuedPulls(); + if (isPrimaryControl) interruptForControl(); return result; }, [Symbol.asyncIterator]() { diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index a9f38fc6..21998f0e 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -847,6 +847,170 @@ describe("DataWeave.initialize() native ref-count safety", () => { await dw.cleanup(); }); + it("abandons unresolved setup for two pulls before return and ignores late fulfillment", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(45); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + const returnGate = deferred(); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const settlements = [0, 0, 0]; + const firstPull = transform.next().finally(() => { settlements[0]++; }); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const secondPull = transform.next().finally(() => { settlements[1]++; }); + const returned = (transform.return as (value: unknown) => Promise>)(returnGate.promise) + .finally(() => { settlements[2]++; }); + const laterPull = transform.next(); + const pendingSetup = Symbol("pending setup"); + + expect(await Promise.race([ + Promise.all([firstPull, secondPull]), + new Promise((resolve) => setImmediate(() => resolve(pendingSetup))), + ])).toEqual([ + { done: true, value: undefined }, + { done: true, value: undefined }, + ]); + + const returnValue = { success: true } as StreamingResult; + returnGate.resolve(returnValue); + await expect(Promise.all([returned, laterPull])).resolves.toEqual([ + { done: true, value: returnValue }, + { done: true, value: undefined }, + ]); + expect(settlements).toEqual([1, 1, 1]); + + readerGate.resolve(() => null); + await readerGate.promise; + await Promise.resolve(); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("abandons unresolved setup for two pulls before throw and ignores late fulfillment", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(46); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const settlements = [0, 0, 0]; + const firstPull = transform.next().finally(() => { settlements[0]++; }); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const secondPull = transform.next().finally(() => { settlements[1]++; }); + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown).finally(() => { settlements[2]++; }); + const laterPull = transform.next(); + const pendingSetup = Symbol("pending setup"); + + expect(await Promise.race([ + Promise.allSettled([firstPull, secondPull, throwing, laterPull]), + new Promise((resolve) => setImmediate(() => resolve(pendingSetup))), + ])).toEqual([ + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "rejected", reason: thrown }, + { status: "fulfilled", value: { done: true, value: undefined } }, + ]); + expect(settlements).toEqual([1, 1, 1]); + + readerGate.resolve(() => null); + await readerGate.promise; + await Promise.resolve(); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + }); + + it("keeps returned setup outcomes unchanged when abandoned setup rejects late", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(47); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + const unhandled: unknown[] = []; + const onUnhandled = (reason: unknown) => { unhandled.push(reason); }; + process.on("unhandledRejection", onUnhandled); + + try { + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const secondPull = transform.next(); + const returnValue = { success: true } as StreamingResult; + const returned = transform.return(returnValue); + const laterPull = transform.next(); + const pendingSetup = Symbol("pending setup"); + const outcomes = await Promise.race([ + Promise.all([firstPull, secondPull, returned, laterPull]), + new Promise((resolve) => setImmediate(() => resolve(pendingSetup))), + ]); + + expect(outcomes).toEqual([ + { done: true, value: undefined }, + { done: true, value: undefined }, + { done: true, value: returnValue }, + { done: true, value: undefined }, + ]); + + readerGate.reject(new Error("late setup boom")); + await readerGate.promise.catch(() => {}); + await Promise.resolve(); + expect(unhandled).toEqual([]); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + } finally { + process.off("unhandledRejection", onUnhandled); + } + }); + + it("keeps thrown setup outcomes unchanged when abandoned setup rejects late", async () => { + vi.mocked(ffi.createEngine).mockReturnValue(48); + const readerGate = deferred>>(); + vi.mocked(createChunkReader).mockReturnValueOnce(readerGate.promise); + const unhandled: unknown[] = []; + const onUnhandled = (reason: unknown) => { unhandled.push(reason); }; + process.on("unhandledRejection", onUnhandled); + + try { + const dw = new DataWeave("/fake/lib"); + dw.initialize(); + const transform = dw.runTransform("output application/json --- payload", []); + const firstPull = transform.next(); + await vi.waitFor(() => expect(createChunkReader).toHaveBeenCalledTimes(1)); + const secondPull = transform.next(); + const thrown = new Error("consumer boom"); + const throwing = transform.throw(thrown); + const laterPull = transform.next(); + const pendingSetup = Symbol("pending setup"); + const outcomes = await Promise.race([ + Promise.allSettled([firstPull, secondPull, throwing, laterPull]), + new Promise((resolve) => setImmediate(() => resolve(pendingSetup))), + ]); + + expect(outcomes).toEqual([ + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "fulfilled", value: { done: true, value: undefined } }, + { status: "rejected", reason: thrown }, + { status: "fulfilled", value: { done: true, value: undefined } }, + ]); + + readerGate.reject(new Error("late setup boom")); + await readerGate.promise.catch(() => {}); + await Promise.resolve(); + expect(unhandled).toEqual([]); + expect(ffi.runScriptTransformEngine).not.toHaveBeenCalled(); + + await dw.cleanup(); + } finally { + process.off("unhandledRejection", onUnhandled); + } + }); + it("delivers a buffered transform chunk to an earlier next before return", async () => { vi.mocked(ffi.createEngine).mockReturnValue(33); const completion = deferred(); From 79e283b25136fcc6cc7423762811de08a864261d Mon Sep 17 00:00:00 2001 From: mlischetti Date: Tue, 8 Sep 2026 17:56:56 -0300 Subject: [PATCH 35/48] fix(node): bound asynchronous output buffering --- native-lib/node/src/addon.c | 830 ++++++++++++++++-- .../integration/stream-backpressure.test.ts | 427 +++++++++ 2 files changed, 1203 insertions(+), 54 deletions(-) create mode 100644 native-lib/node/tests/integration/stream-backpressure.test.ts diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index d9e9ffd7..2cf4607a 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -37,6 +37,7 @@ static void* g_thread = NULL; static int g_initialized = 0; static int g_ref_count = 0; static uv_mutex_t g_mutex; +static uv_mutex_t g_test_output_mutex; static uv_key_t g_native_callback_depth; static int g_native_callback_depth_status; // Guards initialization of the process-global g_mutex. Init() runs once per @@ -1192,6 +1193,493 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { // --- Streaming output --- +#define OUTPUT_HIGH_BYTES (1024 * 1024) +#define OUTPUT_LOW_BYTES (512 * 1024) +#define OUTPUT_HIGH_CHUNKS 128 +#define OUTPUT_LOW_CHUNKS 64 +#define OUTPUT_TSFN_QUEUE_SIZE 129 + +typedef struct output_credit { + size_t bytes; + struct output_credit* previous; + struct output_credit* next; +} output_credit_t; + +typedef struct output_flow { + uv_mutex_t mutex; + uv_cond_t cond; + size_t outstanding_bytes; + size_t outstanding_chunks; + size_t peak_buffered_bytes; + size_t peak_buffered_chunks; + size_t largest_chunk_bytes; + bool paused; + bool cancelled; + bool done; + unsigned int refs; + uint64_t operation_id; + output_credit_t* credit_head; + output_credit_t* credit_tail; + napi_ref thenable_ref; +} output_flow_t; + +typedef struct output_flow_stats { + uint64_t operation_id; + size_t outstanding_bytes; + size_t outstanding_chunks; + size_t peak_buffered_bytes; + size_t peak_buffered_chunks; + size_t largest_chunk_bytes; + bool paused; + bool cancelled; + bool done; + long long live_flows; +} output_flow_stats_t; + +// Test hooks store a value snapshot, never a flow pointer, so close/finalize +// cannot leave introspection pointing at freed operation state. +static output_flow_stats_t g_test_last_output_stats; +static uint64_t g_test_next_output_operation_id = 1; +static long long g_test_live_output_flows = 0; + +static void output_flow_record_stats_locked(output_flow_t* flow) { + if (!g_test_hooks || flow == NULL) return; + uv_mutex_lock(&g_test_output_mutex); + g_test_last_output_stats.operation_id = flow->operation_id; + g_test_last_output_stats.outstanding_bytes = flow->outstanding_bytes; + g_test_last_output_stats.outstanding_chunks = flow->outstanding_chunks; + g_test_last_output_stats.peak_buffered_bytes = flow->peak_buffered_bytes; + g_test_last_output_stats.peak_buffered_chunks = flow->peak_buffered_chunks; + g_test_last_output_stats.largest_chunk_bytes = flow->largest_chunk_bytes; + g_test_last_output_stats.paused = flow->paused; + g_test_last_output_stats.cancelled = flow->cancelled; + g_test_last_output_stats.done = flow->done; + g_test_last_output_stats.live_flows = g_test_live_output_flows; + uv_mutex_unlock(&g_test_output_mutex); +} + +static output_flow_t* output_flow_create(void) { + output_flow_t* flow = (output_flow_t*)calloc(1, sizeof(output_flow_t)); + if (flow == NULL) return NULL; + if (uv_mutex_init(&flow->mutex) != 0) { + free(flow); + return NULL; + } + if (uv_cond_init(&flow->cond) != 0) { + uv_mutex_destroy(&flow->mutex); + free(flow); + return NULL; + } + flow->refs = 1; + if (g_test_hooks) { + uv_mutex_lock(&g_test_output_mutex); + flow->operation_id = g_test_next_output_operation_id++; + g_test_live_output_flows++; + memset(&g_test_last_output_stats, 0, sizeof(g_test_last_output_stats)); + g_test_last_output_stats.operation_id = flow->operation_id; + g_test_last_output_stats.live_flows = g_test_live_output_flows; + uv_mutex_unlock(&g_test_output_mutex); + } + return flow; +} + +static void output_flow_retain(output_flow_t* flow) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + flow->refs++; + uv_mutex_unlock(&flow->mutex); +} + +static void output_flow_release(output_flow_t* flow) { + if (flow == NULL) return; + bool destroy = false; + uv_mutex_lock(&flow->mutex); + if (flow->refs > 0) { + flow->refs--; + destroy = flow->refs == 0; + } + uv_mutex_unlock(&flow->mutex); + if (!destroy) return; + // A dead env auto-reclaims this N-API reference. A live env path deletes it + // in output_flow_mark_done before the final native owner is released. + flow->thenable_ref = NULL; + + if (g_test_hooks) { + uv_mutex_lock(&g_test_output_mutex); + g_test_live_output_flows--; + if (g_test_last_output_stats.operation_id == flow->operation_id) { + g_test_last_output_stats.live_flows = g_test_live_output_flows; + } + uv_mutex_unlock(&g_test_output_mutex); + } + output_credit_t* credit = flow->credit_head; + while (credit != NULL) { + output_credit_t* next = credit->next; + free(credit); + credit = next; + } + uv_cond_destroy(&flow->cond); + uv_mutex_destroy(&flow->mutex); + free(flow); +} + +// Only the native producer waits here. No caller holds g_mutex while waiting. +static bool output_flow_reserve(output_flow_t* flow, size_t bytes) { + if (flow == NULL) return false; + output_credit_t* credit = (output_credit_t*)calloc(1, sizeof(output_credit_t)); + if (credit == NULL) return false; + credit->bytes = bytes; + + uv_mutex_lock(&flow->mutex); + if (flow->cancelled || flow->done) { + flow->paused = false; + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + free(credit); + return false; + } + bool empty = flow->outstanding_bytes == 0 && flow->outstanding_chunks == 0; + bool oversized = bytes > OUTPUT_HIGH_BYTES; + bool oversized_empty = empty && oversized; + bool over_high = + flow->outstanding_chunks + 1 > OUTPUT_HIGH_CHUNKS || + bytes > SIZE_MAX - flow->outstanding_bytes || + flow->outstanding_bytes + bytes > OUTPUT_HIGH_BYTES; + if (over_high && !oversized_empty) { + flow->paused = true; + output_flow_record_stats_locked(flow); + while (!flow->cancelled && !flow->done && + ((oversized && + (flow->outstanding_bytes > 0 || flow->outstanding_chunks > 0)) || + flow->outstanding_bytes > OUTPUT_LOW_BYTES || + flow->outstanding_chunks > OUTPUT_LOW_CHUNKS)) { + uv_cond_wait(&flow->cond, &flow->mutex); + } + } + + if (flow->cancelled || flow->done) { + flow->paused = false; + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + free(credit); + return false; + } + + flow->paused = false; + // The overflow check above routes a huge reservation through the oversized + // wait, which drains prior credit; ordinary reservations are already bounded. + flow->outstanding_bytes += bytes; + flow->outstanding_chunks++; + credit->previous = flow->credit_tail; + if (flow->credit_tail != NULL) flow->credit_tail->next = credit; + else flow->credit_head = credit; + flow->credit_tail = credit; + if (flow->outstanding_bytes > flow->peak_buffered_bytes) { + flow->peak_buffered_bytes = flow->outstanding_bytes; + } + if (flow->outstanding_chunks > flow->peak_buffered_chunks) { + flow->peak_buffered_chunks = flow->outstanding_chunks; + } + if (bytes > flow->largest_chunk_bytes) flow->largest_chunk_bytes = bytes; + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + return true; +} + +static void output_flow_acknowledge(output_flow_t* flow, size_t bytes) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + output_credit_t* credit = flow->credit_head; + // Task 8 acknowledges each dequeued chunk exactly once in FIFO order. A + // mismatched byte count is a no-op rather than corrupting the queue. + if (!flow->cancelled && credit != NULL && credit->bytes == bytes && + bytes <= flow->outstanding_bytes && flow->outstanding_chunks > 0) { + flow->outstanding_bytes -= bytes; + flow->outstanding_chunks--; + flow->credit_head = credit->next; + if (flow->credit_head != NULL) flow->credit_head->previous = NULL; + else flow->credit_tail = NULL; + free(credit); + if (flow->paused && + flow->outstanding_bytes <= OUTPUT_LOW_BYTES && + flow->outstanding_chunks <= OUTPUT_LOW_CHUNKS) { + uv_cond_broadcast(&flow->cond); + } + } + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); +} + +// Enqueue/allocation rollback always targets the newest reservation because +// callbacks reserve and enqueue serially on the sole producer worker. +static void output_flow_rollback(output_flow_t* flow, size_t bytes) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + output_credit_t* credit = flow->credit_tail; + if (credit != NULL && credit->bytes == bytes && + bytes <= flow->outstanding_bytes && flow->outstanding_chunks > 0) { + flow->outstanding_bytes -= bytes; + flow->outstanding_chunks--; + flow->credit_tail = credit->previous; + if (flow->credit_tail != NULL) flow->credit_tail->next = NULL; + else flow->credit_head = NULL; + free(credit); + } + uv_cond_broadcast(&flow->cond); + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); +} + +static void output_flow_cancel(output_flow_t* flow) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + if (!flow->cancelled) { + flow->cancelled = true; + flow->paused = false; + flow->outstanding_bytes = 0; + flow->outstanding_chunks = 0; + output_credit_t* credit = flow->credit_head; + while (credit != NULL) { + output_credit_t* next = credit->next; + free(credit); + credit = next; + } + flow->credit_head = NULL; + flow->credit_tail = NULL; + } + uv_cond_broadcast(&flow->cond); + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); +} + +static bool output_flow_is_cancelled(output_flow_t* flow) { + if (flow == NULL) return true; + uv_mutex_lock(&flow->mutex); + bool cancelled = flow->cancelled; + uv_mutex_unlock(&flow->mutex); + return cancelled; +} + +static void output_flow_retain_thenable( + output_flow_t* flow, napi_env env, napi_value controller) { + if (flow == NULL || env == NULL) return; + uv_mutex_lock(&flow->mutex); + if (!flow->done && flow->thenable_ref == NULL) { + napi_ref thenable_ref = NULL; + if (napi_create_reference(env, controller, 1, &thenable_ref) == napi_ok) { + flow->thenable_ref = thenable_ref; + } + } + uv_mutex_unlock(&flow->mutex); +} + +static void output_flow_mark_done(output_flow_t* flow, napi_env env) { + if (flow == NULL) return; + napi_ref thenable_ref = NULL; + uv_mutex_lock(&flow->mutex); + flow->done = true; + flow->paused = false; + thenable_ref = flow->thenable_ref; + flow->thenable_ref = NULL; + uv_cond_broadcast(&flow->cond); + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + // The reference is only created on this env's JS thread. env == NULL means + // teardown owns automatic N-API reference reclamation. + if (env != NULL && thenable_ref != NULL) napi_delete_reference(env, thenable_ref); +} + +typedef struct output_controller { + uv_mutex_t mutex; + output_flow_t* flow; + bool closed; + uint64_t operation_id; +} output_controller_t; + +static void output_controller_cancel(output_controller_t* holder) { + if (holder == NULL) return; + uv_mutex_lock(&holder->mutex); + output_flow_t* flow = holder->flow; + if (flow != NULL) output_flow_retain(flow); + uv_mutex_unlock(&holder->mutex); + if (flow != NULL) { + output_flow_cancel(flow); + output_flow_release(flow); + } +} + +static void output_controller_close(output_controller_t* holder) { + if (holder == NULL) return; + uv_mutex_lock(&holder->mutex); + output_flow_t* flow = NULL; + if (!holder->closed) { + holder->closed = true; + flow = holder->flow; + holder->flow = NULL; + } + uv_mutex_unlock(&holder->mutex); + if (flow != NULL) output_flow_release(flow); +} + +static void output_controller_finalize(napi_env env, void* data, void* hint) { + (void)env; + (void)hint; + output_controller_t* holder = (output_controller_t*)data; + if (holder == NULL) return; + output_controller_cancel(holder); + output_controller_close(holder); + uv_mutex_destroy(&holder->mutex); + free(holder); +} + +static output_controller_t* output_controller_unwrap( + napi_env env, napi_callback_info info, size_t expected_argc, napi_value* argv) { + napi_value this_arg; + size_t actual = expected_argc; + if (napi_get_cb_info(env, info, &actual, argv, &this_arg, NULL) != napi_ok) { + napi_throw_type_error(env, NULL, "Invalid output controller invocation"); + return NULL; + } + if (actual < expected_argc) { + napi_throw_type_error(env, NULL, "Missing output controller argument"); + return NULL; + } + output_controller_t* holder = NULL; + if (napi_unwrap(env, this_arg, (void**)&holder) != napi_ok || holder == NULL) { + napi_throw_type_error(env, NULL, "Invalid output controller receiver"); + return NULL; + } + return holder; +} + +static napi_value napi_output_acknowledge(napi_env env, napi_callback_info info) { + napi_value argv[1]; + output_controller_t* holder = output_controller_unwrap(env, info, 1, argv); + if (holder == NULL) return NULL; + napi_valuetype type; + double value; + if (napi_typeof(env, argv[0], &type) != napi_ok || type != napi_number || + napi_get_value_double(env, argv[0], &value) != napi_ok || + value != value || value < 0 || value > 9007199254740991.0 || + value > (double)SIZE_MAX || value != (double)(size_t)value) { + napi_throw_range_error(env, NULL, + "acknowledge(bytes) requires a finite non-negative safe integer"); + return NULL; + } + uv_mutex_lock(&holder->mutex); + output_flow_t* flow = holder->flow; + if (flow != NULL) output_flow_retain(flow); + uv_mutex_unlock(&holder->mutex); + if (flow != NULL) { + output_flow_acknowledge(flow, (size_t)value); + output_flow_release(flow); + } + return NULL; +} + +static napi_value napi_output_cancel(napi_env env, napi_callback_info info) { + output_controller_t* holder = output_controller_unwrap(env, info, 0, NULL); + if (holder == NULL) return NULL; + output_controller_cancel(holder); + return NULL; +} + +static napi_value napi_output_close(napi_env env, napi_callback_info info) { + output_controller_t* holder = output_controller_unwrap(env, info, 0, NULL); + if (holder == NULL) return NULL; + output_controller_close(holder); + return NULL; +} + +static napi_value napi_output_promise_method(napi_env env, napi_callback_info info) { + size_t argc = 2; + napi_value argv[2]; + napi_value controller; + void* data; + if (napi_get_cb_info(env, info, &argc, argv, &controller, &data) != napi_ok) { + napi_throw_type_error(env, NULL, "Invalid output controller promise method"); + return NULL; + } + output_controller_t* holder = NULL; + if (napi_unwrap(env, controller, (void**)&holder) != napi_ok || holder == NULL) { + napi_throw_type_error(env, NULL, "Invalid output controller receiver"); + return NULL; + } + uv_mutex_lock(&holder->mutex); + output_flow_t* flow = holder->flow; + if (flow != NULL) output_flow_retain(flow); + uv_mutex_unlock(&holder->mutex); + if (flow != NULL) { + // Retain only once a caller actually treats the controller as thenable. + // A simply dropped controller still finalizes promptly and cancels. + output_flow_retain_thenable(flow, env, controller); + output_flow_release(flow); + } + + const char* name = (const char*)data; + napi_value completion; + napi_value method; + napi_value result; + if (napi_get_named_property(env, controller, "completion", &completion) != napi_ok || + napi_get_named_property(env, completion, name, &method) != napi_ok || + napi_call_function(env, completion, method, argc, argv, &result) != napi_ok) { + return NULL; + } + return result; +} + +static napi_value output_controller_create( + napi_env env, napi_value completion, output_flow_t* flow) { + output_controller_t* holder = (output_controller_t*)calloc(1, sizeof(output_controller_t)); + if (holder == NULL || uv_mutex_init(&holder->mutex) != 0) { + free(holder); + napi_throw_error(env, NULL, "OOM"); + return NULL; + } + holder->flow = flow; + holder->operation_id = flow->operation_id; + output_flow_retain(flow); // JS ownership, released once by close/finalizer. + + napi_value controller; + napi_value method; + if (napi_create_object(env, &controller) != napi_ok || + napi_set_named_property(env, controller, "completion", completion) != napi_ok || + napi_create_function(env, "acknowledge", NAPI_AUTO_LENGTH, + napi_output_acknowledge, NULL, &method) != napi_ok || + napi_set_named_property(env, controller, "acknowledge", method) != napi_ok || + napi_create_function(env, "cancel", NAPI_AUTO_LENGTH, + napi_output_cancel, NULL, &method) != napi_ok || + napi_set_named_property(env, controller, "cancel", method) != napi_ok || + napi_create_function(env, "close", NAPI_AUTO_LENGTH, + napi_output_close, NULL, &method) != napi_ok || + napi_set_named_property(env, controller, "close", method) != napi_ok || + napi_create_function(env, "then", NAPI_AUTO_LENGTH, + napi_output_promise_method, (void*)"then", &method) != napi_ok || + napi_set_named_property(env, controller, "then", method) != napi_ok || + napi_create_function(env, "catch", NAPI_AUTO_LENGTH, + napi_output_promise_method, (void*)"catch", &method) != napi_ok || + napi_set_named_property(env, controller, "catch", method) != napi_ok || + napi_create_function(env, "finally", NAPI_AUTO_LENGTH, + napi_output_promise_method, (void*)"finally", &method) != napi_ok || + napi_set_named_property(env, controller, "finally", method) != napi_ok) { + output_controller_cancel(holder); + output_controller_close(holder); + uv_mutex_destroy(&holder->mutex); + free(holder); + napi_throw_error(env, NULL, "Failed to create output controller"); + return NULL; + } + if (napi_wrap(env, controller, holder, output_controller_finalize, NULL, NULL) != napi_ok) { + output_controller_cancel(holder); + output_controller_close(holder); + uv_mutex_destroy(&holder->mutex); + free(holder); + napi_throw_error(env, NULL, "Failed to wrap output controller"); + return NULL; + } + return controller; +} + // Round-9 (#2): static terminal-error JSON used when a worker thread cannot // even strdup its result string (OOM). It is a file-scope constant, never // heap-allocated, so any code path that would free a sentinel/chunk buffer @@ -1203,6 +1691,8 @@ static const char OOM_JSON[] = "{\"success\":false,\"error\":\"Out of memory\"}" struct chunk_data { char* buf; int len; + output_flow_t* flow; + size_t accounted_bytes; }; struct streaming_work { @@ -1224,8 +1714,19 @@ struct streaming_work { // WITHOUT enqueuing -- but the env is alive on OOM, so the promise would // never settle and the tsfn would never be released: a permanent hang. struct chunk_data* sentinel; + output_flow_t* flow; }; +static void output_chunk_release(struct chunk_data* chunk, bool rollback) { + if (chunk == NULL) return; + if (chunk->flow != NULL) { + if (rollback) output_flow_rollback(chunk->flow, chunk->accounted_bytes); + output_flow_release(chunk->flow); + } + free(chunk->buf); + free(chunk); +} + static void call_js_write(napi_env env, napi_value js_callback, void* context, void* data) { // data == NULL: nothing was queued, nothing to free or finalize. if (data == NULL) return; @@ -1243,10 +1744,14 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // indefinitely. if (env != NULL) { napi_value result; - napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result); - napi_resolve_deferred(env, w->deferred, result); + if (napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result) == napi_ok) { + napi_resolve_deferred(env, w->deferred, result); + } else { + output_flow_cancel(w->flow); + } } + output_flow_mark_done(w->flow, env); if (chunk->buf != OOM_JSON) free(chunk->buf); free(chunk); free(w->script); @@ -1260,6 +1765,7 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // dead/tearing down -- tell bridge_end_op (and any bridge_finalize it // triggers) not to touch the napi_ref, since b->env is this same dead env. bridge_end_op(w->bridge, /*env_still_alive=*/env != NULL); + output_flow_release(w->flow); free(w); return; } @@ -1268,42 +1774,74 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // cannot deliver it to JS; free it and return without touching `w` (its // finalization happens only on the sentinel, above). if (env == NULL) { - free(chunk->buf); - free(chunk); + output_flow_cancel(chunk->flow); + output_chunk_release(chunk, /*rollback=*/true); + return; + } + // cancel() already released this payload's credit. Drop any TSFN payload + // that was queued before cancellation instead of calling JavaScript again. + if (output_flow_is_cancelled(chunk->flow)) { + output_chunk_release(chunk, /*rollback=*/false); return; } napi_value buffer; void* buf_data; - napi_create_buffer_copy(env, chunk->len, chunk->buf, &buf_data, &buffer); - - napi_value global; - napi_get_global(env, &global); - native_callback_enter(); - napi_call_function(env, global, js_callback, 1, &buffer, NULL); - native_callback_exit(); - - free(chunk->buf); - free(chunk); + napi_status status = napi_create_buffer_copy( + env, chunk->len, chunk->buf, &buf_data, &buffer + ); + if (status == napi_ok) { + napi_value global; + status = napi_get_global(env, &global); + if (status == napi_ok) { + native_callback_enter(); + status = napi_call_function(env, global, js_callback, 1, &buffer, NULL); + native_callback_exit(); + } + } + if (status != napi_ok) { + output_flow_cancel(chunk->flow); + if (status == napi_pending_exception) { + napi_value exception; + napi_get_and_clear_last_exception(env, &exception); + } + } + output_chunk_release(chunk, /*rollback=*/status != napi_ok); } static int streaming_write_cb(void* ctx, const char* buf, int len) { - napi_threadsafe_function tsfn = (napi_threadsafe_function)ctx; + struct streaming_work* w = (struct streaming_work*)ctx; + if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; + if (!output_flow_reserve(w->flow, (size_t)len)) return -1; // Round-9 (#2): OOM here must not deref NULL / memcpy into NULL. Returning -1 // aborts the native run cleanly (write-callback contract: non-zero stops the // DataWeave run); the worker then still produces a terminal meta_result and // sentinel, so the op resolves. struct chunk_data* chunk = malloc(sizeof(struct chunk_data)); - if (chunk == NULL) return -1; - chunk->buf = malloc(len); - if (chunk->buf == NULL) { free(chunk); return -1; } - memcpy(chunk->buf, buf, len); + if (chunk == NULL) { + output_flow_rollback(w->flow, (size_t)len); + output_flow_cancel(w->flow); + return -1; + } + chunk->buf = len == 0 ? NULL : malloc((size_t)len); + if (len > 0 && chunk->buf == NULL) { + free(chunk); + output_flow_rollback(w->flow, (size_t)len); + output_flow_cancel(w->flow); + return -1; + } + if (len > 0) memcpy(chunk->buf, buf, (size_t)len); chunk->len = len; + chunk->flow = w->flow; + chunk->accounted_bytes = (size_t)len; + output_flow_retain(w->flow); - napi_status status = napi_call_threadsafe_function(tsfn, chunk, napi_tsfn_blocking); + napi_status status = napi_call_threadsafe_function( + w->tsfn, chunk, napi_tsfn_nonblocking + ); if (status != napi_ok) { - free(chunk->buf); - free(chunk); + output_chunk_release(chunk, /*rollback=*/true); + output_flow_cancel(w->flow); return -1; } return 0; @@ -1329,7 +1867,7 @@ static void streaming_thread_fn(void* arg) { if (meta_result == NULL) meta_result = (char*)OOM_JSON; } else { void* result_ptr = fn_run_script_callback_engine( - worker_thread, w->handle, w->script, w->inputs_json, streaming_write_cb, (void*)w->tsfn + worker_thread, w->handle, w->script, w->inputs_json, streaming_write_cb, (void*)w ); if (result_ptr) { meta_result = strdup((const char*)result_ptr); @@ -1376,6 +1914,8 @@ static void streaming_thread_fn(void* arg) { struct chunk_data* sentinel = w->sentinel; sentinel->buf = meta_result; sentinel->len = -1; + sentinel->flow = NULL; + sentinel->accounted_bytes = 0; napi_status enq = napi_call_threadsafe_function(w->tsfn, sentinel, napi_tsfn_blocking); if (enq != napi_ok) { // The env is tearing down (napi_closing): the sentinel was dropped and @@ -1408,6 +1948,8 @@ static void streaming_thread_fn(void* arg) { free(w->script); free(w->inputs_json); bridge_end_op(w->bridge, /*env_still_alive=*/false); + output_flow_mark_done(w->flow, NULL); + output_flow_release(w->flow); free(w); } } @@ -1516,6 +2058,14 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i napi_throw_error(env, NULL, "runScriptStreamingEngine: failed to read script/inputsJson"); return NULL; } + w->flow = output_flow_create(); + if (w->flow == NULL) { + free(w->script); free(w->inputs_json); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "OOM"); + return NULL; + } // Round-9 (#3, updated round-11 #2): the resource creations below run AFTER // g_active_ops was reserved (and after w + its buffers were allocated), and @@ -1527,13 +2077,17 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // (teardown wedge). napi_value resource_name; if (napi_create_string_utf8(env, "dwStreaming", NAPI_AUTO_LENGTH, &resource_name) != napi_ok) { + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "runScriptStreamingEngine: failed to create resource name"); return NULL; } - if (napi_create_threadsafe_function(env, argv[3], NULL, resource_name, 0, 1, NULL, NULL, w, call_js_write, &w->tsfn) != napi_ok) { + if (napi_create_threadsafe_function(env, argv[3], NULL, resource_name, + OUTPUT_TSFN_QUEUE_SIZE, 1, NULL, NULL, + w, call_js_write, &w->tsfn) != napi_ok) { + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -1551,6 +2105,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i w->sentinel = malloc(sizeof(struct chunk_data)); if (w->sentinel == NULL) { napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -1563,6 +2118,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // The tsfn was created above; release it before freeing w (it holds w as // its context). No worker exists yet, so this release is the sole discharge. napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); + output_flow_release(w->flow); free(w->sentinel); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -1570,6 +2126,16 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i return NULL; } + napi_value controller = output_controller_create(env, promise, w->flow); + if (controller == NULL) { + napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); + output_flow_release(w->flow); + free(w->sentinel); free(w->script); free(w->inputs_json); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + return NULL; + } + // Round-11 (#2): the pin was taken at admission (bridge_begin_op_locked) in // the same critical section as g_active_ops, so a concurrent destroyEngine // could never free this bridge under the admitted op. Just record it on w; @@ -1603,10 +2169,12 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i free(w->sentinel); free(w->script); free(w->inputs_json); + output_flow_mark_done(w->flow, env); + output_flow_release(w->flow); free(w); } - return promise; + return controller; } // --- Bidirectional streaming --- @@ -1632,6 +2200,7 @@ struct transform_work { // terminal path is allocation-free and can ALWAYS enqueue completion. See // the same field on struct streaming_work for the hang this prevents. struct chunk_data* sentinel; + output_flow_t* flow; }; struct read_request { @@ -1758,19 +2327,35 @@ static int transform_read_cb(void* ctx, char* buf, int buf_size) { static int transform_write_cb(void* ctx, const char* buf, int len) { struct transform_work* w = (struct transform_work*)ctx; + if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; + if (!output_flow_reserve(w->flow, (size_t)len)) return -1; // Round-9 (#2): OOM-safe, mirrors streaming_write_cb. Return -1 to abort the // native run cleanly; the worker still delivers a terminal sentinel. struct chunk_data* chunk = malloc(sizeof(struct chunk_data)); - if (chunk == NULL) return -1; - chunk->buf = malloc(len); - if (chunk->buf == NULL) { free(chunk); return -1; } - memcpy(chunk->buf, buf, len); + if (chunk == NULL) { + output_flow_rollback(w->flow, (size_t)len); + output_flow_cancel(w->flow); + return -1; + } + chunk->buf = len == 0 ? NULL : malloc((size_t)len); + if (len > 0 && chunk->buf == NULL) { + free(chunk); + output_flow_rollback(w->flow, (size_t)len); + output_flow_cancel(w->flow); + return -1; + } + if (len > 0) memcpy(chunk->buf, buf, (size_t)len); chunk->len = len; + chunk->flow = w->flow; + chunk->accounted_bytes = (size_t)len; + output_flow_retain(w->flow); - napi_status status = napi_call_threadsafe_function(w->write_tsfn, chunk, napi_tsfn_blocking); + napi_status status = napi_call_threadsafe_function( + w->write_tsfn, chunk, napi_tsfn_nonblocking + ); if (status != napi_ok) { - free(chunk->buf); - free(chunk); + output_chunk_release(chunk, /*rollback=*/true); + output_flow_cancel(w->flow); return -1; } return 0; @@ -1793,10 +2378,14 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // indefinitely. if (env != NULL) { napi_value result; - napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result); - napi_resolve_deferred(env, w->deferred, result); + if (napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result) == napi_ok) { + napi_resolve_deferred(env, w->deferred, result); + } else { + output_flow_cancel(w->flow); + } } + output_flow_mark_done(w->flow, env); if (chunk->buf != OOM_JSON) free(chunk->buf); free(chunk); free(w->script); @@ -1814,6 +2403,7 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // dead/tearing down -- tell bridge_end_op (and any bridge_finalize it // triggers) not to touch the napi_ref, since b->env is this same dead env. bridge_end_op(w->bridge, /*env_still_alive=*/env != NULL); + output_flow_release(w->flow); free(w); return; } @@ -1822,23 +2412,39 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // cannot deliver it to JS; free it and return without touching `w` (its // finalization happens only on the sentinel, above). if (env == NULL) { - free(chunk->buf); - free(chunk); + output_flow_cancel(chunk->flow); + output_chunk_release(chunk, /*rollback=*/true); + return; + } + // cancel() already released this payload's credit. Drop any TSFN payload + // that was queued before cancellation instead of calling JavaScript again. + if (output_flow_is_cancelled(chunk->flow)) { + output_chunk_release(chunk, /*rollback=*/false); return; } napi_value buffer; void* buf_data; - napi_create_buffer_copy(env, chunk->len, chunk->buf, &buf_data, &buffer); - - napi_value global; - napi_get_global(env, &global); - native_callback_enter(); - napi_call_function(env, global, js_callback, 1, &buffer, NULL); - native_callback_exit(); - - free(chunk->buf); - free(chunk); + napi_status status = napi_create_buffer_copy( + env, chunk->len, chunk->buf, &buf_data, &buffer + ); + if (status == napi_ok) { + napi_value global; + status = napi_get_global(env, &global); + if (status == napi_ok) { + native_callback_enter(); + status = napi_call_function(env, global, js_callback, 1, &buffer, NULL); + native_callback_exit(); + } + } + if (status != napi_ok) { + output_flow_cancel(chunk->flow); + if (status == napi_pending_exception) { + napi_value exception; + napi_get_and_clear_last_exception(env, &exception); + } + } + output_chunk_release(chunk, /*rollback=*/status != napi_ok); } static void transform_thread_fn(void* arg) { @@ -1906,6 +2512,8 @@ static void transform_thread_fn(void* arg) { struct chunk_data* sentinel = w->sentinel; sentinel->buf = meta_result; sentinel->len = -1; + sentinel->flow = NULL; + sentinel->accounted_bytes = 0; napi_status enq = napi_call_threadsafe_function(w->write_tsfn, sentinel, napi_tsfn_blocking); if (enq != napi_ok) { // See streaming_thread_fn: env tearing down, sentinel dropped, finalize @@ -1939,6 +2547,8 @@ static void transform_thread_fn(void* arg) { free(w->input_mime_type); free(w->input_charset); bridge_end_op(w->bridge, /*env_still_alive=*/false); + output_flow_mark_done(w->flow, NULL); + output_flow_release(w->flow); free(w); } } @@ -2065,6 +2675,15 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i TRANSFORM_FAIL("runScriptTransformEngine: inputCharset must be a string, null, or undefined"); } #undef TRANSFORM_FAIL + w->flow = output_flow_create(); + if (w->flow == NULL) { + free(w->script); free(w->inputs_json); free(w->input_name); + free(w->input_mime_type); free(w->input_charset); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "OOM"); + return NULL; + } // Round-9 (#3, updated round-11 #2): check each resource creation; on // failure release the engine pin (`pinned`, taken at admission) via @@ -2074,6 +2693,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i // before freeing w if it was created. napi_value resource_name; if (napi_create_string_utf8(env, "dwTransform", NAPI_AUTO_LENGTH, &resource_name) != napi_ok) { + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2082,14 +2702,18 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i } if (napi_create_threadsafe_function(env, argv[6], NULL, resource_name, 0, 1, NULL, NULL, NULL, call_js_read, &w->read_tsfn) != napi_ok) { + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "runScriptTransformEngine: failed to create read threadsafe function"); return NULL; } - if (napi_create_threadsafe_function(env, argv[7], NULL, resource_name, 0, 1, NULL, NULL, w, call_js_transform_write, &w->write_tsfn) != napi_ok) { + if (napi_create_threadsafe_function(env, argv[7], NULL, resource_name, + OUTPUT_TSFN_QUEUE_SIZE, 1, NULL, NULL, + w, call_js_transform_write, &w->write_tsfn) != napi_ok) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2108,6 +2732,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i if (w->sentinel == NULL) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); + output_flow_release(w->flow); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2119,6 +2744,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i if (napi_create_promise(env, &w->deferred, &promise) != napi_ok) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); + output_flow_release(w->flow); free(w->sentinel); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2126,6 +2752,18 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i return NULL; } + napi_value controller = output_controller_create(env, promise, w->flow); + if (controller == NULL) { + napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); + napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); + output_flow_release(w->flow); + free(w->sentinel); free(w->script); free(w->inputs_json); + free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + return NULL; + } + // Round-11 (#2): the pin was taken at admission (bridge_begin_op_locked) in // the same critical section as g_active_ops, so a concurrent destroyEngine // could never free this bridge under the admitted op. Just record it on w; @@ -2164,10 +2802,12 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i free(w->input_name); free(w->input_mime_type); free(w->input_charset); + output_flow_mark_done(w->flow, NULL); + output_flow_release(w->flow); free(w); } - return promise; + return controller; } // --- Resolver callback bridge --- @@ -3457,6 +4097,7 @@ static napi_value napi_cleanup(napi_env env, napi_callback_info info) { static void init_g_mutex(void) { uv_mutex_init(&g_mutex); + uv_mutex_init(&g_test_output_mutex); uv_cond_init(&g_teardown_cond); g_native_callback_depth_status = uv_key_create(&g_native_callback_depth); } @@ -3519,13 +4160,90 @@ static napi_value napi_test_async_op_held(napi_env env, napi_callback_info info) } static napi_value napi_test_release_async_op(napi_env env, napi_callback_info info) { - (void)env; - (void)info; - uv_mutex_lock(&g_mutex); - g_test_release_async_op = true; - uv_cond_broadcast(&g_teardown_cond); - uv_mutex_unlock(&g_mutex); + (void)env; + (void)info; + uv_mutex_lock(&g_mutex); + g_test_release_async_op = true; + uv_cond_broadcast(&g_teardown_cond); + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static void set_named_size(napi_env env, napi_value object, const char* name, size_t value) { + napi_value out; + napi_create_double(env, (double)value, &out); + napi_set_named_property(env, object, name, out); +} + +static void set_named_bool(napi_env env, napi_value object, const char* name, bool value) { + napi_value out; + napi_get_boolean(env, value, &out); + napi_set_named_property(env, object, name, out); +} + +static napi_value output_stats_value(napi_env env, const output_flow_stats_t* stats) { + napi_value out; + napi_value value; + napi_create_object(env, &out); + napi_create_double(env, (double)stats->operation_id, &value); + napi_set_named_property(env, out, "operationId", value); + set_named_size(env, out, "outstandingBytes", stats->outstanding_bytes); + set_named_size(env, out, "outstandingChunks", stats->outstanding_chunks); + set_named_size(env, out, "peakBufferedBytes", stats->peak_buffered_bytes); + set_named_size(env, out, "peakBufferedChunks", stats->peak_buffered_chunks); + set_named_size(env, out, "largestChunkBytes", stats->largest_chunk_bytes); + set_named_size(env, out, "highBytes", OUTPUT_HIGH_BYTES); + set_named_size(env, out, "lowBytes", OUTPUT_LOW_BYTES); + set_named_size(env, out, "highChunks", OUTPUT_HIGH_CHUNKS); + set_named_size(env, out, "lowChunks", OUTPUT_LOW_CHUNKS); + set_named_bool(env, out, "paused", stats->paused); + set_named_bool(env, out, "cancelled", stats->cancelled); + set_named_bool(env, out, "done", stats->done); + napi_create_int64(env, stats->live_flows, &value); + napi_set_named_property(env, out, "liveFlows", value); + return out; +} + +static napi_value napi_test_output_stats(napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + napi_get_cb_info(env, info, &argc, argv, NULL, NULL); + uint64_t requested = 0; + if (argc == 1) { + int64_t id; + if (napi_get_value_int64(env, argv[0], &id) != napi_ok || id < 0) { + napi_throw_type_error(env, NULL, "operationId must be a non-negative integer"); + return NULL; + } + requested = (uint64_t)id; + } + output_flow_stats_t stats; + uv_mutex_lock(&g_test_output_mutex); + stats = g_test_last_output_stats; + stats.live_flows = g_test_live_output_flows; + uv_mutex_unlock(&g_test_output_mutex); + if (requested != 0 && requested != stats.operation_id) { + napi_throw_error(env, NULL, "Output operation statistics are no longer current"); + return NULL; + } + return output_stats_value(env, &stats); +} + +static napi_value napi_test_output_operation_id(napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1) { + napi_throw_type_error(env, NULL, "An output controller is required"); return NULL; + } + output_controller_t* holder = NULL; + if (napi_unwrap(env, argv[0], (void**)&holder) != napi_ok || holder == NULL) { + napi_throw_type_error(env, NULL, "Invalid output controller"); + return NULL; + } + napi_value out; + napi_create_double(env, (double)holder->operation_id, &out); + return out; } static napi_value Init(napi_env env, napi_value exports) { @@ -3585,6 +4303,10 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_asyncOpHeld", fn); napi_create_function(env, "__test_releaseAsyncOp", NAPI_AUTO_LENGTH, napi_test_release_async_op, NULL, &fn); napi_set_named_property(env, exports, "__test_releaseAsyncOp", fn); + napi_create_function(env, "__test_outputStats", NAPI_AUTO_LENGTH, napi_test_output_stats, NULL, &fn); + napi_set_named_property(env, exports, "__test_outputStats", fn); + napi_create_function(env, "__test_outputOperationId", NAPI_AUTO_LENGTH, napi_test_output_operation_id, NULL, &fn); + napi_set_named_property(env, exports, "__test_outputOperationId", fn); } return exports; diff --git a/native-lib/node/tests/integration/stream-backpressure.test.ts b/native-lib/node/tests/integration/stream-backpressure.test.ts new file mode 100644 index 00000000..eb7fb74f --- /dev/null +++ b/native-lib/node/tests/integration/stream-backpressure.test.ts @@ -0,0 +1,427 @@ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { DataWeave } from "../../src/dataweave"; +import { buildInputsJson, findLibrary } from "../../src/utils"; + +interface NativeStreamingOperation { + readonly completion: Promise; + acknowledge(bytes: number): void; + cancel(): void; + close(): void; +} + +interface OutputFlowStats { + operationId: number; + outstandingBytes: number; + outstandingChunks: number; + peakBufferedBytes: number; + peakBufferedChunks: number; + largestChunkBytes: number; + highBytes: number; + lowBytes: number; + highChunks: number; + lowChunks: number; + paused: boolean; + cancelled: boolean; + done: boolean; + liveFlows: number; +} + +interface TestAddon { + initialize(libPath: string): void; + createEngine(): number; + destroyEngine(handle: number): void; + runScriptStreamingEngine( + handle: number, + script: string, + inputsJson: string, + chunkCb: (chunk: Buffer) => void + ): NativeStreamingOperation; + runScriptTransformEngine( + handle: number, + script: string, + inputsJson: string, + inputName: string, + inputMimeType: string, + inputCharset: string | null, + readCb: (bufSize: number) => Buffer | null, + writeCb: (chunk: Buffer) => void + ): NativeStreamingOperation; + cleanup(): Promise; + __test_outputStats(operationId?: number): OutputFlowStats; + __test_outputOperationId(operation: NativeStreamingOperation): number; +} + +const ADDON_PATH = "../../build/Release/dwlib_addon.node"; +// eslint-disable-next-line @typescript-eslint/no-var-requires +const addon = require(ADDON_PATH) as TestAddon; +const LIB_PATH = findLibrary(); +const LARGE_SIZE = 2 * 1024 * 1024 + 32771; +const PASSTHROUGH_SCRIPT = + "output application/octet-stream deferred=true\n---\npayload"; + +function patternedBytes(size = LARGE_SIZE): Buffer { + const bytes = Buffer.allocUnsafe(size); + for (let i = 0; i < size; i++) bytes[i] = 32 + (i % 95); + return bytes; +} + +function octetStreamInputs(payload: Buffer): string { + return buildInputsJson({ + payload: { + content: payload, + mimeType: "application/octet-stream", + }, + }); +} + +function immediate(): Promise { + return new Promise((resolve) => setImmediate(resolve)); +} + +async function withTimeout(promise: Promise, label: string, timeoutMs = 10000): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + promise, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${label} timed out`)), timeoutMs); + }), + ]); + } finally { + if (timer !== undefined) clearTimeout(timer); + } +} + +async function waitForStats( + predicate: (stats: OutputFlowStats) => boolean, + label: string, + operationId?: number, + timeoutMs = 10000 +): Promise { + const deadline = Date.now() + timeoutMs; + while (true) { + const stats = operationId === undefined + ? addon.__test_outputStats() + : addon.__test_outputStats(operationId); + if (predicate(stats)) return stats; + if (Date.now() >= deadline) { + throw new Error(`${label} timed out; last stats: ${JSON.stringify(stats)}`); + } + await immediate(); + } +} + +function expectBounded(stats: OutputFlowStats): void { + expect(stats.peakBufferedChunks).toBeLessThanOrEqual(stats.highChunks + 1); + expect(stats.peakBufferedBytes).toBeLessThanOrEqual( + stats.highBytes + stats.largestChunkBytes + ); +} + +async function waitForPendingChunk(pending: Buffer[], label: string): Promise { + const deadline = Date.now() + 10000; + while (pending.length === 0) { + if (Date.now() >= deadline) throw new Error(`${label} timed out`); + await immediate(); + } + return pending.shift()!; +} + +async function drainAfterPause( + operation: NativeStreamingOperation, + operationId: number, + pending: Buffer[], + received: Buffer[], + settled: () => boolean +): Promise { + const paused = await waitForStats( + (stats) => stats.paused && stats.outstandingChunks === pending.length, + "producer pause with every admitted callback delivered", + operationId + ); + expectBounded(paused); + + const deliveredAtPause = received.length; + let projectedBytes = paused.outstandingBytes; + let projectedChunks = paused.outstandingChunks; + while (projectedBytes > paused.lowBytes || projectedChunks > paused.lowChunks) { + const chunk = await waitForPendingChunk(pending, "low-water acknowledgement"); + operation.acknowledge(chunk.length); + projectedBytes -= chunk.length; + projectedChunks--; + } + + expect(projectedBytes).toBeLessThanOrEqual(paused.lowBytes); + expect(projectedChunks).toBeLessThanOrEqual(paused.lowChunks); + await waitForStats( + () => received.length > deliveredAtPause, + "producer wake below both low watermarks", + operationId + ); + + const deadline = Date.now() + 15000; + while (!settled() || pending.length > 0) { + if (pending.length > 0) { + operation.acknowledge(pending.shift()!.length); + } + if (Date.now() >= deadline) { + throw new Error( + `stepwise drain timed out; stats: ${JSON.stringify(addon.__test_outputStats(operationId))}` + ); + } + await immediate(); + } + + return withTimeout(operation.completion, "native completion after drain"); +} + +async function runWithEngine( + body: (handle: number) => Promise +): Promise { + const handle = addon.createEngine(); + try { + await body(handle); + } finally { + addon.destroyEngine(handle); + } +} + +beforeAll(() => { + addon.initialize(LIB_PATH); +}); + +afterAll(async () => { + await addon.cleanup(); +}); + +describe.sequential("native Node output flow control", () => { + it("returns a validated, idempotent operation controller", async () => { + await runWithEngine(async (handle) => { + const chunks: Buffer[] = []; + let operation: NativeStreamingOperation | undefined; + try { + operation = addon.runScriptStreamingEngine( + handle, + "output application/json deferred=true --- [1, 2, 3]", + "{}", + (chunk) => chunks.push(chunk) + ); + + expect(operation).toEqual( + expect.objectContaining({ + completion: expect.any(Promise), + acknowledge: expect.any(Function), + cancel: expect.any(Function), + close: expect.any(Function), + }) + ); + + for (const invalid of [-1, 0.5, Number.NaN, Number.POSITIVE_INFINITY, "1"]) { + expect(() => operation!.acknowledge(invalid as number)).toThrow(); + } + + const raw = await withTimeout(operation.completion, "controller completion"); + expect(JSON.parse(raw).success).toBe(true); + for (const chunk of chunks) operation.acknowledge(chunk.length); + + expect(() => operation.cancel()).not.toThrow(); + expect(() => operation.cancel()).not.toThrow(); + expect(() => operation.close()).not.toThrow(); + expect(() => operation.close()).not.toThrow(); + expect(() => operation.acknowledge(1)).not.toThrow(); + } finally { + if (operation !== undefined) { + operation.cancel?.(); + operation.close?.(); + const completion = operation.completion ?? (operation as unknown as Promise); + await withTimeout(Promise.resolve(completion), "controller test cleanup"); + } + } + }); + }); + + it("bounds a paused streaming producer and resumes only after low-water credit", async () => { + await runWithEngine(async (handle) => { + expect(typeof addon.__test_outputStats).toBe("function"); + const expected = patternedBytes(); + const received: Buffer[] = []; + const pending: Buffer[] = []; + let operation!: NativeStreamingOperation; + let operationId!: number; + let completionSettled = false; + + operation = addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(expected), + (chunk) => { + received.push(chunk); + pending.push(chunk); + if (received.length === 1) operation.acknowledge(pending.shift()!.length); + } + ); + operationId = addon.__test_outputOperationId(operation); + operation.completion.finally(() => { completionSettled = true; }).catch(() => {}); + + try { + const paused = await waitForStats( + (stats) => stats.paused, + "streaming producer pause", + operationId + ); + expect(completionSettled).toBe(false); + expectBounded(paused); + + const raw = await drainAfterPause( + operation, + operationId, + pending, + received, + () => completionSettled + ); + expect(Buffer.concat(received)).toEqual(expected); + expect(JSON.parse(raw)).toMatchObject({ + success: true, + mimeType: "application/octet-stream", + }); + + const drained = addon.__test_outputStats(operationId); + expect(drained).toMatchObject({ + outstandingBytes: 0, + outstandingChunks: 0, + done: true, + }); + } finally { + operation.cancel(); + operation.close(); + await withTimeout(operation.completion, "streaming test cleanup"); + } + }); + }, 30000); + + it("bounds transform output without changing the transform read bridge", async () => { + await runWithEngine(async (handle) => { + const expected = patternedBytes(); + const received: Buffer[] = []; + const pending: Buffer[] = []; + let offset = 0; + let operation!: NativeStreamingOperation; + let operationId!: number; + let completionSettled = false; + + operation = addon.runScriptTransformEngine( + handle, + PASSTHROUGH_SCRIPT, + "{}", + "payload", + "application/octet-stream", + null, + (bufSize) => { + if (offset >= expected.length) return null; + const chunk = expected.subarray(offset, Math.min(offset + bufSize, expected.length)); + offset += chunk.length; + return chunk; + }, + (chunk) => { + received.push(chunk); + pending.push(chunk); + if (received.length === 1) operation.acknowledge(pending.shift()!.length); + } + ); + operationId = addon.__test_outputOperationId(operation); + operation.completion.finally(() => { completionSettled = true; }).catch(() => {}); + + try { + const paused = await waitForStats( + (stats) => stats.paused, + "transform producer pause", + operationId + ); + expect(completionSettled).toBe(false); + expectBounded(paused); + + const raw = await drainAfterPause( + operation, + operationId, + pending, + received, + () => completionSettled + ); + expect(offset).toBe(expected.length); + expect(Buffer.concat(received)).toEqual(expected); + expect(JSON.parse(raw)).toMatchObject({ + success: true, + mimeType: "application/octet-stream", + }); + } finally { + operation.cancel(); + operation.close(); + await withTimeout(operation.completion, "transform test cleanup"); + } + }); + }, 30000); + + it("settles and closes when an iterator returns while the producer is paused", async () => { + const dw = new DataWeave(); + dw.initialize(); + const stream = dw.runStreaming(PASSTHROUGH_SCRIPT, { + payload: { + content: patternedBytes(), + mimeType: "application/octet-stream", + }, + }); + + try { + const first = await withTimeout(stream.next(), "first iterator chunk"); + expect(first.done).toBe(false); + await waitForStats((stats) => stats.paused, "iterator producer pause"); + + await withTimeout(stream.return(undefined), "paused iterator return"); + await withTimeout(dw.cleanup(), "cleanup after paused iterator return"); + await waitForStats( + (stats) => stats.cancelled && stats.done && stats.outstandingBytes === 0, + "cancelled iterator settlement" + ); + expect(addon.__test_outputStats().liveFlows).toBe(0); + + await expect(stream.next()).resolves.toEqual({ done: true, value: undefined }); + dw.initialize(); + expect(dw.run("output application/json --- 6 * 7").getString()).toBe("42"); + } finally { + await withTimeout(dw.cleanup(), "iterator test final cleanup"); + } + }, 30000); + + it("DataWeave.cleanup cancels a paused producer and deterministically settles its iterator", async () => { + const dw = new DataWeave(); + dw.initialize(); + const anchor = new DataWeave(); + anchor.initialize(); + const stream = dw.runStreaming(PASSTHROUGH_SCRIPT, { + payload: { + content: patternedBytes(), + mimeType: "application/octet-stream", + }, + }); + + try { + const first = await withTimeout(stream.next(), "first cleanup-test chunk"); + expect(first.done).toBe(false); + await waitForStats((stats) => stats.paused, "cleanup-test producer pause"); + + await withTimeout(dw.cleanup(), "DataWeave.cleanup while producer paused"); + await waitForStats( + (stats) => stats.cancelled && stats.done && stats.outstandingChunks === 0, + "cleanup cancellation settlement" + ); + expect(addon.__test_outputStats().liveFlows).toBe(0); + await expect(stream.next()).resolves.toEqual({ done: true, value: undefined }); + + dw.initialize(); + expect(dw.run("output application/json --- 20 + 22").getString()).toBe("42"); + } finally { + await withTimeout(dw.cleanup(), "cleanup test final cleanup"); + await withTimeout(anchor.cleanup(), "cleanup test anchor cleanup"); + } + }, 30000); +}); From 9aa486b792fca9f465034d782f76d649d5a7a721 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Wed, 9 Sep 2026 09:10:57 -0300 Subject: [PATCH 36/48] fix(node): validate asynchronous output credits --- native-lib/node/src/addon.c | 333 +++++++++++++++--- native-lib/node/src/ffi.ts | 12 +- native-lib/node/src/stream.ts | 26 +- .../integration/dataweave-resolver.test.ts | 31 +- .../integration/stream-backpressure.test.ts | 226 ++++++++++-- .../integration/teardown-deadlock.test.ts | 7 +- .../tests/unit/dataweave-initialize.test.ts | 79 +++-- native-lib/node/tests/unit/stream.test.ts | 129 +++++-- 8 files changed, 661 insertions(+), 182 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 2cf4607a..8943d79a 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -199,6 +199,7 @@ static long long g_test_resolver_ref_deletes = 0; static bool g_test_hold_next_async_op = false; static bool g_test_async_op_held = false; static bool g_test_release_async_op = false; +static bool g_test_fail_next_output_settlement = false; // One record per napi_env that has ever taken an init reference (via // initialize()). init_refs is that env's net initialize()-minus-cleanup() @@ -1201,6 +1202,9 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { typedef struct output_credit { size_t bytes; + uint64_t sequence; + bool delivered; + bool acknowledged; struct output_credit* previous; struct output_credit* next; } output_credit_t; @@ -1218,6 +1222,7 @@ typedef struct output_flow { bool done; unsigned int refs; uint64_t operation_id; + uint64_t next_sequence; output_credit_t* credit_head; output_credit_t* credit_tail; napi_ref thenable_ref; @@ -1324,7 +1329,8 @@ static void output_flow_release(output_flow_t* flow) { } // Only the native producer waits here. No caller holds g_mutex while waiting. -static bool output_flow_reserve(output_flow_t* flow, size_t bytes) { +static bool output_flow_reserve( + output_flow_t* flow, size_t bytes, uint64_t* sequence_out) { if (flow == NULL) return false; output_credit_t* credit = (output_credit_t*)calloc(1, sizeof(output_credit_t)); if (credit == NULL) return false; @@ -1370,6 +1376,7 @@ static bool output_flow_reserve(output_flow_t* flow, size_t bytes) { // wait, which drains prior credit; ordinary reservations are already bounded. flow->outstanding_bytes += bytes; flow->outstanding_chunks++; + credit->sequence = ++flow->next_sequence; credit->previous = flow->credit_tail; if (flow->credit_tail != NULL) flow->credit_tail->next = credit; else flow->credit_head = credit; @@ -1382,41 +1389,93 @@ static bool output_flow_reserve(output_flow_t* flow, size_t bytes) { } if (bytes > flow->largest_chunk_bytes) flow->largest_chunk_bytes = bytes; output_flow_record_stats_locked(flow); + *sequence_out = credit->sequence; uv_mutex_unlock(&flow->mutex); return true; } -static void output_flow_acknowledge(output_flow_t* flow, size_t bytes) { - if (flow == NULL) return; +typedef enum { + OUTPUT_ACK_IGNORED = 0, + OUTPUT_ACK_ACCEPTED, + OUTPUT_ACK_INVALID_SEQUENCE, + OUTPUT_ACK_NOT_DELIVERED, + OUTPUT_ACK_OUT_OF_ORDER, + OUTPUT_ACK_BYTES_MISMATCH, + OUTPUT_ACK_DUPLICATE, +} output_ack_result_t; + +static output_ack_result_t output_flow_acknowledge( + output_flow_t* flow, uint64_t sequence, size_t bytes) { + if (flow == NULL) return OUTPUT_ACK_IGNORED; uv_mutex_lock(&flow->mutex); + if (flow->cancelled) { + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + return OUTPUT_ACK_IGNORED; + } + output_credit_t* credit = flow->credit_head; - // Task 8 acknowledges each dequeued chunk exactly once in FIFO order. A - // mismatched byte count is a no-op rather than corrupting the queue. - if (!flow->cancelled && credit != NULL && credit->bytes == bytes && - bytes <= flow->outstanding_bytes && flow->outstanding_chunks > 0) { + output_credit_t* requested = credit; + while (requested != NULL && requested->sequence != sequence) { + requested = requested->next; + } + output_ack_result_t result; + if (requested == NULL) { + result = flow->done + ? OUTPUT_ACK_IGNORED + : sequence <= flow->next_sequence + ? OUTPUT_ACK_DUPLICATE + : OUTPUT_ACK_INVALID_SEQUENCE; + } else if (!requested->delivered) { + result = OUTPUT_ACK_NOT_DELIVERED; + } else if (requested != credit) { + result = OUTPUT_ACK_OUT_OF_ORDER; + } else if (requested->bytes != bytes) { + result = OUTPUT_ACK_BYTES_MISMATCH; + } else if (requested->acknowledged) { + result = OUTPUT_ACK_DUPLICATE; + } else if (bytes <= flow->outstanding_bytes && flow->outstanding_chunks > 0) { + requested->acknowledged = true; flow->outstanding_bytes -= bytes; flow->outstanding_chunks--; - flow->credit_head = credit->next; + flow->credit_head = requested->next; if (flow->credit_head != NULL) flow->credit_head->previous = NULL; else flow->credit_tail = NULL; - free(credit); + free(requested); if (flow->paused && flow->outstanding_bytes <= OUTPUT_LOW_BYTES && flow->outstanding_chunks <= OUTPUT_LOW_CHUNKS) { uv_cond_broadcast(&flow->cond); } + result = OUTPUT_ACK_ACCEPTED; + } else { + result = OUTPUT_ACK_INVALID_SEQUENCE; } output_flow_record_stats_locked(flow); uv_mutex_unlock(&flow->mutex); + return result; +} + +static bool output_flow_mark_delivered(output_flow_t* flow, uint64_t sequence) { + if (flow == NULL) return false; + uv_mutex_lock(&flow->mutex); + output_credit_t* credit = flow->credit_head; + while (credit != NULL && credit->sequence != sequence) credit = credit->next; + bool delivered = !flow->cancelled && !flow->done && credit != NULL; + if (delivered) credit->delivered = true; + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); + return delivered; } // Enqueue/allocation rollback always targets the newest reservation because // callbacks reserve and enqueue serially on the sole producer worker. -static void output_flow_rollback(output_flow_t* flow, size_t bytes) { +static void output_flow_rollback( + output_flow_t* flow, uint64_t sequence, size_t bytes) { if (flow == NULL) return; uv_mutex_lock(&flow->mutex); output_credit_t* credit = flow->credit_tail; - if (credit != NULL && credit->bytes == bytes && + if (credit != NULL && credit->sequence == sequence && credit->bytes == bytes && bytes <= flow->outstanding_bytes && flow->outstanding_chunks > 0) { flow->outstanding_bytes -= bytes; flow->outstanding_chunks--; @@ -1496,6 +1555,11 @@ typedef struct output_controller { uint64_t operation_id; } output_controller_t; +static const napi_type_tag OUTPUT_CONTROLLER_TAG = { + 0x9d29436c7a6848e1ULL, + 0xa8e6af21e52d879bULL, +}; + static void output_controller_cancel(output_controller_t* holder) { if (holder == NULL) return; uv_mutex_lock(&holder->mutex); @@ -1544,6 +1608,12 @@ static output_controller_t* output_controller_unwrap( napi_throw_type_error(env, NULL, "Missing output controller argument"); return NULL; } + bool tagged = false; + if (napi_check_object_type_tag(env, this_arg, &OUTPUT_CONTROLLER_TAG, &tagged) != napi_ok || + !tagged) { + napi_throw_type_error(env, NULL, "Invalid output controller receiver"); + return NULL; + } output_controller_t* holder = NULL; if (napi_unwrap(env, this_arg, (void**)&holder) != napi_ok || holder == NULL) { napi_throw_type_error(env, NULL, "Invalid output controller receiver"); @@ -1553,17 +1623,26 @@ static output_controller_t* output_controller_unwrap( } static napi_value napi_output_acknowledge(napi_env env, napi_callback_info info) { - napi_value argv[1]; - output_controller_t* holder = output_controller_unwrap(env, info, 1, argv); + napi_value argv[2]; + output_controller_t* holder = output_controller_unwrap(env, info, 2, argv); if (holder == NULL) return NULL; napi_valuetype type; + uint64_t sequence; double value; - if (napi_typeof(env, argv[0], &type) != napi_ok || type != napi_number || - napi_get_value_double(env, argv[0], &value) != napi_ok || + bool lossless = false; + if (napi_typeof(env, argv[0], &type) != napi_ok || type != napi_bigint || + napi_get_value_bigint_uint64(env, argv[0], &sequence, &lossless) != napi_ok || + !lossless || sequence == 0) { + napi_throw_range_error(env, NULL, + "acknowledge(sequence, bytes) requires a positive uint64 BigInt sequence"); + return NULL; + } + if (napi_typeof(env, argv[1], &type) != napi_ok || type != napi_number || + napi_get_value_double(env, argv[1], &value) != napi_ok || value != value || value < 0 || value > 9007199254740991.0 || value > (double)SIZE_MAX || value != (double)(size_t)value) { napi_throw_range_error(env, NULL, - "acknowledge(bytes) requires a finite non-negative safe integer"); + "acknowledge(sequence, bytes) requires finite non-negative safe integer bytes"); return NULL; } uv_mutex_lock(&holder->mutex); @@ -1571,8 +1650,28 @@ static napi_value napi_output_acknowledge(napi_env env, napi_callback_info info) if (flow != NULL) output_flow_retain(flow); uv_mutex_unlock(&holder->mutex); if (flow != NULL) { - output_flow_acknowledge(flow, (size_t)value); + output_ack_result_t result = output_flow_acknowledge(flow, sequence, (size_t)value); output_flow_release(flow); + switch (result) { + case OUTPUT_ACK_IGNORED: + case OUTPUT_ACK_ACCEPTED: + return NULL; + case OUTPUT_ACK_INVALID_SEQUENCE: + napi_throw_range_error(env, NULL, "Unknown output sequence"); + return NULL; + case OUTPUT_ACK_NOT_DELIVERED: + napi_throw_error(env, NULL, "Output sequence has not been delivered"); + return NULL; + case OUTPUT_ACK_OUT_OF_ORDER: + napi_throw_error(env, NULL, "Output acknowledgement is out of order"); + return NULL; + case OUTPUT_ACK_BYTES_MISMATCH: + napi_throw_range_error(env, NULL, "Output acknowledgement byte count does not match"); + return NULL; + case OUTPUT_ACK_DUPLICATE: + napi_throw_error(env, NULL, "Output sequence was already acknowledged"); + return NULL; + } } return NULL; } @@ -1600,6 +1699,22 @@ static napi_value napi_output_promise_method(napi_env env, napi_callback_info in napi_throw_type_error(env, NULL, "Invalid output controller promise method"); return NULL; } + bool tagged = false; + if (napi_check_object_type_tag(env, controller, &OUTPUT_CONTROLLER_TAG, &tagged) != napi_ok || + !tagged) { + napi_throw_type_error(env, NULL, "Invalid output controller receiver"); + return NULL; + } + const char* name = (const char*)data; + napi_value completion; + napi_value method; + napi_value result; + if (napi_get_named_property(env, controller, "completion", &completion) != napi_ok || + napi_get_named_property(env, completion, name, &method) != napi_ok || + napi_call_function(env, completion, method, argc, argv, &result) != napi_ok) { + return NULL; + } + output_controller_t* holder = NULL; if (napi_unwrap(env, controller, (void**)&holder) != napi_ok || holder == NULL) { napi_throw_type_error(env, NULL, "Invalid output controller receiver"); @@ -1610,21 +1725,11 @@ static napi_value napi_output_promise_method(napi_env env, napi_callback_info in if (flow != NULL) output_flow_retain(flow); uv_mutex_unlock(&holder->mutex); if (flow != NULL) { - // Retain only once a caller actually treats the controller as thenable. - // A simply dropped controller still finalizes promptly and cancels. + // Retain only after the Promise method call succeeds. Failed assimilation + // must not root a controller that no caller can use to close the flow. output_flow_retain_thenable(flow, env, controller); output_flow_release(flow); } - - const char* name = (const char*)data; - napi_value completion; - napi_value method; - napi_value result; - if (napi_get_named_property(env, controller, "completion", &completion) != napi_ok || - napi_get_named_property(env, completion, name, &method) != napi_ok || - napi_call_function(env, completion, method, argc, argv, &result) != napi_ok) { - return NULL; - } return result; } @@ -1677,6 +1782,16 @@ static napi_value output_controller_create( napi_throw_error(env, NULL, "Failed to wrap output controller"); return NULL; } + if (napi_type_tag_object(env, controller, &OUTPUT_CONTROLLER_TAG) != napi_ok) { + void* removed = NULL; + napi_remove_wrap(env, controller, &removed); + output_controller_cancel(holder); + output_controller_close(holder); + uv_mutex_destroy(&holder->mutex); + free(holder); + napi_throw_error(env, NULL, "Failed to tag output controller"); + return NULL; + } return controller; } @@ -1686,6 +1801,44 @@ static napi_value output_controller_create( // must first check `buf != OOM_JSON` -- freeing a static pointer is UB. The // wording matches the existing terse worker error style ("Empty response"). static const char OOM_JSON[] = "{\"success\":false,\"error\":\"Out of memory\"}"; +static const char SETTLEMENT_ERROR_JSON[] = + "{\"success\":false,\"error\":\"Failed to settle native output completion\"}"; + +static bool test_consume_output_settlement_fault(void) { + if (!g_test_hooks) return false; + uv_mutex_lock(&g_test_output_mutex); + bool fail = g_test_fail_next_output_settlement; + g_test_fail_next_output_settlement = false; + uv_mutex_unlock(&g_test_output_mutex); + return fail; +} + +static void settle_output_deferred( + napi_env env, napi_deferred deferred, const char* result_json) { + napi_value result; + napi_status status = napi_create_string_utf8( + env, result_json, strlen(result_json), &result + ); + if (status == napi_ok) { + status = test_consume_output_settlement_fault() + ? napi_generic_failure + : napi_resolve_deferred(env, deferred, result); + } + if (status == napi_ok) return; + + napi_value fallback; + if (napi_create_string_utf8( + env, SETTLEMENT_ERROR_JSON, NAPI_AUTO_LENGTH, &fallback) == napi_ok) { + if (napi_resolve_deferred(env, deferred, fallback) == napi_ok) return; + } + + // napi_get_undefined does not allocate. If string creation itself failed + // under memory pressure, still make a final allocation-free settlement + // attempt rather than leaving the operation permanently pending. + if (napi_get_undefined(env, &fallback) == napi_ok) { + napi_resolve_deferred(env, deferred, fallback); + } +} // chunk_data with len == -1 is a sentinel indicating completion (buf holds meta JSON) struct chunk_data { @@ -1693,6 +1846,7 @@ struct chunk_data { int len; output_flow_t* flow; size_t accounted_bytes; + uint64_t sequence; }; struct streaming_work { @@ -1720,7 +1874,9 @@ struct streaming_work { static void output_chunk_release(struct chunk_data* chunk, bool rollback) { if (chunk == NULL) return; if (chunk->flow != NULL) { - if (rollback) output_flow_rollback(chunk->flow, chunk->accounted_bytes); + if (rollback) { + output_flow_rollback(chunk->flow, chunk->sequence, chunk->accounted_bytes); + } output_flow_release(chunk->flow); } free(chunk->buf); @@ -1743,12 +1899,7 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // would leak `w` and could strand a bridge marked for deferred destruction // indefinitely. if (env != NULL) { - napi_value result; - if (napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result) == napi_ok) { - napi_resolve_deferred(env, w->deferred, result); - } else { - output_flow_cancel(w->flow); - } + settle_output_deferred(env, w->deferred, chunk->buf); } output_flow_mark_done(w->flow, env); @@ -1786,16 +1937,25 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v } napi_value buffer; + napi_value sequence; void* buf_data; napi_status status = napi_create_buffer_copy( env, chunk->len, chunk->buf, &buf_data, &buffer ); + if (status == napi_ok) { + status = napi_create_bigint_uint64(env, chunk->sequence, &sequence); + } if (status == napi_ok) { napi_value global; status = napi_get_global(env, &global); if (status == napi_ok) { + if (!output_flow_mark_delivered(chunk->flow, chunk->sequence)) { + output_chunk_release(chunk, /*rollback=*/false); + return; + } native_callback_enter(); - status = napi_call_function(env, global, js_callback, 1, &buffer, NULL); + napi_value argv[2] = {buffer, sequence}; + status = napi_call_function(env, global, js_callback, 2, argv, NULL); native_callback_exit(); } } @@ -1812,21 +1972,22 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v static int streaming_write_cb(void* ctx, const char* buf, int len) { struct streaming_work* w = (struct streaming_work*)ctx; if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; - if (!output_flow_reserve(w->flow, (size_t)len)) return -1; + uint64_t sequence; + if (!output_flow_reserve(w->flow, (size_t)len, &sequence)) return -1; // Round-9 (#2): OOM here must not deref NULL / memcpy into NULL. Returning -1 // aborts the native run cleanly (write-callback contract: non-zero stops the // DataWeave run); the worker then still produces a terminal meta_result and // sentinel, so the op resolves. struct chunk_data* chunk = malloc(sizeof(struct chunk_data)); if (chunk == NULL) { - output_flow_rollback(w->flow, (size_t)len); + output_flow_rollback(w->flow, sequence, (size_t)len); output_flow_cancel(w->flow); return -1; } chunk->buf = len == 0 ? NULL : malloc((size_t)len); if (len > 0 && chunk->buf == NULL) { free(chunk); - output_flow_rollback(w->flow, (size_t)len); + output_flow_rollback(w->flow, sequence, (size_t)len); output_flow_cancel(w->flow); return -1; } @@ -1834,6 +1995,7 @@ static int streaming_write_cb(void* ctx, const char* buf, int len) { chunk->len = len; chunk->flow = w->flow; chunk->accounted_bytes = (size_t)len; + chunk->sequence = sequence; output_flow_retain(w->flow); napi_status status = napi_call_threadsafe_function( @@ -1916,6 +2078,7 @@ static void streaming_thread_fn(void* arg) { sentinel->len = -1; sentinel->flow = NULL; sentinel->accounted_bytes = 0; + sentinel->sequence = 0; napi_status enq = napi_call_threadsafe_function(w->tsfn, sentinel, napi_tsfn_blocking); if (enq != napi_ok) { // The env is tearing down (napi_closing): the sentinel was dropped and @@ -2162,9 +2325,10 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i bridge_end_op(w->bridge, /*env_still_alive=*/true); napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); - napi_value result; - napi_create_string_utf8(env, "{\"success\":false,\"error\":\"Failed to spawn streaming worker thread\"}", NAPI_AUTO_LENGTH, &result); - napi_resolve_deferred(env, w->deferred, result); + settle_output_deferred( + env, w->deferred, + "{\"success\":false,\"error\":\"Failed to spawn streaming worker thread\"}" + ); free(w->sentinel); free(w->script); @@ -2328,19 +2492,20 @@ static int transform_read_cb(void* ctx, char* buf, int buf_size) { static int transform_write_cb(void* ctx, const char* buf, int len) { struct transform_work* w = (struct transform_work*)ctx; if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; - if (!output_flow_reserve(w->flow, (size_t)len)) return -1; + uint64_t sequence; + if (!output_flow_reserve(w->flow, (size_t)len, &sequence)) return -1; // Round-9 (#2): OOM-safe, mirrors streaming_write_cb. Return -1 to abort the // native run cleanly; the worker still delivers a terminal sentinel. struct chunk_data* chunk = malloc(sizeof(struct chunk_data)); if (chunk == NULL) { - output_flow_rollback(w->flow, (size_t)len); + output_flow_rollback(w->flow, sequence, (size_t)len); output_flow_cancel(w->flow); return -1; } chunk->buf = len == 0 ? NULL : malloc((size_t)len); if (len > 0 && chunk->buf == NULL) { free(chunk); - output_flow_rollback(w->flow, (size_t)len); + output_flow_rollback(w->flow, sequence, (size_t)len); output_flow_cancel(w->flow); return -1; } @@ -2348,6 +2513,7 @@ static int transform_write_cb(void* ctx, const char* buf, int len) { chunk->len = len; chunk->flow = w->flow; chunk->accounted_bytes = (size_t)len; + chunk->sequence = sequence; output_flow_retain(w->flow); napi_status status = napi_call_threadsafe_function( @@ -2377,12 +2543,7 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // would leak `w` and could strand a bridge marked for deferred destruction // indefinitely. if (env != NULL) { - napi_value result; - if (napi_create_string_utf8(env, chunk->buf, strlen(chunk->buf), &result) == napi_ok) { - napi_resolve_deferred(env, w->deferred, result); - } else { - output_flow_cancel(w->flow); - } + settle_output_deferred(env, w->deferred, chunk->buf); } output_flow_mark_done(w->flow, env); @@ -2424,16 +2585,25 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* } napi_value buffer; + napi_value sequence; void* buf_data; napi_status status = napi_create_buffer_copy( env, chunk->len, chunk->buf, &buf_data, &buffer ); + if (status == napi_ok) { + status = napi_create_bigint_uint64(env, chunk->sequence, &sequence); + } if (status == napi_ok) { napi_value global; status = napi_get_global(env, &global); if (status == napi_ok) { + if (!output_flow_mark_delivered(chunk->flow, chunk->sequence)) { + output_chunk_release(chunk, /*rollback=*/false); + return; + } native_callback_enter(); - status = napi_call_function(env, global, js_callback, 1, &buffer, NULL); + napi_value argv[2] = {buffer, sequence}; + status = napi_call_function(env, global, js_callback, 2, argv, NULL); native_callback_exit(); } } @@ -2514,6 +2684,7 @@ static void transform_thread_fn(void* arg) { sentinel->len = -1; sentinel->flow = NULL; sentinel->accounted_bytes = 0; + sentinel->sequence = 0; napi_status enq = napi_call_threadsafe_function(w->write_tsfn, sentinel, napi_tsfn_blocking); if (enq != napi_ok) { // See streaming_thread_fn: env tearing down, sentinel dropped, finalize @@ -2792,9 +2963,10 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); - napi_value result; - napi_create_string_utf8(env, "{\"success\":false,\"error\":\"Failed to spawn transform worker thread\"}", NAPI_AUTO_LENGTH, &result); - napi_resolve_deferred(env, w->deferred, result); + settle_output_deferred( + env, w->deferred, + "{\"success\":false,\"error\":\"Failed to spawn transform worker thread\"}" + ); free(w->sentinel); free(w->script); @@ -4169,6 +4341,45 @@ static napi_value napi_test_release_async_op(napi_env env, napi_callback_info in return NULL; } +typedef struct foreign_wrapped_value { + uint64_t marker; +} foreign_wrapped_value_t; + +static void foreign_wrapped_finalize(napi_env env, void* data, void* hint) { + (void)env; + (void)hint; + free(data); +} + +static napi_value napi_test_create_foreign_wrapped_object( + napi_env env, napi_callback_info info) { + (void)info; + foreign_wrapped_value_t* value = calloc(1, sizeof(foreign_wrapped_value_t)); + if (value == NULL) { + napi_throw_error(env, NULL, "OOM"); + return NULL; + } + value->marker = 424242; + napi_value object; + if (napi_create_object(env, &object) != napi_ok || + napi_wrap(env, object, value, foreign_wrapped_finalize, NULL, NULL) != napi_ok) { + free(value); + napi_throw_error(env, NULL, "Failed to create foreign wrapped object"); + return NULL; + } + return object; +} + +static napi_value napi_test_fail_next_output_settlement( + napi_env env, napi_callback_info info) { + (void)env; + (void)info; + uv_mutex_lock(&g_test_output_mutex); + g_test_fail_next_output_settlement = true; + uv_mutex_unlock(&g_test_output_mutex); + return NULL; +} + static void set_named_size(napi_env env, napi_value object, const char* name, size_t value) { napi_value out; napi_create_double(env, (double)value, &out); @@ -4236,6 +4447,12 @@ static napi_value napi_test_output_operation_id(napi_env env, napi_callback_info napi_throw_type_error(env, NULL, "An output controller is required"); return NULL; } + bool tagged = false; + if (napi_check_object_type_tag(env, argv[0], &OUTPUT_CONTROLLER_TAG, &tagged) != napi_ok || + !tagged) { + napi_throw_type_error(env, NULL, "Invalid output controller"); + return NULL; + } output_controller_t* holder = NULL; if (napi_unwrap(env, argv[0], (void**)&holder) != napi_ok || holder == NULL) { napi_throw_type_error(env, NULL, "Invalid output controller"); @@ -4307,6 +4524,10 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_outputStats", fn); napi_create_function(env, "__test_outputOperationId", NAPI_AUTO_LENGTH, napi_test_output_operation_id, NULL, &fn); napi_set_named_property(env, exports, "__test_outputOperationId", fn); + napi_create_function(env, "__test_createForeignWrappedObject", NAPI_AUTO_LENGTH, napi_test_create_foreign_wrapped_object, NULL, &fn); + napi_set_named_property(env, exports, "__test_createForeignWrappedObject", fn); + napi_create_function(env, "__test_failNextOutputSettlement", NAPI_AUTO_LENGTH, napi_test_fail_next_output_settlement, NULL, &fn); + napi_set_named_property(env, exports, "__test_failNextOutputSettlement", fn); } return exports; diff --git a/native-lib/node/src/ffi.ts b/native-lib/node/src/ffi.ts index 21b1cce6..113bf92f 100644 --- a/native-lib/node/src/ffi.ts +++ b/native-lib/node/src/ffi.ts @@ -4,11 +4,13 @@ import type { ModuleResolver } from "./resolver"; export interface NativeStreamingOperation { readonly completion: Promise; - acknowledge(bytes: number): void; + acknowledge(sequence: bigint, bytes: number): void; cancel(): void; close(): void; } +export type NativeChunkCallback = (chunk: Buffer, sequence: bigint) => void; + interface NativeAddon { initialize(libPath: string): void; createEngine(): number; @@ -19,7 +21,7 @@ interface NativeAddon { handle: number, script: string, inputsJson: string, - chunkCb: (chunk: Buffer) => void + chunkCb: NativeChunkCallback ): NativeStreamingOperation; runScriptTransformEngine( handle: number, @@ -29,7 +31,7 @@ interface NativeAddon { inputMimeType: string, inputCharset: string | null, readCb: (bufSize: number) => Buffer | null, - writeCb: (chunk: Buffer) => void + writeCb: NativeChunkCallback ): NativeStreamingOperation; cleanup(): Promise; } @@ -84,7 +86,7 @@ export function runScriptStreamingEngine( handle: number, script: string, inputsJson: string, - chunkCb: (chunk: Buffer) => void + chunkCb: NativeChunkCallback ): NativeStreamingOperation { return callNative(() => getAddon().runScriptStreamingEngine(handle, script, inputsJson, chunkCb) @@ -99,7 +101,7 @@ export function runScriptTransformEngine( inputMimeType: string, inputCharset: string | null, readCb: (bufSize: number) => Buffer | null, - writeCb: (chunk: Buffer) => void + writeCb: NativeChunkCallback ): NativeStreamingOperation { return callNative(() => getAddon().runScriptTransformEngine( diff --git a/native-lib/node/src/stream.ts b/native-lib/node/src/stream.ts index 2a22bed1..60a389c3 100644 --- a/native-lib/node/src/stream.ts +++ b/native-lib/node/src/stream.ts @@ -1,12 +1,17 @@ import { parseStreamingResult } from "./result"; -import type { NativeStreamingOperation } from "./ffi"; +import type { NativeChunkCallback, NativeStreamingOperation } from "./ffi"; import type { StreamingResult } from "./types"; /** * Starts a native streaming call, wiring its chunk callback to `chunkCb` and * returning its controller once native admission succeeds. */ -export type StartStreaming = (chunkCb: (chunk: Buffer) => void) => NativeStreamingOperation; +export type StartStreaming = (chunkCb: NativeChunkCallback) => NativeStreamingOperation; + +interface NativeChunk { + readonly chunk: Buffer; + readonly sequence: bigint; +} interface InterruptibleAsyncGenerator extends AsyncGenerator { readonly parked: Promise; @@ -50,7 +55,7 @@ export function streamFromNative( onStart?: (operation: NativeStreamingOperation) => void, onClose?: (operation: NativeStreamingOperation) => void ): AsyncGenerator { - const chunks: Buffer[] = []; + const chunks: NativeChunk[] = []; const pendingResolves: Array<() => void> = []; let operation: NativeStreamingOperation | undefined; let nativeSettled = false; @@ -80,7 +85,8 @@ export function streamFromNative( const acknowledgeBufferedChunks = () => { while (chunks.length > 0) { - operation!.acknowledge(chunks.shift()!.length); + const { chunk, sequence } = chunks.shift()!; + operation!.acknowledge(sequence, chunk.length); } }; @@ -128,17 +134,17 @@ export function streamFromNative( if (lifecycleError.hasError) throw lifecycleError.error; }; - const chunkCb = (chunk: Buffer) => { + const chunkCb: NativeChunkCallback = (chunk, sequence) => { if (finalized || cancellationRequested) { try { - operation?.acknowledge(chunk.length); + operation?.acknowledge(sequence, chunk.length); } catch { // No consumer remains to observe a late callback failure. Ownership is // retained unless close succeeds, so DataWeave cleanup can still retry. } return; } - chunks.push(chunk); + chunks.push({ chunk, sequence }); pendingResolves.shift()?.(); }; @@ -165,7 +171,7 @@ export function streamFromNative( const startedOperation = nativeOperation; operation = { completion: startedOperation.completion, - acknowledge: (bytes) => startedOperation.acknowledge(bytes), + acknowledge: (sequence, bytes) => startedOperation.acknowledge(sequence, bytes), cancel, close, }; @@ -189,8 +195,8 @@ export function streamFromNative( while (true) { if (!cancellationRequested && chunks.length > 0) { - const chunk = chunks.shift()!; - operation.acknowledge(chunk.length); + const { chunk, sequence } = chunks.shift()!; + operation.acknowledge(sequence, chunk.length); yield chunk; continue; } diff --git a/native-lib/node/tests/integration/dataweave-resolver.test.ts b/native-lib/node/tests/integration/dataweave-resolver.test.ts index eca77297..ed374edb 100644 --- a/native-lib/node/tests/integration/dataweave-resolver.test.ts +++ b/native-lib/node/tests/integration/dataweave-resolver.test.ts @@ -308,12 +308,10 @@ describe('DataWeave with resolver', () => { ]) ).resolves.toBeUndefined(); - // Drain whatever remains; the stream itself must also settle, not hang. - let result = await firstNext; - while (!result.done) { - result = await gen.next(); - } - expect(result.value).toBeDefined(); + // cleanup cancels the iterator; Task 8 defines cancellation as a terminal + // undefined return rather than native metadata from the abandoned run. + await firstNext; + await expect(gen.next()).resolves.toEqual({ done: true, value: undefined }); }, 15000); // Same deadlock regression as above, for runTransform() -- the design doc @@ -338,16 +336,11 @@ describe('DataWeave with resolver', () => { const firstNext = gen.next(); - // Unlike runStreaming (whose native call is synchronous up to its first - // await), runTransform's generator body awaits createChunkReader(input) - // -- itself a microtask, not real async work for a sync-iterable input -- - // before reaching the native runScriptTransformEngine call. A single - // un-awaited .next() only advances the generator to that intermediate - // await, not past it, so the native op would not yet be dispatched - // (g_active_ops still 0) when cleanup() below fires. One extra microtask - // tick lets that internal await settle so the native call is actually - // in flight, which is what this test needs to race against. - await Promise.resolve(); + // Wait until the native transform is admitted, not merely one microtask. + // Input preparation has multiple async boundaries and cleanup before the + // admission boundary correctly invalidates the operation generation. + const firstChunk = await firstNext; + expect(firstChunk.done).toBe(false); const cleanupPromise = dw.cleanup(); @@ -358,11 +351,7 @@ describe('DataWeave with resolver', () => { ]) ).resolves.toBeUndefined(); - let result = await firstNext; - while (!result.done) { - result = await gen.next(); - } - expect(result.value).toBeDefined(); + await expect(gen.next()).resolves.toEqual({ done: true, value: undefined }); }, 15000); // Fast-path regression guard: cleanup() called once a stream has already diff --git a/native-lib/node/tests/integration/stream-backpressure.test.ts b/native-lib/node/tests/integration/stream-backpressure.test.ts index eb7fb74f..f0c84f77 100644 --- a/native-lib/node/tests/integration/stream-backpressure.test.ts +++ b/native-lib/node/tests/integration/stream-backpressure.test.ts @@ -4,9 +4,17 @@ import { buildInputsJson, findLibrary } from "../../src/utils"; interface NativeStreamingOperation { readonly completion: Promise; - acknowledge(bytes: number): void; + acknowledge(sequence: bigint, bytes: number): void; cancel(): void; close(): void; + then( + onfulfilled?: ((value: string) => TResult1 | PromiseLike) | null, + onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null + ): Promise; + catch( + onrejected?: ((reason: unknown) => TResult | PromiseLike) | null + ): Promise; + finally(onfinally?: (() => void) | null): Promise; } interface OutputFlowStats { @@ -34,7 +42,7 @@ interface TestAddon { handle: number, script: string, inputsJson: string, - chunkCb: (chunk: Buffer) => void + chunkCb: (chunk: Buffer, sequence: bigint) => void ): NativeStreamingOperation; runScriptTransformEngine( handle: number, @@ -44,11 +52,18 @@ interface TestAddon { inputMimeType: string, inputCharset: string | null, readCb: (bufSize: number) => Buffer | null, - writeCb: (chunk: Buffer) => void + writeCb: (chunk: Buffer, sequence: bigint) => void ): NativeStreamingOperation; cleanup(): Promise; __test_outputStats(operationId?: number): OutputFlowStats; __test_outputOperationId(operation: NativeStreamingOperation): number; + __test_createForeignWrappedObject(): object; + __test_failNextOutputSettlement(): void; +} + +interface PendingChunk { + readonly chunk: Buffer; + readonly sequence: bigint; } const ADDON_PATH = "../../build/Release/dwlib_addon.node"; @@ -118,7 +133,7 @@ function expectBounded(stats: OutputFlowStats): void { ); } -async function waitForPendingChunk(pending: Buffer[], label: string): Promise { +async function waitForPendingChunk(pending: PendingChunk[], label: string): Promise { const deadline = Date.now() + 10000; while (pending.length === 0) { if (Date.now() >= deadline) throw new Error(`${label} timed out`); @@ -130,7 +145,7 @@ async function waitForPendingChunk(pending: Buffer[], label: string): Promise boolean ): Promise { @@ -145,8 +160,8 @@ async function drainAfterPause( let projectedBytes = paused.outstandingBytes; let projectedChunks = paused.outstandingChunks; while (projectedBytes > paused.lowBytes || projectedChunks > paused.lowChunks) { - const chunk = await waitForPendingChunk(pending, "low-water acknowledgement"); - operation.acknowledge(chunk.length); + const { chunk, sequence } = await waitForPendingChunk(pending, "low-water acknowledgement"); + operation.acknowledge(sequence, chunk.length); projectedBytes -= chunk.length; projectedChunks--; } @@ -160,9 +175,14 @@ async function drainAfterPause( ); const deadline = Date.now() + 15000; - while (!settled() || pending.length > 0) { + while ( + !settled() || + pending.length > 0 || + addon.__test_outputStats(operationId).outstandingChunks > 0 + ) { if (pending.length > 0) { - operation.acknowledge(pending.shift()!.length); + const { chunk, sequence } = pending.shift()!; + operation.acknowledge(sequence, chunk.length); } if (Date.now() >= deadline) { throw new Error( @@ -217,18 +237,30 @@ describe.sequential("native Node output flow control", () => { ); for (const invalid of [-1, 0.5, Number.NaN, Number.POSITIVE_INFINITY, "1"]) { - expect(() => operation!.acknowledge(invalid as number)).toThrow(); + expect(() => operation!.acknowledge(1n, invalid as number)).toThrow(); + } + + for (const invalid of [ + -1, + 0, + 0.5, + Number.NaN, + Number.POSITIVE_INFINITY, + "1", + -1n, + 0n, + 1n << 64n, + ]) { + expect(() => operation!.acknowledge(invalid as bigint, 1)).toThrow(); } const raw = await withTimeout(operation.completion, "controller completion"); expect(JSON.parse(raw).success).toBe(true); - for (const chunk of chunks) operation.acknowledge(chunk.length); - expect(() => operation.cancel()).not.toThrow(); expect(() => operation.cancel()).not.toThrow(); expect(() => operation.close()).not.toThrow(); expect(() => operation.close()).not.toThrow(); - expect(() => operation.acknowledge(1)).not.toThrow(); + expect(() => operation.acknowledge(1n, 1)).not.toThrow(); } finally { if (operation !== undefined) { operation.cancel?.(); @@ -240,12 +272,41 @@ describe.sequential("native Node output flow control", () => { }); }); + it("rejects borrowed controller methods and test introspection on a foreign wrapper", async () => { + await runWithEngine(async (handle) => { + const operation = addon.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[]", + "{}", + () => {} + ); + const foreign = addon.__test_createForeignWrappedObject(); + + try { + for (const method of [operation.acknowledge, operation.cancel, operation.close]) { + expect(() => method.call(foreign, 1n, 1)).toThrow(TypeError); + } + expect(() => operation.then.call(foreign, () => {})).toThrow(TypeError); + expect(() => operation.catch.call(foreign, () => {})).toThrow(TypeError); + expect(() => operation.finally.call(foreign, () => {})).toThrow(TypeError); + expect(() => addon.__test_outputOperationId(foreign as NativeStreamingOperation)).toThrow( + TypeError + ); + + await withTimeout(operation.completion, "receiver-tag operation completion"); + } finally { + operation.cancel(); + operation.close(); + } + }); + }); + it("bounds a paused streaming producer and resumes only after low-water credit", async () => { await runWithEngine(async (handle) => { expect(typeof addon.__test_outputStats).toBe("function"); const expected = patternedBytes(); const received: Buffer[] = []; - const pending: Buffer[] = []; + const pending: PendingChunk[] = []; let operation!: NativeStreamingOperation; let operationId!: number; let completionSettled = false; @@ -254,10 +315,13 @@ describe.sequential("native Node output flow control", () => { handle, PASSTHROUGH_SCRIPT, octetStreamInputs(expected), - (chunk) => { + (chunk, sequence) => { received.push(chunk); - pending.push(chunk); - if (received.length === 1) operation.acknowledge(pending.shift()!.length); + pending.push({ chunk, sequence }); + if (received.length === 1) { + const first = pending.shift()!; + operation.acknowledge(first.sequence, first.chunk.length); + } } ); operationId = addon.__test_outputOperationId(operation); @@ -299,11 +363,89 @@ describe.sequential("native Node output flow control", () => { }); }, 30000); + it("rejects duplicate, early, out-of-order, and mismatched acknowledgements without releasing credit", async () => { + await runWithEngine(async (handle) => { + const pending: PendingChunk[] = []; + let earlyAcknowledgementRejected = false; + let operation!: NativeStreamingOperation; + operation = addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(Buffer.alloc(LARGE_SIZE, 65)), + (chunk, sequence) => { + if (pending.length === 0) { + try { + operation.acknowledge(sequence + 1n, chunk.length); + } catch { + earlyAcknowledgementRejected = true; + } + } + pending.push({ chunk, sequence }); + } + ); + const operationId = addon.__test_outputOperationId(operation); + + try { + const paused = await waitForStats( + (stats) => stats.paused && pending.length >= 2, + "equal-sized acknowledgement validation window", + operationId + ); + const first = pending[0]; + const second = pending[1]; + expect(first.chunk.length).toBe(second.chunk.length); + expect(earlyAcknowledgementRejected).toBe(true); + + const expectRejectedWithoutAccountingChange = ( + invoke: () => void, + before: OutputFlowStats + ): void => { + expect(invoke).toThrow(); + expect(addon.__test_outputStats(operationId)).toMatchObject({ + outstandingBytes: before.outstandingBytes, + outstandingChunks: before.outstandingChunks, + paused: true, + }); + }; + + expectRejectedWithoutAccountingChange( + () => operation.acknowledge(second.sequence, second.chunk.length), + paused + ); + expectRejectedWithoutAccountingChange( + () => operation.acknowledge(first.sequence + 1000000n, first.chunk.length), + paused + ); + expectRejectedWithoutAccountingChange( + () => operation.acknowledge(first.sequence, first.chunk.length - 1), + paused + ); + + operation.acknowledge(first.sequence, first.chunk.length); + pending.shift(); + const afterFirst = addon.__test_outputStats(operationId); + expect(afterFirst.outstandingChunks).toBe(paused.outstandingChunks - 1); + expectRejectedWithoutAccountingChange( + () => operation.acknowledge(first.sequence, first.chunk.length), + afterFirst + ); + + operation.cancel(); + expect(() => operation.acknowledge(first.sequence, first.chunk.length)).not.toThrow(); + await withTimeout(operation.completion, "invalid acknowledgement cleanup"); + } finally { + operation.cancel(); + operation.close(); + await withTimeout(operation.completion, "acknowledgement test cleanup"); + } + }); + }, 30000); + it("bounds transform output without changing the transform read bridge", async () => { await runWithEngine(async (handle) => { const expected = patternedBytes(); const received: Buffer[] = []; - const pending: Buffer[] = []; + const pending: PendingChunk[] = []; let offset = 0; let operation!: NativeStreamingOperation; let operationId!: number; @@ -322,10 +464,13 @@ describe.sequential("native Node output flow control", () => { offset += chunk.length; return chunk; }, - (chunk) => { + (chunk, sequence) => { received.push(chunk); - pending.push(chunk); - if (received.length === 1) operation.acknowledge(pending.shift()!.length); + pending.push({ chunk, sequence }); + if (received.length === 1) { + const first = pending.shift()!; + operation.acknowledge(first.sequence, first.chunk.length); + } } ); operationId = addon.__test_outputOperationId(operation); @@ -424,4 +569,43 @@ describe.sequential("native Node output flow control", () => { await withTimeout(anchor.cleanup(), "cleanup test anchor cleanup"); } }, 30000); + + it.each([ + { + name: "streaming", + start: (handle: number) => addon.runScriptStreamingEngine( + handle, + "%dw 2.0\noutput application/json\n---\n[]", + "{}", + () => {} + ), + }, + { + name: "transform", + start: (handle: number) => addon.runScriptTransformEngine( + handle, + "%dw 2.0\noutput application/json\n---\npayload", + "{}", + "payload", + "application/json", + "UTF-8", + () => null, + () => {} + ), + }, + ])("settles $name completion when the normal terminal resolution fails", async ({ start }) => { + await runWithEngine(async (handle) => { + addon.__test_failNextOutputSettlement(); + const operation = start(handle); + + try { + await expect( + withTimeout(operation.completion, "fault-injected terminal settlement") + ).resolves.toContain("Failed to settle native output completion"); + } finally { + operation.cancel(); + operation.close(); + } + }); + }); }); diff --git a/native-lib/node/tests/integration/teardown-deadlock.test.ts b/native-lib/node/tests/integration/teardown-deadlock.test.ts index cb828711..193651f9 100644 --- a/native-lib/node/tests/integration/teardown-deadlock.test.ts +++ b/native-lib/node/tests/integration/teardown-deadlock.test.ts @@ -98,11 +98,8 @@ describe("re-init during pending teardown (W-23692110, round 5 P1)", () => { testAddon.__test_releaseAsyncOp(); gateReleased = true; - let outerResult = await firstNext; - while (!outerResult.done) { - outerResult = await outer.next(); - } - expect(outerResult.value.success).toBe(true); + await firstNext; + await expect(outer.next()).resolves.toEqual({ done: true, value: undefined }); await cleanupPromise; } finally { if (gateArmed && !gateReleased) testAddon.__test_releaseAsyncOp(); diff --git a/native-lib/node/tests/unit/dataweave-initialize.test.ts b/native-lib/node/tests/unit/dataweave-initialize.test.ts index 21998f0e..a051d7eb 100644 --- a/native-lib/node/tests/unit/dataweave-initialize.test.ts +++ b/native-lib/node/tests/unit/dataweave-initialize.test.ts @@ -43,6 +43,13 @@ function operation(completion: Promise): NativeStreamingOperation { }; } +function sequencedCallback void>( + callback: T +): (chunk: Buffer) => void { + let sequence = 0n; + return (chunk) => callback(chunk, ++sequence); +} + const okStreamingMeta = () => JSON.stringify({ success: true, mimeType: "application/json", @@ -565,8 +572,9 @@ describe("DataWeave.initialize() native ref-count safety", () => { const completion = deferred(); const nativeOperation = operation(completion.promise); vi.mocked(ffi.runScriptStreamingEngine).mockImplementation((_handle, _script, _inputs, cb) => { - cb(Buffer.from("x")); - cb(Buffer.from("yy")); + const push = sequencedCallback(cb); + push(Buffer.from("x")); + push(Buffer.from("yy")); return nativeOperation; }); @@ -575,10 +583,10 @@ describe("DataWeave.initialize() native ref-count safety", () => { const stream = dw.runStreaming("output application/json --- [1]"); const first = await stream.next(); expect(first.value?.toString()).toBe("x"); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1n, 1); const cleanupPromise = dw.cleanup(); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 2); expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); @@ -1018,8 +1026,9 @@ describe("DataWeave.initialize() native ref-count safety", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - writeCb(Buffer.from("a")); - writeCb(Buffer.from("b")); + const push = sequencedCallback(writeCb); + push(Buffer.from("a")); + push(Buffer.from("b")); return nativeOperation; } ); @@ -1034,8 +1043,8 @@ describe("DataWeave.initialize() native ref-count safety", () => { await expect(secondPull).resolves.toEqual({ done: false, value: Buffer.from("b") }); await expect(returned).resolves.toEqual({ done: true, value: undefined }); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1n, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 1); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); await dw.cleanup(); @@ -1049,7 +1058,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { let push!: (chunk: Buffer) => void; vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - push = writeCb; + push = sequencedCallback(writeCb); return nativeOperation; } ); @@ -1072,7 +1081,11 @@ describe("DataWeave.initialize() native ref-count safety", () => { { done: false, value: Buffer.from("bb") }, { done: true, value: undefined }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[8], [1], [2]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([ + [1n, 8], + [2n, 1], + [3n, 2], + ]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); @@ -1087,7 +1100,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { let push!: (chunk: Buffer) => void; vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - push = writeCb; + push = sequencedCallback(writeCb); return nativeOperation; } ); @@ -1112,7 +1125,11 @@ describe("DataWeave.initialize() native ref-count safety", () => { { status: "fulfilled", value: { done: false, value: Buffer.from("bb") } }, { status: "rejected", reason: thrown }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[8], [1], [2]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([ + [1n, 8], + [2n, 1], + [3n, 2], + ]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); @@ -1126,10 +1143,11 @@ describe("DataWeave.initialize() native ref-count safety", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - writeCb(Buffer.from("a")); - writeCb(Buffer.from("bb")); - writeCb(Buffer.from("ccc")); - writeCb(Buffer.from("dddd")); + const push = sequencedCallback(writeCb); + push(Buffer.from("a")); + push(Buffer.from("bb")); + push(Buffer.from("ccc")); + push(Buffer.from("dddd")); return nativeOperation; } ); @@ -1153,7 +1171,12 @@ describe("DataWeave.initialize() native ref-count safety", () => { { done: true, value: undefined }, ]); expect(settlements).toEqual([1, 1, 1, 1, 1]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2], [3], [4]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([ + [1n, 1], + [2n, 2], + [3n, 3], + [4n, 4], + ]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); await dw.cleanup(); @@ -1166,8 +1189,9 @@ describe("DataWeave.initialize() native ref-count safety", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - writeCb(Buffer.from("a")); - writeCb(Buffer.from("bb")); + const push = sequencedCallback(writeCb); + push(Buffer.from("a")); + push(Buffer.from("bb")); return nativeOperation; } ); @@ -1189,7 +1213,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { { done: true, value: undefined }, { done: true, value: undefined }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1n, 1], [2n, 2]]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); @@ -1203,8 +1227,9 @@ describe("DataWeave.initialize() native ref-count safety", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - writeCb(Buffer.from("a")); - writeCb(Buffer.from("bb")); + const push = sequencedCallback(writeCb); + push(Buffer.from("a")); + push(Buffer.from("bb")); return nativeOperation; } ); @@ -1228,7 +1253,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { { status: "fulfilled", value: { done: true, value: undefined } }, { status: "rejected", reason: thrown }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[1], [2]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1n, 1], [2n, 2]]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); @@ -1262,7 +1287,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okStreamingMeta())); vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - writeCb(Buffer.from("a")); + sequencedCallback(writeCb)(Buffer.from("a")); return nativeOperation; } ); @@ -1283,7 +1308,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { { done: true, value: undefined }, { done: true, value: undefined }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[1]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1n, 1]]); expect(nativeOperation.close).toHaveBeenCalledTimes(1); await dw.cleanup(); @@ -1297,7 +1322,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { let push!: (chunk: Buffer) => void; vi.mocked(ffi.runScriptTransformEngine).mockImplementation( (_handle, _script, _inputs, _inputName, _mimeType, _charset, _readCb, writeCb) => { - push = writeCb; + push = sequencedCallback(writeCb); return nativeOperation; } ); @@ -1318,7 +1343,7 @@ describe("DataWeave.initialize() native ref-count safety", () => { { done: true, value: undefined }, { done: true, value: undefined }, ]); - expect(nativeOperation.acknowledge.mock.calls).toEqual([[1]]); + expect(nativeOperation.acknowledge.mock.calls).toEqual([[1n, 1]]); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); diff --git a/native-lib/node/tests/unit/stream.test.ts b/native-lib/node/tests/unit/stream.test.ts index 9a8ebf53..d5da6a70 100644 --- a/native-lib/node/tests/unit/stream.test.ts +++ b/native-lib/node/tests/unit/stream.test.ts @@ -35,21 +35,76 @@ function operation(completion: Promise): NativeStreamingOperation { }; } +function deliver( + callback: (chunk: Buffer, sequence: bigint) => void, + chunk: Buffer, + sequence: bigint +): void { + callback(chunk, sequence); +} + describe("streamFromNative", () => { + it("preserves opaque sequence identity until dequeue while yielding only Buffer", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + let push!: (chunk: Buffer, sequence: bigint) => void; + const gen = streamFromNative((callback) => { + push = callback as unknown as typeof push; + return nativeOperation; + }); + + const pending = gen.next(); + push(Buffer.from("same"), 9007199254740993n); + expect(nativeOperation.acknowledge).not.toHaveBeenCalled(); + + const first = await pending; + expect(first).toEqual({ done: false, value: Buffer.from("same") }); + expect(Buffer.isBuffer(first.value)).toBe(true); + expect(nativeOperation.acknowledge).toHaveBeenCalledExactlyOnceWith( + 9007199254740993n, + 4 + ); + + completion.resolve(okMeta()); + await gen.next(); + }); + + it("returns each abandoned callback's exact sequence and bytes once", async () => { + const completion = deferred(); + const nativeOperation = operation(completion.promise); + nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); + const gen = streamFromNative((callback) => { + deliver(callback, Buffer.from("x"), 41n); + deliver(callback, Buffer.from("x"), 42n); + deliver(callback, Buffer.from("yy"), 43n); + return nativeOperation; + }); + + const first = await gen.next(); + expect(first.value).toEqual(Buffer.from("x")); + await gen.return(undefined); + + expect(nativeOperation.acknowledge.mock.calls).toEqual([ + [41n, 1], + [42n, 1], + [43n, 2], + ]); + }); + it("yields chunks pushed before completion, in order", async () => { const nativeOperation = operation(Promise.resolve(okMeta())); const { chunks, result } = await collect( streamFromNative((cb) => { - cb(Buffer.from("a")); - cb(Buffer.from("b")); - cb(Buffer.from("c")); + deliver(cb, Buffer.from("a"), 1n); + deliver(cb, Buffer.from("b"), 2n); + deliver(cb, Buffer.from("c"), 3n); return nativeOperation; }) ); expect(chunks.map((c) => c.toString())).toEqual(["a", "b", "c"]); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 1); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(3, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1n, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(3, 3n, 1); expect(result.success).toBe(true); expect(result.mimeType).toBe("application/json"); expect(nativeOperation.cancel).not.toHaveBeenCalled(); @@ -85,7 +140,7 @@ describe("streamFromNative", () => { it("parks the consumer until a chunk arrives, then wakes it", async () => { const meta = deferred(); const nativeOperation = operation(meta.promise); - let push!: (chunk: Buffer) => void; + let push!: (chunk: Buffer, sequence: bigint) => void; const gen = streamFromNative((cb) => { push = cb; return nativeOperation; @@ -94,14 +149,14 @@ describe("streamFromNative", () => { // First pull starts the generator and parks — no chunk is ready yet. const pending = gen.next(); // Producing a chunk should wake the parked consumer. - push(Buffer.from("late")); + push(Buffer.from("late"), 1n); // Callback enqueue alone does not return native credit. expect(nativeOperation.acknowledge).not.toHaveBeenCalled(); const first = await pending; expect(first.done).toBe(false); expect(first.value!.toString()).toBe("late"); expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(1); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(4); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1n, 4); // Completing the stream ends the generator with the parsed metadata. meta.resolve(okMeta({ mimeType: "text/plain" })); @@ -143,21 +198,21 @@ describe("streamFromNative", () => { const { chunks, result } = await collect( streamFromNative((cb) => { // Chunks buffered but not yet consumed when the native call resolves. - cb(Buffer.from("x")); - cb(Buffer.from("yy")); + deliver(cb, Buffer.from("x"), 1n); + deliver(cb, Buffer.from("yy"), 2n); return nativeOperation; }) ); expect(chunks.map((c) => c.toString())).toEqual(["x", "yy"]); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1n, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 2); expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); expect(result.success).toBe(true); }); it("acknowledges a late callback after completion and close", async () => { const nativeOperation = operation(Promise.resolve(okMeta())); - let push!: (chunk: Buffer) => void; + let push!: (chunk: Buffer, sequence: bigint) => void; const gen = streamFromNative((cb) => { push = cb; return nativeOperation; @@ -166,9 +221,9 @@ describe("streamFromNative", () => { await collect(gen); expect(nativeOperation.close).toHaveBeenCalledTimes(1); - push(Buffer.from("late")); + push(Buffer.from("late"), 1n); expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(1); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(4); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1n, 4); }); it("propagates a failure envelope as the terminal result", async () => { @@ -210,8 +265,8 @@ describe("streamFromNative", () => { it("drains buffered chunks, then throws, when start() rejects after pushing chunks", async () => { const nativeOperation = operation(Promise.reject(new Error("late boom"))); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); - cb(Buffer.from("yy")); + deliver(cb, Buffer.from("x"), 1n); + deliver(cb, Buffer.from("yy"), 2n); return nativeOperation; }); @@ -219,8 +274,8 @@ describe("streamFromNative", () => { const a = await gen.next(); const b = await gen.next(); expect([a.value?.toString(), b.value?.toString()]).toEqual(["x", "yy"]); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(1, 1n, 1); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 2); // ...then the drained generator surfaces the start error. await expect(gen.next()).rejects.toThrow("late boom"); @@ -278,7 +333,7 @@ describe("streamFromNative", () => { const nativeOperation = operation(new Promise(() => {})); nativeOperation.cancel = vi.fn(() => { throw thrown; }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -338,17 +393,17 @@ describe("streamFromNative", () => { const completion = deferred(); const nativeOperation = operation(completion.promise); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); - cb(Buffer.from("yy")); + deliver(cb, Buffer.from("x"), 1n); + deliver(cb, Buffer.from("yy"), 2n); return nativeOperation; }); const first = await gen.next(); expect(first.value?.toString()).toBe("x"); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1n, 1); await expect(gen.return(undefined)).resolves.toEqual({ done: true, value: undefined }); - expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2); + expect(nativeOperation.acknowledge).toHaveBeenNthCalledWith(2, 2n, 2); expect(nativeOperation.acknowledge).toHaveBeenCalledTimes(2); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); @@ -359,7 +414,7 @@ describe("streamFromNative", () => { const nativeOperation = operation(completion.promise); nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -377,7 +432,7 @@ describe("streamFromNative", () => { const nativeOperation = operation(completion.promise); const gen = streamFromNative( (cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }, (managedOperation) => managedOperation.cancel() @@ -387,7 +442,7 @@ describe("streamFromNative", () => { completion.resolve(okMeta()); await expect(pending).resolves.toEqual({ done: true, value: undefined }); - expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1); + expect(nativeOperation.acknowledge).toHaveBeenCalledWith(1n, 1); expect(nativeOperation.cancel).toHaveBeenCalledTimes(1); expect(nativeOperation.close).toHaveBeenCalledTimes(1); }); @@ -396,7 +451,7 @@ describe("streamFromNative", () => { const completion = deferred(); const nativeOperation = operation(completion.promise); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -415,7 +470,7 @@ describe("streamFromNative", () => { throw new Error("cancel boom"); }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -432,7 +487,7 @@ describe("streamFromNative", () => { .mockImplementationOnce(() => { throw undefined; }) .mockImplementationOnce(() => { throw null; }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -453,7 +508,7 @@ describe("streamFromNative", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); @@ -471,8 +526,8 @@ describe("streamFromNative", () => { throw new Error("ack boom"); }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); - cb(Buffer.from("y")); + deliver(cb, Buffer.from("x"), 1n); + deliver(cb, Buffer.from("y"), 2n); return nativeOperation; }); @@ -491,7 +546,7 @@ describe("streamFromNative", () => { }); nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); let managedOperation!: NativeStreamingOperation; - let push!: (chunk: Buffer) => void; + let push!: (chunk: Buffer, sequence: bigint) => void; const gen = streamFromNative( (cb) => { push = cb; @@ -501,7 +556,7 @@ describe("streamFromNative", () => { ); const firstPull = gen.next(); await vi.waitFor(() => expect(managedOperation).toBeDefined()); - push(Buffer.from("x")); + push(Buffer.from("x"), 1n); expect(() => managedOperation.cancel()).toThrow("ack boom"); @@ -580,7 +635,7 @@ describe("streamFromNative", () => { nativeOperation.cancel = vi.fn(() => completion.resolve(okMeta())); nativeOperation.close = vi.fn(() => { throw new Error("close boom"); }); const gen = streamFromNative((cb) => { - cb(Buffer.from("x")); + deliver(cb, Buffer.from("x"), 1n); return nativeOperation; }); From 087ed5ed5a841cad9a1338cc992ce099a1d57ea4 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Wed, 9 Sep 2026 11:49:52 -0300 Subject: [PATCH 37/48] fix(node): harden output completion lifecycle --- native-lib/node/src/addon.c | 439 +++++++++++++++--- .../fixtures/output-controller-finalizer.cjs | 75 +++ .../fixtures/output-settlement-after-call.cjs | 22 + .../integration/stream-backpressure.test.ts | 354 +++++++++++++- 4 files changed, 807 insertions(+), 83 deletions(-) create mode 100644 native-lib/node/tests/integration/fixtures/output-controller-finalizer.cjs create mode 100644 native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 8943d79a..8206e419 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -199,7 +199,24 @@ static long long g_test_resolver_ref_deletes = 0; static bool g_test_hold_next_async_op = false; static bool g_test_async_op_held = false; static bool g_test_release_async_op = false; -static bool g_test_fail_next_output_settlement = false; + +typedef enum { + OUTPUT_SETTLEMENT_FAULT_NONE = 0, + OUTPUT_SETTLEMENT_FAULT_INITIAL_CREATE_GENERIC, + OUTPUT_SETTLEMENT_FAULT_INITIAL_PENDING_EXCEPTION, + OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL, + OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC, + OUTPUT_SETTLEMENT_FAULT_FALLBACK_PENDING_EXCEPTION, + OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC_AFTER_CALL, +} output_settlement_fault_t; + +static output_settlement_fault_t g_test_next_output_settlement_fault = + OUTPUT_SETTLEMENT_FAULT_NONE; +static bool g_test_hold_next_output_delivery = false; +static bool g_test_output_delivery_held = false; +static bool g_test_release_output_delivery = false; +static uint64_t g_test_held_output_sequence = 0; +static size_t g_test_held_output_bytes = 0; // One record per napi_env that has ever taken an init reference (via // initialize()). init_refs is that env's net initialize()-minus-cleanup() @@ -1226,6 +1243,8 @@ typedef struct output_flow { output_credit_t* credit_head; output_credit_t* credit_tail; napi_ref thenable_ref; + napi_ref settlement_fallback_ref; + bool settlement_started; } output_flow_t; typedef struct output_flow_stats { @@ -1295,7 +1314,7 @@ static void output_flow_retain(output_flow_t* flow) { uv_mutex_unlock(&flow->mutex); } -static void output_flow_release(output_flow_t* flow) { +static void output_flow_release(output_flow_t* flow, napi_env env) { if (flow == NULL) return; bool destroy = false; uv_mutex_lock(&flow->mutex); @@ -1305,9 +1324,13 @@ static void output_flow_release(output_flow_t* flow) { } uv_mutex_unlock(&flow->mutex); if (!destroy) return; - // A dead env auto-reclaims this N-API reference. A live env path deletes it - // in output_flow_mark_done before the final native owner is released. + // A dead env auto-reclaims N-API references. Live-env terminal paths delete + // them before releasing the final native owner. flow->thenable_ref = NULL; + if (env != NULL && flow->settlement_fallback_ref != NULL) { + napi_delete_reference(env, flow->settlement_fallback_ref); + } + flow->settlement_fallback_ref = NULL; if (g_test_hooks) { uv_mutex_lock(&g_test_output_mutex); @@ -1468,6 +1491,34 @@ static bool output_flow_mark_delivered(output_flow_t* flow, uint64_t sequence) { return delivered; } +static bool output_flow_is_cancelled(output_flow_t* flow); + +static bool test_hold_output_delivery_if_armed( + output_flow_t* flow, uint64_t sequence, size_t bytes) { + if (!g_test_hooks) return true; + bool cancelled = false; + uv_mutex_lock(&g_test_output_mutex); + if (g_test_hold_next_output_delivery) { + g_test_hold_next_output_delivery = false; + g_test_output_delivery_held = true; + g_test_held_output_sequence = sequence; + g_test_held_output_bytes = bytes; + while (!g_test_release_output_delivery) { + uv_mutex_unlock(&g_test_output_mutex); + cancelled = output_flow_is_cancelled(flow); + if (!cancelled) uv_sleep(1); + uv_mutex_lock(&g_test_output_mutex); + if (cancelled) break; + } + g_test_release_output_delivery = false; + g_test_output_delivery_held = false; + g_test_held_output_sequence = 0; + g_test_held_output_bytes = 0; + } + uv_mutex_unlock(&g_test_output_mutex); + return !cancelled && !output_flow_is_cancelled(flow); +} + // Enqueue/allocation rollback always targets the newest reservation because // callbacks reserve and enqueue serially on the sole producer worker. static void output_flow_rollback( @@ -1489,9 +1540,7 @@ static void output_flow_rollback( uv_mutex_unlock(&flow->mutex); } -static void output_flow_cancel(output_flow_t* flow) { - if (flow == NULL) return; - uv_mutex_lock(&flow->mutex); +static void output_flow_cancel_locked(output_flow_t* flow) { if (!flow->cancelled) { flow->cancelled = true; flow->paused = false; @@ -1507,6 +1556,20 @@ static void output_flow_cancel(output_flow_t* flow) { flow->credit_tail = NULL; } uv_cond_broadcast(&flow->cond); +} + +static void output_flow_cancel(output_flow_t* flow) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + output_flow_cancel_locked(flow); + output_flow_record_stats_locked(flow); + uv_mutex_unlock(&flow->mutex); +} + +static void output_flow_cancel_if_running(output_flow_t* flow) { + if (flow == NULL) return; + uv_mutex_lock(&flow->mutex); + if (!flow->done) output_flow_cancel_locked(flow); output_flow_record_stats_locked(flow); uv_mutex_unlock(&flow->mutex); } @@ -1532,6 +1595,24 @@ static void output_flow_retain_thenable( uv_mutex_unlock(&flow->mutex); } +static bool output_flow_begin_settlement(output_flow_t* flow) { + if (flow == NULL) return false; + uv_mutex_lock(&flow->mutex); + bool begin = !flow->settlement_started; + if (begin) flow->settlement_started = true; + uv_mutex_unlock(&flow->mutex); + return begin; +} + +static void output_flow_release_settlement_ref(output_flow_t* flow, napi_env env) { + if (flow == NULL || env == NULL) return; + uv_mutex_lock(&flow->mutex); + napi_ref fallback_ref = flow->settlement_fallback_ref; + flow->settlement_fallback_ref = NULL; + uv_mutex_unlock(&flow->mutex); + if (fallback_ref != NULL) napi_delete_reference(env, fallback_ref); +} + static void output_flow_mark_done(output_flow_t* flow, napi_env env) { if (flow == NULL) return; napi_ref thenable_ref = NULL; @@ -1568,11 +1649,11 @@ static void output_controller_cancel(output_controller_t* holder) { uv_mutex_unlock(&holder->mutex); if (flow != NULL) { output_flow_cancel(flow); - output_flow_release(flow); + output_flow_release(flow, NULL); } } -static void output_controller_close(output_controller_t* holder) { +static void output_controller_close(output_controller_t* holder, napi_env env) { if (holder == NULL) return; uv_mutex_lock(&holder->mutex); output_flow_t* flow = NULL; @@ -1582,16 +1663,18 @@ static void output_controller_close(output_controller_t* holder) { holder->flow = NULL; } uv_mutex_unlock(&holder->mutex); - if (flow != NULL) output_flow_release(flow); + if (flow != NULL) { + output_flow_cancel_if_running(flow); + output_flow_release(flow, env); + } } static void output_controller_finalize(napi_env env, void* data, void* hint) { - (void)env; (void)hint; output_controller_t* holder = (output_controller_t*)data; if (holder == NULL) return; output_controller_cancel(holder); - output_controller_close(holder); + output_controller_close(holder, env); uv_mutex_destroy(&holder->mutex); free(holder); } @@ -1651,7 +1734,7 @@ static napi_value napi_output_acknowledge(napi_env env, napi_callback_info info) uv_mutex_unlock(&holder->mutex); if (flow != NULL) { output_ack_result_t result = output_flow_acknowledge(flow, sequence, (size_t)value); - output_flow_release(flow); + output_flow_release(flow, NULL); switch (result) { case OUTPUT_ACK_IGNORED: case OUTPUT_ACK_ACCEPTED: @@ -1686,7 +1769,7 @@ static napi_value napi_output_cancel(napi_env env, napi_callback_info info) { static napi_value napi_output_close(napi_env env, napi_callback_info info) { output_controller_t* holder = output_controller_unwrap(env, info, 0, NULL); if (holder == NULL) return NULL; - output_controller_close(holder); + output_controller_close(holder, env); return NULL; } @@ -1728,7 +1811,7 @@ static napi_value napi_output_promise_method(napi_env env, napi_callback_info in // Retain only after the Promise method call succeeds. Failed assimilation // must not root a controller that no caller can use to close the flow. output_flow_retain_thenable(flow, env, controller); - output_flow_release(flow); + output_flow_release(flow, NULL); } return result; } @@ -1768,7 +1851,7 @@ static napi_value output_controller_create( napi_output_promise_method, (void*)"finally", &method) != napi_ok || napi_set_named_property(env, controller, "finally", method) != napi_ok) { output_controller_cancel(holder); - output_controller_close(holder); + output_controller_close(holder, env); uv_mutex_destroy(&holder->mutex); free(holder); napi_throw_error(env, NULL, "Failed to create output controller"); @@ -1776,7 +1859,7 @@ static napi_value output_controller_create( } if (napi_wrap(env, controller, holder, output_controller_finalize, NULL, NULL) != napi_ok) { output_controller_cancel(holder); - output_controller_close(holder); + output_controller_close(holder, env); uv_mutex_destroy(&holder->mutex); free(holder); napi_throw_error(env, NULL, "Failed to wrap output controller"); @@ -1786,7 +1869,7 @@ static napi_value output_controller_create( void* removed = NULL; napi_remove_wrap(env, controller, &removed); output_controller_cancel(holder); - output_controller_close(holder); + output_controller_close(holder, env); uv_mutex_destroy(&holder->mutex); free(holder); napi_throw_error(env, NULL, "Failed to tag output controller"); @@ -1804,40 +1887,161 @@ static const char OOM_JSON[] = "{\"success\":false,\"error\":\"Out of memory\"}" static const char SETTLEMENT_ERROR_JSON[] = "{\"success\":false,\"error\":\"Failed to settle native output completion\"}"; -static bool test_consume_output_settlement_fault(void) { - if (!g_test_hooks) return false; +static output_settlement_fault_t test_consume_output_settlement_fault(void) { + if (!g_test_hooks) return OUTPUT_SETTLEMENT_FAULT_NONE; uv_mutex_lock(&g_test_output_mutex); - bool fail = g_test_fail_next_output_settlement; - g_test_fail_next_output_settlement = false; + output_settlement_fault_t fault = g_test_next_output_settlement_fault; + g_test_next_output_settlement_fault = OUTPUT_SETTLEMENT_FAULT_NONE; uv_mutex_unlock(&g_test_output_mutex); - return fail; + return fault; } -static void settle_output_deferred( - napi_env env, napi_deferred deferred, const char* result_json) { - napi_value result; - napi_status status = napi_create_string_utf8( - env, result_json, strlen(result_json), &result +static bool clear_pending_exception(napi_env env) { + bool pending = false; + if (napi_is_exception_pending(env, &pending) != napi_ok || !pending) { + return false; + } + napi_value exception; + if (napi_get_and_clear_last_exception(env, &exception) != napi_ok) { + fprintf(stderr, + "[DataWeave Node addon] Failed to clear an output settlement exception.\n"); + return false; + } + return true; +} + +static napi_status settle_output_fallback( + napi_env env, napi_deferred deferred, napi_ref fallback_ref, + bool* conclude_called) { + *conclude_called = false; + napi_value holder; + napi_status status = napi_get_reference_value(env, fallback_ref, &holder); + if (status != napi_ok) return status; + napi_value fallback; + status = napi_get_named_property(env, holder, "value", &fallback); + if (status != napi_ok) return status; + *conclude_called = true; + return napi_resolve_deferred(env, deferred, fallback); +} + +static void output_settlement_fail_closed(napi_status status) { + // Node frees a deferred whenever napi_resolve_deferred/reject_deferred is + // invoked, including failure returns. Continuing would either reuse freed + // memory or leave callers waiting forever, so terminate deterministically. + char message[128]; + int length = snprintf( + message, sizeof(message), + "Output completion settlement failed after deferred consumption (napi status %d)", + (int)status + ); + size_t message_length = length > 0 + ? ((size_t)length < sizeof(message) ? (size_t)length : sizeof(message) - 1) + : 0; + static const char location[] = "DataWeave Node addon"; + napi_fatal_error( + location, sizeof(location) - 1, message, message_length ); +} + +static napi_status settle_output_deferred( + napi_env env, napi_deferred deferred, output_flow_t* flow, + const char* result_json) { + if (env == NULL || flow == NULL || !output_flow_begin_settlement(flow)) { + return napi_ok; + } + + output_settlement_fault_t fault = test_consume_output_settlement_fault(); + napi_value result; + bool fail_initial = + fault == OUTPUT_SETTLEMENT_FAULT_INITIAL_CREATE_GENERIC || + fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC || + fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_PENDING_EXCEPTION || + fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC_AFTER_CALL; + napi_status status = fail_initial + ? napi_generic_failure + : napi_create_string_utf8(env, result_json, strlen(result_json), &result); if (status == napi_ok) { - status = test_consume_output_settlement_fault() - ? napi_generic_failure - : napi_resolve_deferred(env, deferred, result); + if (fault == OUTPUT_SETTLEMENT_FAULT_INITIAL_PENDING_EXCEPTION) { + napi_throw_error(env, NULL, "Injected initial output settlement exception"); + status = napi_pending_exception; + } else { + status = napi_resolve_deferred(env, deferred, result); + if (status == napi_ok && + fault == OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL) { + status = napi_generic_failure; + } + if (status != napi_ok) { + if (status == napi_pending_exception && + !clear_pending_exception(env)) { + output_flow_release_settlement_ref(flow, env); + return status; + } + output_settlement_fail_closed(status); + } + output_flow_release_settlement_ref(flow, env); + return napi_ok; + } + } + if (status == napi_pending_exception && !clear_pending_exception(env)) { + output_flow_release_settlement_ref(flow, env); + return status; } - if (status == napi_ok) return; + bool conclude_called = false; + if (fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_PENDING_EXCEPTION) { + napi_throw_error(env, NULL, "Injected fallback output settlement exception"); + status = napi_pending_exception; + } else if (fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC) { + status = napi_generic_failure; + } else { + status = settle_output_fallback( + env, deferred, flow->settlement_fallback_ref, &conclude_called + ); + } + if (status == napi_ok && + fault == OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC_AFTER_CALL) { + status = napi_generic_failure; + } + if (status == napi_ok) { + output_flow_release_settlement_ref(flow, env); + return napi_ok; + } + if (status == napi_pending_exception && !clear_pending_exception(env)) { + output_flow_release_settlement_ref(flow, env); + return status; + } + if (conclude_called) output_settlement_fail_closed(status); + + status = settle_output_fallback( + env, deferred, flow->settlement_fallback_ref, &conclude_called + ); + if (status != napi_ok) { + if (status == napi_pending_exception && !conclude_called) { + if (!clear_pending_exception(env)) { + output_flow_release_settlement_ref(flow, env); + return status; + } + } + output_settlement_fail_closed(status); + } + output_flow_release_settlement_ref(flow, env); + return napi_ok; +} + +static bool prepare_output_settlement(napi_env env, output_flow_t* flow) { napi_value fallback; if (napi_create_string_utf8( - env, SETTLEMENT_ERROR_JSON, NAPI_AUTO_LENGTH, &fallback) == napi_ok) { - if (napi_resolve_deferred(env, deferred, fallback) == napi_ok) return; + env, SETTLEMENT_ERROR_JSON, NAPI_AUTO_LENGTH, &fallback) != napi_ok) { + return false; } - - // napi_get_undefined does not allocate. If string creation itself failed - // under memory pressure, still make a final allocation-free settlement - // attempt rather than leaving the operation permanently pending. - if (napi_get_undefined(env, &fallback) == napi_ok) { - napi_resolve_deferred(env, deferred, fallback); + // N-API v8 cannot retain a primitive string directly, so keep it reachable + // through a referenced object before the asynchronous operation starts. + napi_value holder; + if (napi_create_object(env, &holder) != napi_ok || + napi_set_named_property(env, holder, "value", fallback) != napi_ok) { + return false; } + return napi_create_reference(env, holder, 1, &flow->settlement_fallback_ref) == napi_ok; } // chunk_data with len == -1 is a sentinel indicating completion (buf holds meta JSON) @@ -1877,7 +2081,7 @@ static void output_chunk_release(struct chunk_data* chunk, bool rollback) { if (rollback) { output_flow_rollback(chunk->flow, chunk->sequence, chunk->accounted_bytes); } - output_flow_release(chunk->flow); + output_flow_release(chunk->flow, NULL); } free(chunk->buf); free(chunk); @@ -1899,7 +2103,7 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // would leak `w` and could strand a bridge marked for deferred destruction // indefinitely. if (env != NULL) { - settle_output_deferred(env, w->deferred, chunk->buf); + settle_output_deferred(env, w->deferred, w->flow, chunk->buf); } output_flow_mark_done(w->flow, env); @@ -1916,7 +2120,7 @@ static void call_js_write(napi_env env, napi_value js_callback, void* context, v // dead/tearing down -- tell bridge_end_op (and any bridge_finalize it // triggers) not to touch the napi_ref, since b->env is this same dead env. bridge_end_op(w->bridge, /*env_still_alive=*/env != NULL); - output_flow_release(w->flow); + output_flow_release(w->flow, NULL); free(w); return; } @@ -1974,6 +2178,9 @@ static int streaming_write_cb(void* ctx, const char* buf, int len) { if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; uint64_t sequence; if (!output_flow_reserve(w->flow, (size_t)len, &sequence)) return -1; + if (!test_hold_output_delivery_if_armed(w->flow, sequence, (size_t)len)) { + return -1; + } // Round-9 (#2): OOM here must not deref NULL / memcpy into NULL. Returning -1 // aborts the native run cleanly (write-callback contract: non-zero stops the // DataWeave run); the worker then still produces a terminal meta_result and @@ -2112,7 +2319,7 @@ static void streaming_thread_fn(void* arg) { free(w->inputs_json); bridge_end_op(w->bridge, /*env_still_alive=*/false); output_flow_mark_done(w->flow, NULL); - output_flow_release(w->flow); + output_flow_release(w->flow, NULL); free(w); } } @@ -2229,6 +2436,14 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i napi_throw_error(env, NULL, "OOM"); return NULL; } + if (!prepare_output_settlement(env, w->flow)) { + output_flow_release(w->flow, NULL); + free(w->script); free(w->inputs_json); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "runScriptStreamingEngine: failed to create settlement fallback"); + return NULL; + } // Round-9 (#3, updated round-11 #2): the resource creations below run AFTER // g_active_ops was reserved (and after w + its buffers were allocated), and @@ -2240,7 +2455,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // (teardown wedge). napi_value resource_name; if (napi_create_string_utf8(env, "dwStreaming", NAPI_AUTO_LENGTH, &resource_name) != napi_ok) { - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2250,7 +2465,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i if (napi_create_threadsafe_function(env, argv[3], NULL, resource_name, OUTPUT_TSFN_QUEUE_SIZE, 1, NULL, NULL, w, call_js_write, &w->tsfn) != napi_ok) { - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2268,7 +2483,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i w->sentinel = malloc(sizeof(struct chunk_data)); if (w->sentinel == NULL) { napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2281,7 +2496,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // The tsfn was created above; release it before freeing w (it holds w as // its context). No worker exists yet, so this release is the sole discharge. napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->sentinel); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2292,7 +2507,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i napi_value controller = output_controller_create(env, promise, w->flow); if (controller == NULL) { napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->sentinel); free(w->script); free(w->inputs_json); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2326,7 +2541,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i napi_release_threadsafe_function(w->tsfn, napi_tsfn_release); settle_output_deferred( - env, w->deferred, + env, w->deferred, w->flow, "{\"success\":false,\"error\":\"Failed to spawn streaming worker thread\"}" ); @@ -2334,7 +2549,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i free(w->script); free(w->inputs_json); output_flow_mark_done(w->flow, env); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w); } @@ -2494,6 +2709,9 @@ static int transform_write_cb(void* ctx, const char* buf, int len) { if (len < 0 || output_flow_is_cancelled(w->flow)) return -1; uint64_t sequence; if (!output_flow_reserve(w->flow, (size_t)len, &sequence)) return -1; + if (!test_hold_output_delivery_if_armed(w->flow, sequence, (size_t)len)) { + return -1; + } // Round-9 (#2): OOM-safe, mirrors streaming_write_cb. Return -1 to abort the // native run cleanly; the worker still delivers a terminal sentinel. struct chunk_data* chunk = malloc(sizeof(struct chunk_data)); @@ -2543,7 +2761,7 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // would leak `w` and could strand a bridge marked for deferred destruction // indefinitely. if (env != NULL) { - settle_output_deferred(env, w->deferred, chunk->buf); + settle_output_deferred(env, w->deferred, w->flow, chunk->buf); } output_flow_mark_done(w->flow, env); @@ -2564,7 +2782,7 @@ static void call_js_transform_write(napi_env env, napi_value js_callback, void* // dead/tearing down -- tell bridge_end_op (and any bridge_finalize it // triggers) not to touch the napi_ref, since b->env is this same dead env. bridge_end_op(w->bridge, /*env_still_alive=*/env != NULL); - output_flow_release(w->flow); + output_flow_release(w->flow, NULL); free(w); return; } @@ -2719,7 +2937,7 @@ static void transform_thread_fn(void* arg) { free(w->input_charset); bridge_end_op(w->bridge, /*env_still_alive=*/false); output_flow_mark_done(w->flow, NULL); - output_flow_release(w->flow); + output_flow_release(w->flow, NULL); free(w); } } @@ -2855,6 +3073,15 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i napi_throw_error(env, NULL, "OOM"); return NULL; } + if (!prepare_output_settlement(env, w->flow)) { + output_flow_release(w->flow, env); + free(w->script); free(w->inputs_json); free(w->input_name); + free(w->input_mime_type); free(w->input_charset); free(w); + bridge_end_op(pinned, /*env_still_alive=*/true); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "runScriptTransformEngine: failed to create settlement fallback"); + return NULL; + } // Round-9 (#3, updated round-11 #2): check each resource creation; on // failure release the engine pin (`pinned`, taken at admission) via @@ -2864,7 +3091,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i // before freeing w if it was created. napi_value resource_name; if (napi_create_string_utf8(env, "dwTransform", NAPI_AUTO_LENGTH, &resource_name) != napi_ok) { - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2873,7 +3100,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i } if (napi_create_threadsafe_function(env, argv[6], NULL, resource_name, 0, 1, NULL, NULL, NULL, call_js_read, &w->read_tsfn) != napi_ok) { - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2884,7 +3111,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i OUTPUT_TSFN_QUEUE_SIZE, 1, NULL, NULL, w, call_js_transform_write, &w->write_tsfn) != napi_ok) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2903,7 +3130,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i if (w->sentinel == NULL) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2915,7 +3142,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i if (napi_create_promise(env, &w->deferred, &promise) != napi_ok) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->sentinel); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -2927,7 +3154,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i if (controller == NULL) { napi_release_threadsafe_function(w->read_tsfn, napi_tsfn_release); napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); - output_flow_release(w->flow); + output_flow_release(w->flow, env); free(w->sentinel); free(w->script); free(w->inputs_json); free(w->input_name); free(w->input_mime_type); free(w->input_charset); free(w); bridge_end_op(pinned, /*env_still_alive=*/true); @@ -2964,7 +3191,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i napi_release_threadsafe_function(w->write_tsfn, napi_tsfn_release); settle_output_deferred( - env, w->deferred, + env, w->deferred, w->flow, "{\"success\":false,\"error\":\"Failed to spawn transform worker thread\"}" ); @@ -2975,7 +3202,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i free(w->input_mime_type); free(w->input_charset); output_flow_mark_done(w->flow, NULL); - output_flow_release(w->flow); + output_flow_release(w->flow, NULL); free(w); } @@ -4372,10 +4599,90 @@ static napi_value napi_test_create_foreign_wrapped_object( static napi_value napi_test_fail_next_output_settlement( napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + size_t length; + char stage[64]; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || + napi_get_value_string_utf8(env, argv[0], stage, sizeof(stage), &length) != napi_ok) { + napi_throw_type_error(env, NULL, "A settlement fault stage is required"); + return NULL; + } + output_settlement_fault_t fault = OUTPUT_SETTLEMENT_FAULT_NONE; + if (strcmp(stage, "initial-create-generic") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_CREATE_GENERIC; + } else if (strcmp(stage, "initial-pending-exception") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_PENDING_EXCEPTION; + } else if (strcmp(stage, "initial-call-generic-after-call") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL; + } else if (strcmp(stage, "fallback-call-generic") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC; + } else if (strcmp(stage, "fallback-pending-exception") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_FALLBACK_PENDING_EXCEPTION; + } else if (strcmp(stage, "fallback-call-generic-after-call") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC_AFTER_CALL; + } else { + napi_throw_range_error(env, NULL, "Unknown settlement fault stage"); + return NULL; + } + uv_mutex_lock(&g_test_output_mutex); + if (g_test_next_output_settlement_fault != OUTPUT_SETTLEMENT_FAULT_NONE) { + uv_mutex_unlock(&g_test_output_mutex); + napi_throw_error(env, NULL, "An output settlement fault is already armed"); + return NULL; + } + g_test_next_output_settlement_fault = fault; + uv_mutex_unlock(&g_test_output_mutex); + return NULL; +} + +static napi_value napi_test_hold_next_output_delivery( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_test_output_mutex); + if (g_test_hold_next_output_delivery || g_test_output_delivery_held) { + uv_mutex_unlock(&g_test_output_mutex); + napi_throw_error(env, NULL, "An output delivery gate is already armed"); + return NULL; + } + g_test_hold_next_output_delivery = true; + g_test_release_output_delivery = false; + uv_mutex_unlock(&g_test_output_mutex); + return NULL; +} + +static napi_value napi_test_held_output_delivery( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_test_output_mutex); + bool held = g_test_output_delivery_held; + uint64_t sequence = g_test_held_output_sequence; + size_t bytes = g_test_held_output_bytes; + uv_mutex_unlock(&g_test_output_mutex); + + napi_value out; + napi_value value; + napi_create_object(env, &out); + napi_get_boolean(env, held, &value); + napi_set_named_property(env, out, "held", value); + napi_create_bigint_uint64(env, sequence, &value); + napi_set_named_property(env, out, "sequence", value); + napi_create_double(env, (double)bytes, &value); + napi_set_named_property(env, out, "bytes", value); + return out; +} + +static napi_value napi_test_release_output_delivery( + napi_env env, napi_callback_info info) { (void)env; (void)info; uv_mutex_lock(&g_test_output_mutex); - g_test_fail_next_output_settlement = true; + g_test_release_output_delivery = true; + if (!g_test_output_delivery_held) { + g_test_hold_next_output_delivery = false; + g_test_held_output_sequence = 0; + g_test_held_output_bytes = 0; + } uv_mutex_unlock(&g_test_output_mutex); return NULL; } @@ -4528,6 +4835,12 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_createForeignWrappedObject", fn); napi_create_function(env, "__test_failNextOutputSettlement", NAPI_AUTO_LENGTH, napi_test_fail_next_output_settlement, NULL, &fn); napi_set_named_property(env, exports, "__test_failNextOutputSettlement", fn); + napi_create_function(env, "__test_holdNextOutputDelivery", NAPI_AUTO_LENGTH, napi_test_hold_next_output_delivery, NULL, &fn); + napi_set_named_property(env, exports, "__test_holdNextOutputDelivery", fn); + napi_create_function(env, "__test_heldOutputDelivery", NAPI_AUTO_LENGTH, napi_test_held_output_delivery, NULL, &fn); + napi_set_named_property(env, exports, "__test_heldOutputDelivery", fn); + napi_create_function(env, "__test_releaseOutputDelivery", NAPI_AUTO_LENGTH, napi_test_release_output_delivery, NULL, &fn); + napi_set_named_property(env, exports, "__test_releaseOutputDelivery", fn); } return exports; diff --git a/native-lib/node/tests/integration/fixtures/output-controller-finalizer.cjs b/native-lib/node/tests/integration/fixtures/output-controller-finalizer.cjs new file mode 100644 index 00000000..62478eca --- /dev/null +++ b/native-lib/node/tests/integration/fixtures/output-controller-finalizer.cjs @@ -0,0 +1,75 @@ +"use strict"; + +const path = require("node:path"); + +const addonPath = path.resolve(process.cwd(), process.argv[2]); +const libPath = process.argv[3]; +const addon = require(addonPath); + +async function timeout(promise, label) { + let timer; + try { + return await Promise.race([ + promise, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${label} timed out`)), 10000); + }), + ]); + } finally { + clearTimeout(timer); + } +} + +const immediate = () => new Promise((resolve) => setImmediate(resolve)); + +async function main() { + if (typeof global.gc !== "function") throw new Error("global.gc is unavailable"); + addon.initialize(libPath); + const handle = addon.createEngine(); + + const completion = (() => { + const payload = Buffer.alloc(2 * 1024 * 1024 + 32771, 65).toString("base64"); + const operation = addon.runScriptStreamingEngine( + handle, + "output application/octet-stream deferred=true\n---\npayload", + JSON.stringify({ + payload: { content: payload, mimeType: "application/octet-stream" }, + }), + () => {} + ); + return operation.completion; + })(); + + const deadline = Date.now() + 10000; + while (addon.__test_outputStats().liveFlows !== 0) { + global.gc(); + await immediate(); + if (Date.now() >= deadline) throw new Error("controller finalizer timed out"); + } + + await timeout(completion, "finalized controller completion"); + const completionSettled = true; + addon.destroyEngine(handle); + await timeout(addon.cleanup(), "finalized controller cleanup"); + + addon.initialize(libPath); + const reuseHandle = addon.createEngine(); + const reused = JSON.parse(addon.runScriptEngine( + reuseHandle, + "output application/json --- 6 * 7", + "{}" + )); + addon.destroyEngine(reuseHandle); + await timeout(addon.cleanup(), "reuse cleanup"); + + process.stdout.write(JSON.stringify({ + completionSettled, + liveFlows: addon.__test_outputStats().liveFlows, + reusedResult: Buffer.from(reused.result, "base64").toString("utf-8"), + })); +} + +main().catch((error) => { + process.stderr.write(`${error.stack ?? error}\n`); + process.exitCode = 1; +}); diff --git a/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs b/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs new file mode 100644 index 00000000..43a365f0 --- /dev/null +++ b/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs @@ -0,0 +1,22 @@ +"use strict"; + +const path = require("node:path"); + +const addonPath = path.resolve(process.cwd(), process.argv[2]); +const libPath = process.argv[3]; +const fault = process.argv[4]; +const addon = require(addonPath); + +function main() { + addon.initialize(libPath); + const handle = addon.createEngine(); + addon.__test_failNextOutputSettlement(fault); + addon.runScriptStreamingEngine( + handle, + "output application/json\n---\n[]", + "{}", + () => {} + ); +} + +main(); diff --git a/native-lib/node/tests/integration/stream-backpressure.test.ts b/native-lib/node/tests/integration/stream-backpressure.test.ts index f0c84f77..f4b0b096 100644 --- a/native-lib/node/tests/integration/stream-backpressure.test.ts +++ b/native-lib/node/tests/integration/stream-backpressure.test.ts @@ -1,4 +1,6 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { spawnSync } from "node:child_process"; +import { join } from "node:path"; import { DataWeave } from "../../src/dataweave"; import { buildInputsJson, findLibrary } from "../../src/utils"; @@ -34,10 +36,19 @@ interface OutputFlowStats { liveFlows: number; } +type OutputSettlementFault = + | "initial-create-generic" + | "initial-pending-exception" + | "initial-call-generic-after-call" + | "fallback-call-generic" + | "fallback-pending-exception" + | "fallback-call-generic-after-call"; + interface TestAddon { initialize(libPath: string): void; createEngine(): number; destroyEngine(handle: number): void; + runScriptEngine(handle: number, script: string, inputsJson: string): string; runScriptStreamingEngine( handle: number, script: string, @@ -58,7 +69,13 @@ interface TestAddon { __test_outputStats(operationId?: number): OutputFlowStats; __test_outputOperationId(operation: NativeStreamingOperation): number; __test_createForeignWrappedObject(): object; - __test_failNextOutputSettlement(): void; + __test_failNextOutputSettlement(stage: OutputSettlementFault): void; + __test_holdNextOutputDelivery(): void; + __test_heldOutputDelivery(): { held: boolean; sequence: bigint; bytes: number }; + __test_releaseOutputDelivery(): void; + __test_holdNextAsyncOp(): void; + __test_asyncOpHeld(): boolean; + __test_releaseAsyncOp(): void; } interface PendingChunk { @@ -217,14 +234,14 @@ afterAll(async () => { describe.sequential("native Node output flow control", () => { it("returns a validated, idempotent operation controller", async () => { await runWithEngine(async (handle) => { - const chunks: Buffer[] = []; + const chunks: PendingChunk[] = []; let operation: NativeStreamingOperation | undefined; try { operation = addon.runScriptStreamingEngine( handle, "output application/json deferred=true --- [1, 2, 3]", "{}", - (chunk) => chunks.push(chunk) + (chunk, sequence) => chunks.push({ chunk, sequence }) ); expect(operation).toEqual( @@ -256,6 +273,9 @@ describe.sequential("native Node output flow control", () => { const raw = await withTimeout(operation.completion, "controller completion"); expect(JSON.parse(raw).success).toBe(true); + for (const { chunk, sequence } of chunks) { + operation.acknowledge(sequence, chunk.length); + } expect(() => operation.cancel()).not.toThrow(); expect(() => operation.cancel()).not.toThrow(); expect(() => operation.close()).not.toThrow(); @@ -283,15 +303,17 @@ describe.sequential("native Node output flow control", () => { const foreign = addon.__test_createForeignWrappedObject(); try { - for (const method of [operation.acknowledge, operation.cancel, operation.close]) { - expect(() => method.call(foreign, 1n, 1)).toThrow(TypeError); + for (const receiver of [foreign, {}]) { + for (const method of [operation.acknowledge, operation.cancel, operation.close]) { + expect(() => method.call(receiver, 1n, 1)).toThrow(TypeError); + } + expect(() => operation.then.call(receiver, () => {})).toThrow(TypeError); + expect(() => operation.catch.call(receiver, () => {})).toThrow(TypeError); + expect(() => operation.finally.call(receiver, () => {})).toThrow(TypeError); + expect(() => addon.__test_outputOperationId(receiver as NativeStreamingOperation)).toThrow( + TypeError + ); } - expect(() => operation.then.call(foreign, () => {})).toThrow(TypeError); - expect(() => operation.catch.call(foreign, () => {})).toThrow(TypeError); - expect(() => operation.finally.call(foreign, () => {})).toThrow(TypeError); - expect(() => addon.__test_outputOperationId(foreign as NativeStreamingOperation)).toThrow( - TypeError - ); await withTimeout(operation.completion, "receiver-tag operation completion"); } finally { @@ -441,6 +463,214 @@ describe.sequential("native Node output flow control", () => { }); }, 30000); + it.each([ + { + name: "streaming", + expected: Buffer.from("reserved streaming output"), + start: ( + handle: number, + received: Buffer[], + acknowledge: (chunk: Buffer, sequence: bigint) => void + ) => addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(Buffer.from("reserved streaming output")), + (chunk, sequence) => { + received.push(chunk); + acknowledge(chunk, sequence); + } + ), + }, + { + name: "transform", + expected: Buffer.from("reserved transform output"), + start: ( + handle: number, + received: Buffer[], + acknowledge: (chunk: Buffer, sequence: bigint) => void + ) => { + const input = Buffer.from("reserved transform output"); + let offset = 0; + return addon.runScriptTransformEngine( + handle, + PASSTHROUGH_SCRIPT, + "{}", + "payload", + "application/octet-stream", + null, + (bufSize) => { + if (offset >= input.length) return null; + const chunk = input.subarray(offset, Math.min(offset + bufSize, input.length)); + offset += chunk.length; + return chunk; + }, + (chunk, sequence) => { + received.push(chunk); + acknowledge(chunk, sequence); + } + ); + }, + }, + ])("rejects a reserved-but-undelivered $name acknowledgement without changing accounting", async ({ expected, start }) => { + const received: Buffer[] = []; + await runWithEngine(async (handle) => { + addon.__test_holdNextOutputDelivery(); + let operation!: NativeStreamingOperation; + operation = start(handle, received, (chunk, sequence) => { + operation.acknowledge(sequence, chunk.length); + }); + const operationId = addon.__test_outputOperationId(operation); + + try { + const held = await waitForOutputDeliveryBarrier("reserved output delivery"); + const before = addon.__test_outputStats(operationId); + expect(held).toMatchObject({ held: true, sequence: 1n }); + expect(before).toMatchObject({ + outstandingBytes: held.bytes, + outstandingChunks: 1, + }); + + expect(() => operation.acknowledge(held.sequence, held.bytes)).toThrow( + "has not been delivered" + ); + expect(addon.__test_outputStats(operationId)).toEqual(before); + + addon.__test_releaseOutputDelivery(); + const raw = await withTimeout(operation.completion, "released output completion"); + expect(Buffer.concat(received)).toEqual(expected); + expect(JSON.parse(raw)).toMatchObject({ success: true }); + expect(addon.__test_outputStats(operationId)).toMatchObject({ + outstandingBytes: 0, + outstandingChunks: 0, + done: true, + }); + } finally { + addon.__test_releaseOutputDelivery(); + operation?.cancel(); + operation?.close(); + } + }); + }); + + it.each([ + { + name: "streaming", + start: (handle: number, callback: (chunk: Buffer, sequence: bigint) => void) => + addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(patternedBytes()), + callback + ), + }, + { + name: "transform", + start: (handle: number, callback: (chunk: Buffer, sequence: bigint) => void) => { + const input = patternedBytes(); + let offset = 0; + return addon.runScriptTransformEngine( + handle, + PASSTHROUGH_SCRIPT, + "{}", + "payload", + "application/octet-stream", + null, + (bufSize) => { + if (offset >= input.length) return null; + const chunk = input.subarray(offset, Math.min(offset + bufSize, input.length)); + offset += chunk.length; + return chunk; + }, + callback + ); + }, + }, + ])("cancels and settles raw $name output when its callback throws", async ({ start }) => { + await runWithEngine(async (handle) => { + const operation = start(handle, () => { throw new Error("output callback boom"); }); + const operationId = addon.__test_outputOperationId(operation); + + try { + const raw = await withTimeout(operation.completion, "throwing callback completion"); + expect(JSON.parse(raw)).toMatchObject({ success: false }); + await waitForStats( + (stats) => stats.cancelled && stats.done && stats.outstandingChunks === 0, + "throwing callback cancellation", + operationId + ); + } finally { + operation.close(); + } + + await waitForStats((stats) => stats.liveFlows === 0, "throwing callback flow release"); + expect(JSON.parse(addon.runScriptEngine(handle, "output application/json --- 6 * 7", "{}"))) + .toMatchObject({ success: true }); + }); + }); + + it("settles without leaking when JS ownership closes while a producer is paused", async () => { + await runWithEngine(async (handle) => { + const operation = addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(patternedBytes()), + () => {} + ); + const operationId = addon.__test_outputOperationId(operation); + + await waitForStats((stats) => stats.paused, "close-before-completion pause", operationId); + operation.close(); + operation.cancel(); + await withTimeout(operation.completion, "close-before-completion settlement"); + await waitForStats( + (stats) => stats.cancelled && stats.done && stats.liveFlows === 0, + "close-before-completion flow release", + operationId + ); + }); + }); + + it("settles without leaking when close cancels a paused producer by itself", async () => { + await runWithEngine(async (handle) => { + const operation = addon.runScriptStreamingEngine( + handle, + PASSTHROUGH_SCRIPT, + octetStreamInputs(patternedBytes()), + () => {} + ); + const operationId = addon.__test_outputOperationId(operation); + + await waitForStats((stats) => stats.paused, "close-only pause", operationId); + operation.close(); + await withTimeout(operation.completion, "close-only settlement"); + await waitForStats( + (stats) => stats.cancelled && stats.done && stats.liveFlows === 0, + "close-only flow release", + operationId + ); + }); + }); + + it("runs the exact controller finalizer path under exposed GC", () => { + const fixture = join(__dirname, "fixtures", "output-controller-finalizer.cjs"); + const addonPath = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); + const child = spawnSync(process.execPath, ["--expose-gc", fixture, addonPath, LIB_PATH], { + cwd: __dirname, + encoding: "utf-8", + timeout: 30000, + env: { ...process.env, DATAWEAVE_TEST_HOOKS: "1" }, + }); + + expect(child.error, child.error?.message).toBeUndefined(); + expect(child.signal, child.stderr).toBeNull(); + expect(child.status, child.stderr).toBe(0); + expect(JSON.parse(child.stdout.trim())).toEqual({ + completionSettled: true, + liveFlows: 0, + reusedResult: "42", + }); + }); + it("bounds transform output without changing the transform read bridge", async () => { await runWithEngine(async (handle) => { const expected = patternedBytes(); @@ -579,6 +809,9 @@ describe.sequential("native Node output flow control", () => { "{}", () => {} ), + startPublic: (dw: DataWeave) => dw.runStreaming( + "%dw 2.0\noutput application/json\n---\n[]" + ), }, { name: "transform", @@ -592,20 +825,101 @@ describe.sequential("native Node output flow control", () => { () => null, () => {} ), + startPublic: (dw: DataWeave) => dw.runTransform( + "%dw 2.0\noutput application/json\n---\npayload", + [Buffer.from("[]")] + ), }, - ])("settles $name completion when the normal terminal resolution fails", async ({ start }) => { - await runWithEngine(async (handle) => { - addon.__test_failNextOutputSettlement(); - const operation = start(handle); + ])("settles $name completion and cleanup for every recoverable terminal N-API fault", async ({ start, startPublic }) => { + for (const fault of [ + "initial-create-generic", + "initial-pending-exception", + "fallback-call-generic", + "fallback-pending-exception", + ] as const) { + await runWithEngine(async (handle) => { + addon.__test_failNextOutputSettlement(fault); + const operation = start(handle); + const operationId = addon.__test_outputOperationId(operation); + + try { + const raw = await withTimeout( + operation.completion, + `${fault} terminal settlement` + ); + expect(JSON.parse(raw)).toMatchObject({ success: false }); + } finally { + operation.cancel(); + operation.close(); + } + + await waitForStats( + (stats) => stats.done && stats.liveFlows === 0, + `${fault} terminal flow completion`, + operationId + ); + }); + const dw = new DataWeave(); + dw.initialize(); + addon.__test_holdNextAsyncOp(); + addon.__test_failNextOutputSettlement(fault); + const stream = startPublic(dw); + const firstPull = stream.next(); try { - await expect( - withTimeout(operation.completion, "fault-injected terminal settlement") - ).resolves.toContain("Failed to settle native output completion"); + await waitForAsyncOpHeld(`${fault} public operation admission`); + const cleanup = dw.cleanup(); + addon.__test_releaseAsyncOp(); + await withTimeout(cleanup, `${fault} DataWeave cleanup`); + await firstPull; + await waitForStats((stats) => stats.liveFlows === 0, `${fault} public flow release`); + dw.initialize(); + expect(dw.run("output application/json --- 6 * 7").getString()).toBe("42"); } finally { - operation.cancel(); - operation.close(); + addon.__test_releaseAsyncOp(); + await withTimeout(dw.cleanup(), `${fault} final cleanup`); } + } + }, 30000); + + it.each([ + "initial-call-generic-after-call", + "fallback-call-generic-after-call", + ] as const)("fails closed instead of reusing a consumed deferred after %s", (fault) => { + const fixture = join(__dirname, "fixtures", "output-settlement-after-call.cjs"); + const addonPath = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); + const child = spawnSync(process.execPath, [fixture, addonPath, LIB_PATH, fault], { + cwd: __dirname, + encoding: "utf-8", + timeout: 30000, + env: { ...process.env, DATAWEAVE_TEST_HOOKS: "1" }, }); + + expect(child.error, child.error?.message).toBeUndefined(); + expect(child.status === 0 && child.signal === null, child.stderr).toBe(false); + expect(child.signal).not.toBe("SIGSEGV"); + expect(child.stderr).toContain("Output completion settlement failed after deferred consumption"); }); }); + +async function waitForOutputDeliveryBarrier(label: string): Promise<{ + held: boolean; + sequence: bigint; + bytes: number; +}> { + const deadline = Date.now() + 10000; + while (true) { + const held = addon.__test_heldOutputDelivery(); + if (held.held) return held; + if (Date.now() >= deadline) throw new Error(`${label} timed out`); + await immediate(); + } +} + +async function waitForAsyncOpHeld(label: string): Promise { + const deadline = Date.now() + 10000; + while (!addon.__test_asyncOpHeld()) { + if (Date.now() >= deadline) throw new Error(`${label} timed out`); + await immediate(); + } +} From 87ed27370fa6bcf4dc3aaac6a2fce381485eb0d4 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Wed, 9 Sep 2026 12:06:33 -0300 Subject: [PATCH 38/48] fix(node): fail closed on settlement clear errors --- native-lib/node/src/addon.c | 125 +++++++++++++++--- .../fixtures/output-settlement-after-call.cjs | 30 ++++- .../integration/stream-backpressure.test.ts | 40 ++++++ 3 files changed, 171 insertions(+), 24 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 8206e419..54c09728 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -205,13 +205,22 @@ typedef enum { OUTPUT_SETTLEMENT_FAULT_INITIAL_CREATE_GENERIC, OUTPUT_SETTLEMENT_FAULT_INITIAL_PENDING_EXCEPTION, OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL, + OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_PENDING_AFTER_CALL, OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC, OUTPUT_SETTLEMENT_FAULT_FALLBACK_PENDING_EXCEPTION, OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC_AFTER_CALL, } output_settlement_fault_t; +typedef enum { + OUTPUT_EXCEPTION_CLEAR_FAULT_NONE = 0, + OUTPUT_EXCEPTION_CLEAR_FAULT_IS_PENDING, + OUTPUT_EXCEPTION_CLEAR_FAULT_GET_AND_CLEAR, +} output_exception_clear_fault_t; + static output_settlement_fault_t g_test_next_output_settlement_fault = OUTPUT_SETTLEMENT_FAULT_NONE; +static output_exception_clear_fault_t g_test_next_output_exception_clear_fault = + OUTPUT_EXCEPTION_CLEAR_FAULT_NONE; static bool g_test_hold_next_output_delivery = false; static bool g_test_output_delivery_held = false; static bool g_test_release_output_delivery = false; @@ -1896,18 +1905,36 @@ static output_settlement_fault_t test_consume_output_settlement_fault(void) { return fault; } -static bool clear_pending_exception(napi_env env) { +typedef enum { + OUTPUT_EXCEPTION_CLEARED, + OUTPUT_EXCEPTION_NOT_PENDING, + OUTPUT_EXCEPTION_CLEAR_FAILED, +} output_exception_clear_result_t; + +static output_exception_clear_result_t clear_pending_exception(napi_env env) { + output_exception_clear_fault_t fault = OUTPUT_EXCEPTION_CLEAR_FAULT_NONE; + if (g_test_hooks) { + uv_mutex_lock(&g_test_output_mutex); + fault = g_test_next_output_exception_clear_fault; + g_test_next_output_exception_clear_fault = OUTPUT_EXCEPTION_CLEAR_FAULT_NONE; + uv_mutex_unlock(&g_test_output_mutex); + } bool pending = false; - if (napi_is_exception_pending(env, &pending) != napi_ok || !pending) { - return false; + if (fault == OUTPUT_EXCEPTION_CLEAR_FAULT_IS_PENDING || + napi_is_exception_pending(env, &pending) != napi_ok) { + return OUTPUT_EXCEPTION_CLEAR_FAILED; + } + if (!pending) { + return OUTPUT_EXCEPTION_NOT_PENDING; } napi_value exception; - if (napi_get_and_clear_last_exception(env, &exception) != napi_ok) { + if (fault == OUTPUT_EXCEPTION_CLEAR_FAULT_GET_AND_CLEAR || + napi_get_and_clear_last_exception(env, &exception) != napi_ok) { fprintf(stderr, "[DataWeave Node addon] Failed to clear an output settlement exception.\n"); - return false; + return OUTPUT_EXCEPTION_CLEAR_FAILED; } - return true; + return OUTPUT_EXCEPTION_CLEARED; } static napi_status settle_output_fallback( @@ -1969,12 +1996,22 @@ static napi_status settle_output_deferred( if (status == napi_ok && fault == OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL) { status = napi_generic_failure; + } else if (status == napi_ok && + fault == OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_PENDING_AFTER_CALL) { + napi_throw_error(env, NULL, "Injected consumed output settlement exception"); + status = napi_pending_exception; } if (status != napi_ok) { - if (status == napi_pending_exception && - !clear_pending_exception(env)) { - output_flow_release_settlement_ref(flow, env); - return status; + if (status == napi_pending_exception) { + output_exception_clear_result_t clear_result = + clear_pending_exception(env); + if (clear_result == OUTPUT_EXCEPTION_CLEAR_FAILED) { + output_settlement_fail_closed(status); + } + if (clear_result == OUTPUT_EXCEPTION_NOT_PENDING) { + output_flow_release_settlement_ref(flow, env); + return status; + } } output_settlement_fail_closed(status); } @@ -1982,9 +2019,15 @@ static napi_status settle_output_deferred( return napi_ok; } } - if (status == napi_pending_exception && !clear_pending_exception(env)) { - output_flow_release_settlement_ref(flow, env); - return status; + if (status == napi_pending_exception) { + output_exception_clear_result_t clear_result = clear_pending_exception(env); + if (clear_result == OUTPUT_EXCEPTION_CLEAR_FAILED) { + output_settlement_fail_closed(status); + } + if (clear_result == OUTPUT_EXCEPTION_NOT_PENDING) { + output_flow_release_settlement_ref(flow, env); + return status; + } } bool conclude_called = false; @@ -2006,9 +2049,15 @@ static napi_status settle_output_deferred( output_flow_release_settlement_ref(flow, env); return napi_ok; } - if (status == napi_pending_exception && !clear_pending_exception(env)) { - output_flow_release_settlement_ref(flow, env); - return status; + if (status == napi_pending_exception) { + output_exception_clear_result_t clear_result = clear_pending_exception(env); + if (clear_result == OUTPUT_EXCEPTION_CLEAR_FAILED) { + output_settlement_fail_closed(status); + } + if (clear_result == OUTPUT_EXCEPTION_NOT_PENDING) { + output_flow_release_settlement_ref(flow, env); + return status; + } } if (conclude_called) output_settlement_fail_closed(status); @@ -2016,8 +2065,12 @@ static napi_status settle_output_deferred( env, deferred, flow->settlement_fallback_ref, &conclude_called ); if (status != napi_ok) { - if (status == napi_pending_exception && !conclude_called) { - if (!clear_pending_exception(env)) { + if (status == napi_pending_exception) { + output_exception_clear_result_t clear_result = clear_pending_exception(env); + if (clear_result == OUTPUT_EXCEPTION_CLEAR_FAILED) { + output_settlement_fail_closed(status); + } + if (clear_result == OUTPUT_EXCEPTION_NOT_PENDING) { output_flow_release_settlement_ref(flow, env); return status; } @@ -4615,6 +4668,8 @@ static napi_value napi_test_fail_next_output_settlement( fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_PENDING_EXCEPTION; } else if (strcmp(stage, "initial-call-generic-after-call") == 0) { fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_GENERIC_AFTER_CALL; + } else if (strcmp(stage, "initial-call-pending-after-call") == 0) { + fault = OUTPUT_SETTLEMENT_FAULT_INITIAL_CALL_PENDING_AFTER_CALL; } else if (strcmp(stage, "fallback-call-generic") == 0) { fault = OUTPUT_SETTLEMENT_FAULT_FALLBACK_CALL_GENERIC; } else if (strcmp(stage, "fallback-pending-exception") == 0) { @@ -4636,6 +4691,38 @@ static napi_value napi_test_fail_next_output_settlement( return NULL; } +static napi_value napi_test_fail_next_output_exception_clear( + napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + size_t length; + char stage[64]; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || + napi_get_value_string_utf8(env, argv[0], stage, sizeof(stage), &length) != napi_ok) { + napi_throw_type_error(env, NULL, "An exception clear fault stage is required"); + return NULL; + } + output_exception_clear_fault_t fault = OUTPUT_EXCEPTION_CLEAR_FAULT_NONE; + if (strcmp(stage, "is-exception-pending") == 0) { + fault = OUTPUT_EXCEPTION_CLEAR_FAULT_IS_PENDING; + } else if (strcmp(stage, "get-and-clear-last-exception") == 0) { + fault = OUTPUT_EXCEPTION_CLEAR_FAULT_GET_AND_CLEAR; + } else { + napi_throw_range_error(env, NULL, "Unknown exception clear fault stage"); + return NULL; + } + uv_mutex_lock(&g_test_output_mutex); + if (g_test_next_output_exception_clear_fault != + OUTPUT_EXCEPTION_CLEAR_FAULT_NONE) { + uv_mutex_unlock(&g_test_output_mutex); + napi_throw_error(env, NULL, "An output exception clear fault is already armed"); + return NULL; + } + g_test_next_output_exception_clear_fault = fault; + uv_mutex_unlock(&g_test_output_mutex); + return NULL; +} + static napi_value napi_test_hold_next_output_delivery( napi_env env, napi_callback_info info) { (void)info; @@ -4835,6 +4922,8 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_createForeignWrappedObject", fn); napi_create_function(env, "__test_failNextOutputSettlement", NAPI_AUTO_LENGTH, napi_test_fail_next_output_settlement, NULL, &fn); napi_set_named_property(env, exports, "__test_failNextOutputSettlement", fn); + napi_create_function(env, "__test_failNextOutputExceptionClear", NAPI_AUTO_LENGTH, napi_test_fail_next_output_exception_clear, NULL, &fn); + napi_set_named_property(env, exports, "__test_failNextOutputExceptionClear", fn); napi_create_function(env, "__test_holdNextOutputDelivery", NAPI_AUTO_LENGTH, napi_test_hold_next_output_delivery, NULL, &fn); napi_set_named_property(env, exports, "__test_holdNextOutputDelivery", fn); napi_create_function(env, "__test_heldOutputDelivery", NAPI_AUTO_LENGTH, napi_test_held_output_delivery, NULL, &fn); diff --git a/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs b/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs index 43a365f0..5cea6d1a 100644 --- a/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs +++ b/native-lib/node/tests/integration/fixtures/output-settlement-after-call.cjs @@ -5,18 +5,36 @@ const path = require("node:path"); const addonPath = path.resolve(process.cwd(), process.argv[2]); const libPath = process.argv[3]; const fault = process.argv[4]; +const mode = process.argv[5] ?? "streaming"; +const clearFault = process.argv[6]; const addon = require(addonPath); function main() { addon.initialize(libPath); const handle = addon.createEngine(); addon.__test_failNextOutputSettlement(fault); - addon.runScriptStreamingEngine( - handle, - "output application/json\n---\n[]", - "{}", - () => {} - ); + if (clearFault) addon.__test_failNextOutputExceptionClear(clearFault); + if (mode === "streaming") { + addon.runScriptStreamingEngine( + handle, + "output application/json\n---\n[]", + "{}", + () => {} + ); + } else if (mode === "transform") { + addon.runScriptTransformEngine( + handle, + "output application/json\n---\npayload", + "{}", + "payload", + "application/json", + "UTF-8", + () => null, + () => {} + ); + } else { + throw new Error(`Unknown output mode: ${mode}`); + } } main(); diff --git a/native-lib/node/tests/integration/stream-backpressure.test.ts b/native-lib/node/tests/integration/stream-backpressure.test.ts index f4b0b096..240ea33f 100644 --- a/native-lib/node/tests/integration/stream-backpressure.test.ts +++ b/native-lib/node/tests/integration/stream-backpressure.test.ts @@ -40,10 +40,15 @@ type OutputSettlementFault = | "initial-create-generic" | "initial-pending-exception" | "initial-call-generic-after-call" + | "initial-call-pending-after-call" | "fallback-call-generic" | "fallback-pending-exception" | "fallback-call-generic-after-call"; +type OutputExceptionClearFault = + | "is-exception-pending" + | "get-and-clear-last-exception"; + interface TestAddon { initialize(libPath: string): void; createEngine(): number; @@ -70,6 +75,7 @@ interface TestAddon { __test_outputOperationId(operation: NativeStreamingOperation): number; __test_createForeignWrappedObject(): object; __test_failNextOutputSettlement(stage: OutputSettlementFault): void; + __test_failNextOutputExceptionClear(stage: OutputExceptionClearFault): void; __test_holdNextOutputDelivery(): void; __test_heldOutputDelivery(): { held: boolean; sequence: bigint; bytes: number }; __test_releaseOutputDelivery(): void; @@ -900,6 +906,40 @@ describe.sequential("native Node output flow control", () => { expect(child.signal).not.toBe("SIGSEGV"); expect(child.stderr).toContain("Output completion settlement failed after deferred consumption"); }); + + it.each([ + { settlement: "initial-pending-exception", point: "before consumption" }, + { settlement: "initial-call-pending-after-call", point: "after consumption" }, + ] as const)("fails closed when pending-exception clearing fails $point", ({ settlement }) => { + const fixture = join(__dirname, "fixtures", "output-settlement-after-call.cjs"); + const addonPath = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); + + for (const mode of ["streaming", "transform"] as const) { + for (const clearFault of [ + "is-exception-pending", + "get-and-clear-last-exception", + ] as const) { + const child = spawnSync( + process.execPath, + [fixture, addonPath, LIB_PATH, settlement, mode, clearFault], + { + cwd: __dirname, + encoding: "utf-8", + timeout: 30000, + env: { ...process.env, DATAWEAVE_TEST_HOOKS: "1" }, + } + ); + + expect(child.error, `${mode}/${clearFault}: ${child.error?.message}`).toBeUndefined(); + expect( + child.status === 0 && child.signal === null, + `${mode}/${clearFault}: ${child.stderr}` + ).toBe(false); + expect(child.signal, `${mode}/${clearFault}: ${child.stderr}`).not.toBe("SIGSEGV"); + expect(child.stderr).toContain("Output completion settlement failed"); + } + } + }); }); async function waitForOutputDeliveryBarrier(label: string): Promise<{ From cc601da8cd3114d59fa307838d4132d0438db82d Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 11:00:49 -0300 Subject: [PATCH 39/48] test(node): inject detach failures across isolate recovery --- native-lib/node/src/addon.c | 299 +++++++++++++++--- .../integration/detach-poison-hook.test.ts | 121 +++++++ .../fixtures/detach-poison-sync.cjs | 257 +++++++++++++++ .../fixtures/detach-poison-transform.cjs | 124 ++++++++ native-lib/node/vitest.config.ts | 5 +- 5 files changed, 759 insertions(+), 47 deletions(-) create mode 100644 native-lib/node/tests/integration/detach-poison-hook.test.ts create mode 100644 native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs create mode 100644 native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 54c09728..4605874f 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -49,6 +49,8 @@ static int g_native_callback_depth_status; static uv_once_t g_mutex_once = UV_ONCE_INIT; #define CALLBACK_REENTRANCY_CODE "ERR_DATAWEAVE_CALLBACK_REENTRANCY" +#define ISOLATE_POISONED_MESSAGE \ + "DataWeave isolate is unavailable after a thread detach failure; clean up and initialize again." static unsigned native_callback_depth(void) { return (unsigned)(uintptr_t)uv_key_get(&g_native_callback_depth); @@ -227,6 +229,26 @@ static bool g_test_release_output_delivery = false; static uint64_t g_test_held_output_sequence = 0; static size_t g_test_held_output_bytes = 0; +typedef enum { + DETACH_SITE_NONE = 0, + DETACH_SITE_BRIDGE_FINALIZE, + DETACH_SITE_STREAM_WORKER, + DETACH_SITE_TRANSFORM_WORKER, + DETACH_SITE_CREATE_ENGINE, + DETACH_SITE_CREATE_ROLLBACK, + DETACH_SITE_RESOLVER_CREATE, + DETACH_SITE_UNKNOWN_DESTROY, + DETACH_SITE_SYNCHRONOUS_RUN, +} detach_site_t; + +// Process-lifetime test statistics and the one-shot detach fault arm. Every +// read/write is under g_mutex; only the N-API accessors are test-only exports. +static detach_site_t g_test_detach_failure_site = DETACH_SITE_NONE; +static uint64_t g_test_forced_detach_failures = 0; +static uint64_t g_test_isolate_creations = 0; +static uint64_t g_test_teardown_calls = 0; +static uint64_t g_test_abandoned_isolates = 0; + // One record per napi_env that has ever taken an init reference (via // initialize()). init_refs is that env's net initialize()-minus-cleanup() // balance. Created lazily on the env's first initialize(); registers exactly @@ -294,6 +316,10 @@ typedef enum { CLEANUP_RETAIN, CLEANUP_UNRECOVERABLE, } cleanup_result_t; +typedef struct cleanup_thread_result { + cleanup_result_t outcome; + bool teardown_callable; +} cleanup_thread_result_t; static teardown_state_t g_teardown_state = TEARDOWN_NONE; // Set by an adopting initialize() to tell the waiter thread to abort its // queued teardown and leave the live isolate intact. Read/reset by the waiter. @@ -354,6 +380,7 @@ static void test_hold_async_op_if_armed(void) { // leak is observable. Mirrors Python native.py's leak-and-continue // (_release_isolate / _retry_pending_teardown_locked). Caller holds g_mutex. static void abandon_unrecoverable_isolate_locked(void) { + if (g_test_hooks && g_isolate != NULL) g_test_abandoned_isolates++; g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -386,11 +413,23 @@ static void poison_isolate_detach_failure_locked(int detach_rc) { g_isolate_poisoned = true; } -// Lock-taking wrapper for call sites that are NOT already holding g_mutex. -static void poison_isolate_detach_failure(int detach_rc) { +// Centralized policy for every ordinary operation detach. The real Graal +// detach always runs first. Test injection may turn only a successful detach +// at the selected site into one synthetic failure; real failures never consume +// the arm. Any nonzero result poisons this published isolate before later work +// can be admitted. +static int detach_thread_checked(detach_site_t site, void* thread) { + int detach_rc = fn_detach_thread(thread); + if (detach_rc == 0 && !g_test_hooks) return 0; uv_mutex_lock(&g_mutex); - poison_isolate_detach_failure_locked(detach_rc); + if (detach_rc == 0 && g_test_hooks && g_test_detach_failure_site == site) { + g_test_detach_failure_site = DETACH_SITE_NONE; + g_test_forced_detach_failures++; + detach_rc = -1; + } + if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); uv_mutex_unlock(&g_mutex); + return detach_rc; } // One node per cleanup() call that arrived while a teardown was already @@ -583,8 +622,7 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { bool destroyed = false; if (fn_attach_thread(g_isolate, &thread) == 0 && thread != NULL) { fn_destroy_engine(thread, b->handle); - int detach_rc = fn_detach_thread(thread); - if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); + detach_thread_checked(DETACH_SITE_BRIDGE_FINALIZE, thread); destroyed = true; // registry entry removed -> resolver ctx is now dead } // else: attach failed while the isolate is STILL LIVE -- destroy was skipped, @@ -908,6 +946,8 @@ static void init_thread_fn(void* arg) { return; } + if (g_test_hooks) g_test_isolate_creations++; + // review #21 #1: a brand-new isolate starts un-poisoned. Any poison flag left // over from a previously abandoned/leaked isolate must not carry onto this // fresh one. Runs under the init caller's g_mutex (see the g_mutex discipline @@ -932,7 +972,11 @@ static void init_thread_fn(void* arg) { // the caller's `args->result != 0` path (addon.c ~955) sees the recoverable // "no isolate" state, exactly like every other init failure path. if (fn_detach_thread && fn_detach_thread(boot_thread) != 0) { - int td_rc = fn_tear_down_isolate ? fn_tear_down_isolate(boot_thread) : -1; + int td_rc = -1; + if (fn_tear_down_isolate) { + if (g_test_hooks) g_test_teardown_calls++; + td_rc = fn_tear_down_isolate(boot_thread); + } if (td_rc != 0) { fprintf(stderr, "[DataWeave Node addon] bootstrap thread detach AND isolate " @@ -1027,6 +1071,14 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { return NULL; } + uv_mutex_lock(&g_mutex); + bool poisoned_before_drain = g_isolate_poisoned; + uv_mutex_unlock(&g_mutex); + if (poisoned_before_drain) { + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } + // Round-15 (svacas P1): retry any bridge whose engine destroy was skipped on a // transient attach failure (g_stranded_bridges). Drain before taking g_mutex // (drain_stranded_bridges locks internally). If a live isolate survives from a @@ -1037,6 +1089,14 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { uv_mutex_lock(&g_mutex); + // A detach failure is terminal for the currently published isolate. It may + // not be adopted or reused; explicit cleanup must abandon it first. + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } + // A prior last-release could not tear the isolate down and armed the retry // signal (review #6 #3/#4). Because retries otherwise fire only at op // completion (the streaming/transform drains), a zero-op stranded isolate @@ -1084,6 +1144,11 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { // cancel the queued teardown, take a fresh ref, and wake the waiter so it // aborts without tearing down. g_initialized is already 1, so fall through // to the ref-count path below is unnecessary -- return directly. + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!env_init_acquire_and_hook(env)) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Failed to allocate/register env init record"); @@ -1105,6 +1170,11 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { } if (g_initialized) { + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!env_init_acquire_and_hook(env)) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Failed to allocate/register env init record"); @@ -1162,18 +1232,19 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { uv_thread_options_t cleanup_opts; cleanup_opts.flags = UV_THREAD_HAS_STACK_SIZE; cleanup_opts.stack_size = 2 * 1024 * 1024; - cleanup_result_t result = CLEANUP_RETAIN; + cleanup_thread_result_t result = {CLEANUP_RETAIN, false}; int cleanup_spawn_rc = uv_thread_create_ex(&cleanup_tid, &cleanup_opts, cleanup_thread_fn, &result); if (cleanup_spawn_rc == 0) { uv_thread_join(&cleanup_tid); } - if (result == CLEANUP_TORN_DOWN) { + if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; + if (result.outcome == CLEANUP_TORN_DOWN) { // Teardown ran (or there was nothing to tear down) -- clear the globals // so the next initialize() sees a clean slate. g_ref_count is already 0. g_thread = NULL; g_isolate = NULL; g_initialized = 0; - } else if (result == CLEANUP_UNRECOVERABLE) { + } else if (result.outcome == CLEANUP_UNRECOVERABLE) { // teardown+detach double failure (review #17 #1): abandon the isolate and // reset published state so this same initialize() failure path throws // below and a LATER initialize() builds a fresh isolate. Does NOT arm the @@ -2281,7 +2352,6 @@ static void streaming_thread_fn(void* arg) { // back to the OOM_JSON static (which must never be freed; see the guarded // frees below and in call_js_write). char* meta_result = NULL; - int detach_rc = 0; if (rc != 0) { char err[256]; snprintf(err, sizeof(err), "{\"success\":false,\"error\":\"Failed to attach thread (code %d)\"}", rc); @@ -2299,7 +2369,7 @@ static void streaming_thread_fn(void* arg) { meta_result = strdup("{\"success\":false,\"error\":\"Empty response\"}"); if (meta_result == NULL) meta_result = (char*)OOM_JSON; } - detach_rc = fn_detach_thread(worker_thread); + detach_thread_checked(DETACH_SITE_STREAM_WORKER, worker_thread); } // Decrement here, once this thread has fully detached from the isolate -- @@ -2310,7 +2380,6 @@ static void streaming_thread_fn(void* arg) { // here ties g_active_ops to the actual invariant isolate teardown needs // (no GraalVM-attached thread remains), independent of the event loop. uv_mutex_lock(&g_mutex); - if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); // Round-14 (#2/#3): if a prior last-release could not tear the isolate down @@ -2421,6 +2490,11 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // genuine (non-cancelled) PENDING_WAIT or a committed TEARING_DOWN still // rejects. uv_mutex_lock(&g_mutex); + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!g_initialized || (g_teardown_state != TEARDOWN_NONE && !g_teardown_cancelled)) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); @@ -2899,7 +2973,6 @@ static void transform_thread_fn(void* arg) { // so the sentinel below still delivers a terminal result. Mirrors // streaming_thread_fn. char* meta_result = NULL; - int detach_rc = 0; if (rc != 0) { char err[256]; snprintf(err, sizeof(err), "{\"success\":false,\"error\":\"Failed to attach thread (code %d)\"}", rc); @@ -2920,14 +2993,13 @@ static void transform_thread_fn(void* arg) { meta_result = strdup("{\"success\":false,\"error\":\"Empty response\"}"); if (meta_result == NULL) meta_result = (char*)OOM_JSON; } - detach_rc = fn_detach_thread(worker_thread); + detach_thread_checked(DETACH_SITE_TRANSFORM_WORKER, worker_thread); } // See streaming_thread_fn's comment: decrement here (after detach), not in // call_js_transform_write's completion branch, to avoid the same // circular-wait deadlock against napi_initialize's pending-teardown wait. uv_mutex_lock(&g_mutex); - if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); // Round-14 (#2/#3): retry a stranded teardown now that this op has drained. @@ -3036,6 +3108,11 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i // post-adoption op throws "Not initialized". A genuine (non-cancelled) // PENDING_WAIT or a committed TEARING_DOWN still rejects. uv_mutex_lock(&g_mutex); + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!g_initialized || (g_teardown_state != TEARDOWN_NONE && !g_teardown_cancelled)) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); @@ -3422,6 +3499,11 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { // teardown transition and the g_active_ops==0 fast path also hold g_mutex. uv_mutex_lock(&g_mutex); env_init_rec_t* self = env_init_rec_find_locked(env); + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!g_initialized || g_isolate == NULL || g_teardown_state == TEARDOWN_TEARING_DOWN || self == NULL || self->init_refs == 0) { @@ -3438,8 +3520,7 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { napi_throw_error(env, NULL, "Failed to attach thread"); return NULL; } long long handle = fn_create_engine(thread); - int detach_rc = fn_detach_thread(thread); - if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); + detach_thread_checked(DETACH_SITE_CREATE_ENGINE, thread); // The Java @CEntryPoint exception handler explicitly returns 0 as its ABI // exception sentinel when engine construction throws. The real handle // registry only ever hands out handles >= 1, so any handle <= 0 means @@ -3467,17 +3548,18 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { if (rec == NULL) { // Roll back the engine we just created so we don't leak a registered but // unrecorded handle. fn_destroy_engine attaches its own thread. - int detach_rc = 0; if (fn_destroy_engine) { void* t2 = NULL; - if (fn_attach_thread(g_isolate, &t2) == 0) { fn_destroy_engine(t2, handle); detach_rc = fn_detach_thread(t2); } + if (fn_attach_thread(g_isolate, &t2) == 0) { + fn_destroy_engine(t2, handle); + detach_thread_checked(DETACH_SITE_CREATE_ROLLBACK, t2); + } } // review #21 #1 (final-review completeness): this OOM-rollback detach is an // ordinary detach too -- a failure here strands a phantom thread and would // wedge a later teardown, so poison in the same critical section as the // g_active_ops-- (before the decrement/broadcast), matching the other sites. uv_mutex_lock(&g_mutex); - if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Failed to allocate engine record"); return NULL; @@ -3559,6 +3641,12 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i // must not decrement a reservation not yet held) and BEFORE fn_attach_thread. uv_mutex_lock(&g_mutex); env_init_rec_t* self = env_init_rec_find_locked(env); + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + napi_delete_reference(env, bridge->resolver_js); free(bridge); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!g_initialized || g_isolate == NULL || g_teardown_state == TEARDOWN_TEARING_DOWN || self == NULL || self->init_refs == 0) { @@ -3577,8 +3665,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i napi_throw_error(env, NULL, "Failed to attach thread"); return NULL; } long long handle = fn_create_engine_with_resolver(thread, resolve_module_callback, (void*)bridge); - int detach_rc = fn_detach_thread(thread); - if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); + detach_thread_checked(DETACH_SITE_RESOLVER_CREATE, thread); // Same invalid-handle guard as napi_create_engine: the Java @CEntryPoint // exception handler explicitly returns handle == 0 as its ABI sentinel, @@ -3767,14 +3854,12 @@ static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { g_active_ops++; // pins the live isolate against teardown for this attach uv_mutex_unlock(&g_mutex); void* thread = NULL; - int detach_rc = 0; if (fn_attach_thread(g_isolate, &thread) == 0 && thread != NULL) { fn_destroy_engine(thread, handle); - detach_rc = fn_detach_thread(thread); + detach_thread_checked(DETACH_SITE_UNKNOWN_DESTROY, thread); } // Verbatim g_active_ops release pattern. uv_mutex_lock(&g_mutex); - if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); @@ -3836,6 +3921,12 @@ static napi_value napi_run_script_engine(napi_env env, napi_callback_info info) // post-adoption op throws "Not initialized". A genuine (non-cancelled) // PENDING_WAIT or a committed TEARING_DOWN still rejects. uv_mutex_lock(&g_mutex); + if (g_isolate_poisoned) { + uv_mutex_unlock(&g_mutex); + free(script); free(inputs); + napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); + return NULL; + } if (!g_initialized || (g_teardown_state != TEARDOWN_NONE && !g_teardown_cancelled)) { uv_mutex_unlock(&g_mutex); free(script); free(inputs); @@ -3870,8 +3961,7 @@ static napi_value napi_run_script_engine(napi_env env, napi_callback_info info) char* result_copy = result ? strdup(result) : NULL; if (result != NULL) fn_free_cstring(thread, result); - int detach_rc = fn_detach_thread(thread); - if (detach_rc != 0) poison_isolate_detach_failure(detach_rc); + detach_thread_checked(DETACH_SITE_SYNCHRONOUS_RUN, thread); free(script); free(inputs); // Round-11 (#3): release the per-engine pin (may finalize a destroy that a @@ -3926,7 +4016,7 @@ static void call_js_teardown_done(napi_env env, napi_value js_callback, void* co free(waiter); } -// `arg` is a cleanup_result_t* out-param: the caller must set it to +// `arg` is a cleanup_thread_result_t* out-param: the caller must initialize its // CLEANUP_RETAIN before spawning this thread (so a spawn that never runs, or the // attach-failure early return, leaves the live isolate retained) and read it // after uv_thread_join returns. Mirrors teardown_waiter_thread_fn's outcome @@ -3934,8 +4024,10 @@ static void call_js_teardown_done(napi_env env, napi_value js_callback, void* co // down" (clear g_thread/g_isolate/g_initialized/g_ref_count) from "attach or // teardown failed but the isolate is still reachable" (retain + arm retry) from // "teardown AND detach both failed" (unrecoverable -- leak the isolate). +// Follow-up detaches remain direct: they classify that teardown double failure, +// rather than poisoning an otherwise completed ordinary operation. static void cleanup_thread_fn(void* arg) { - cleanup_result_t* out_result = (cleanup_result_t*)arg; + cleanup_thread_result_t* result = (cleanup_thread_result_t*)arg; // graal_tear_down_isolate() must be passed the IsolateThread belonging to the // *calling* OS thread. g_thread was created by graal_create_isolate() on the // (now-exited, already-joined) init thread, so it is invalid here — passing it @@ -3944,7 +4036,7 @@ static void cleanup_thread_fn(void* arg) { // to obtain a valid local IsolateThread, then tear down with that. if (!fn_tear_down_isolate || !fn_attach_thread || !g_isolate) { // Nothing to tear down (no isolate / FFI unavailable) -- safe to clear. - *out_result = CLEANUP_TORN_DOWN; + result->outcome = CLEANUP_TORN_DOWN; return; } if (g_isolate_poisoned) { @@ -3953,7 +4045,7 @@ static void cleanup_thread_fn(void* arg) { // NOT attempt teardown -- signal leak-and-continue (the caller runs // abandon_unrecoverable_isolate_locked()). Reading g_isolate_poisoned unlocked // is safe: the caller spawns+joins this thread while holding g_mutex. - *out_result = CLEANUP_UNRECOVERABLE; + result->outcome = CLEANUP_UNRECOVERABLE; return; } void* local_thread = NULL; @@ -3963,22 +4055,26 @@ static void cleanup_thread_fn(void* arg) { // (or it becomes unreachable and can never be torn down) and arms the retry. return; } + // The attached worker is about to invoke teardown. The caller increments the + // counter after join, avoiding a second g_mutex lock while synchronous callers + // deliberately hold it across this worker. + result->teardown_callable = true; // Check the teardown return code (0 == success). On nonzero the isolate is // still live and this thread is still attached to it -- detach before exiting // or the live isolate keeps a phantom attached thread that can block/fail a // later retry teardown (review #7 #1). On success the isolate is gone: do NOT // detach (would be a UAF). if (fn_tear_down_isolate(local_thread) == 0) { - *out_result = CLEANUP_TORN_DOWN; + result->outcome = CLEANUP_TORN_DOWN; } else if (fn_detach_thread(local_thread) == 0) { // Teardown failed but the worker detached cleanly: the isolate is live and // reachable -- retain it and (per the caller's own logic) arm the retry // (review #6 #3). - *out_result = CLEANUP_RETAIN; + result->outcome = CLEANUP_RETAIN; } else { // Teardown AND detach both failed (review #17 #1): the worker is stuck // attached, so this isolate can never be torn down. Signal leak-and-continue. - *out_result = CLEANUP_UNRECOVERABLE; + result->outcome = CLEANUP_UNRECOVERABLE; } } @@ -3987,6 +4083,8 @@ static void cleanup_thread_fn(void* arg) { // op has drained, performs isolate teardown exactly like cleanup_thread_fn // does on the unchanged fast path, then resolves every caller who is waiting // on this same teardown (there may be more than one -- see g_teardown_waiters). +// Its teardown-failure follow-up detach is direct for the same double-failure +// classification documented on cleanup_thread_fn. static void teardown_waiter_thread_fn(void* arg) { (void)arg; @@ -4017,6 +4115,11 @@ static void teardown_waiter_thread_fn(void* arg) { } else if (!cancelled && fn_tear_down_isolate && fn_attach_thread && g_isolate) { void* local_thread = NULL; if (fn_attach_thread(g_isolate, &local_thread) == 0 && local_thread != NULL) { + if (g_test_hooks) { + uv_mutex_lock(&g_mutex); + g_test_teardown_calls++; + uv_mutex_unlock(&g_mutex); + } if (fn_tear_down_isolate(local_thread) == 0) { result = CLEANUP_TORN_DOWN; } else if (fn_detach_thread(local_thread) == 0) { @@ -4173,8 +4276,8 @@ static napi_value already_resolved_promise(napi_env env) { // failure it leaves g_teardown_needed set to retry on the next drain. Spawns+joins // cleanup_thread_fn while holding g_mutex, exactly as the Case-4 / // isolate_ref_release_n_locked g_active_ops==0 branch does; cleanup_thread_fn -// takes no lock and makes no napi call, so this is deadlock-free and thread-safe -// from any drain site. +// makes no N-API calls and writes only its caller-owned result struct, so this +// is deadlock-free and thread-safe from any drain site. static void retry_stranded_teardown_locked(void) { if (!g_teardown_needed) return; if (g_ref_count > 0) { g_teardown_needed = false; return; } // adopted -> keep @@ -4185,16 +4288,17 @@ static void retry_stranded_teardown_locked(void) { uv_thread_options_t opts; opts.flags = UV_THREAD_HAS_STACK_SIZE; opts.stack_size = 2 * 1024 * 1024; - cleanup_result_t result = CLEANUP_RETAIN; + cleanup_thread_result_t result = {CLEANUP_RETAIN, false}; int spawn_rc = uv_thread_create_ex(&tid, &opts, cleanup_thread_fn, &result); if (spawn_rc == 0) uv_thread_join(&tid); - if (result == CLEANUP_TORN_DOWN) { + if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; + if (result.outcome == CLEANUP_TORN_DOWN) { g_thread = NULL; g_isolate = NULL; g_initialized = 0; g_ref_count = 0; g_teardown_needed = false; - } else if (result == CLEANUP_UNRECOVERABLE) { + } else if (result.outcome == CLEANUP_UNRECOVERABLE) { // teardown+detach double failure (review #17 #1): abandon + leak; the helper // also clears g_teardown_needed so this stranded-teardown retry stops. abandon_unrecoverable_isolate_locked(); @@ -4214,17 +4318,18 @@ static void isolate_ref_release_n_locked(int n) { uv_thread_options_t opts; opts.flags = UV_THREAD_HAS_STACK_SIZE; opts.stack_size = 2 * 1024 * 1024; - cleanup_result_t result = CLEANUP_RETAIN; + cleanup_thread_result_t result = {CLEANUP_RETAIN, false}; int spawn_rc = uv_thread_create_ex(&tid, &opts, cleanup_thread_fn, &result); if (spawn_rc == 0) { uv_thread_join(&tid); } - if (result == CLEANUP_TORN_DOWN) { + if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; + if (result.outcome == CLEANUP_TORN_DOWN) { g_thread = NULL; g_isolate = NULL; g_initialized = 0; g_ref_count = 0; - } else if (result == CLEANUP_UNRECOVERABLE) { + } else if (result.outcome == CLEANUP_UNRECOVERABLE) { // teardown+detach double failure (review #17 #1): abandon + leak the // isolate; do NOT arm the retry. Mirrors Python native.py leak-and-continue. abandon_unrecoverable_isolate_locked(); @@ -4400,7 +4505,7 @@ static napi_value release_isolate_ref_locked(napi_env env) { // never touches it), leaves the live isolate retained + the retry armed. // uv_thread_join is synchronous, so when spawn_rc == 0 this stack variable safely // outlives the thread's write to it. - cleanup_result_t result = CLEANUP_RETAIN; + cleanup_thread_result_t result = {CLEANUP_RETAIN, false}; int spawn_rc = uv_thread_create_ex(&tid, &opts, cleanup_thread_fn, &result); if (spawn_rc == 0) { uv_thread_join(&tid); @@ -4416,12 +4521,13 @@ static napi_value release_isolate_ref_locked(napi_env env) { // initialize() correctly ref-counts the surviving isolate instead of // building a second one (identical semantics to teardown_waiter_thread_fn's // attach-failure path). - if (result == CLEANUP_TORN_DOWN) { + if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; + if (result.outcome == CLEANUP_TORN_DOWN) { g_thread = NULL; g_isolate = NULL; g_initialized = 0; g_ref_count = 0; - } else if (result == CLEANUP_UNRECOVERABLE) { + } else if (result.outcome == CLEANUP_UNRECOVERABLE) { // teardown+detach double failure (review #17 #1): abandon + leak the // isolate; the promise below still RESOLVES (deliberate, per the note that // follows). The helper emits its own stderr diagnostic. Mirrors Python @@ -4587,6 +4693,97 @@ static napi_value napi_test_resolver_ref_delete_count(napi_env env, napi_callbac return out; } +static detach_site_t detach_site_from_name(const char* name) { + if (strcmp(name, "bridge-finalize") == 0) return DETACH_SITE_BRIDGE_FINALIZE; + if (strcmp(name, "stream-worker") == 0) return DETACH_SITE_STREAM_WORKER; + if (strcmp(name, "transform-worker") == 0) return DETACH_SITE_TRANSFORM_WORKER; + if (strcmp(name, "create-engine") == 0) return DETACH_SITE_CREATE_ENGINE; + if (strcmp(name, "create-rollback") == 0) return DETACH_SITE_CREATE_ROLLBACK; + if (strcmp(name, "resolver-create") == 0) return DETACH_SITE_RESOLVER_CREATE; + if (strcmp(name, "unknown-destroy") == 0) return DETACH_SITE_UNKNOWN_DESTROY; + if (strcmp(name, "synchronous-run") == 0) return DETACH_SITE_SYNCHRONOUS_RUN; + return DETACH_SITE_NONE; +} + +static napi_value napi_test_force_detach_failure_once( + napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + napi_valuetype type; + char site_name[64]; + size_t length; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || + napi_typeof(env, argv[0], &type) != napi_ok || type != napi_string || + napi_get_value_string_utf8( + env, argv[0], site_name, sizeof(site_name), &length) != napi_ok) { + napi_throw_type_error(env, NULL, "A detach site string is required"); + return NULL; + } + detach_site_t site = detach_site_from_name(site_name); + if (site == DETACH_SITE_NONE) { + napi_throw_range_error(env, NULL, "Unknown detach site"); + return NULL; + } + uv_mutex_lock(&g_mutex); + if (g_test_detach_failure_site != DETACH_SITE_NONE) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "A detach failure is already armed"); + return NULL; + } + g_test_detach_failure_site = site; + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_isolate_poisoned(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + bool poisoned = g_isolate_poisoned; + uv_mutex_unlock(&g_mutex); + napi_value out; + napi_get_boolean(env, poisoned, &out); + return out; +} + +static napi_value test_uint64_counter(napi_env env, uint64_t value) { + napi_value out; + napi_create_bigint_uint64(env, value, &out); + return out; +} + +static napi_value napi_test_isolate_creation_count(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t value = g_test_isolate_creations; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, value); +} + +static napi_value napi_test_teardown_call_count(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t value = g_test_teardown_calls; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, value); +} + +static napi_value napi_test_abandoned_isolate_count(napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t value = g_test_abandoned_isolates; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, value); +} + +static napi_value napi_test_forced_detach_failure_count( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t value = g_test_forced_detach_failures; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, value); +} + static napi_value napi_test_hold_next_async_op(napi_env env, napi_callback_info info) { (void)info; uv_mutex_lock(&g_mutex); @@ -4908,6 +5105,18 @@ static napi_value Init(napi_env env, napi_value exports) { napi_set_named_property(env, exports, "__test_strandedCount", fn); napi_create_function(env, "__test_resolverRefDeleteCount", NAPI_AUTO_LENGTH, napi_test_resolver_ref_delete_count, NULL, &fn); napi_set_named_property(env, exports, "__test_resolverRefDeleteCount", fn); + napi_create_function(env, "__test_forceDetachFailureOnce", NAPI_AUTO_LENGTH, napi_test_force_detach_failure_once, NULL, &fn); + napi_set_named_property(env, exports, "__test_forceDetachFailureOnce", fn); + napi_create_function(env, "__test_isolatePoisoned", NAPI_AUTO_LENGTH, napi_test_isolate_poisoned, NULL, &fn); + napi_set_named_property(env, exports, "__test_isolatePoisoned", fn); + napi_create_function(env, "__test_isolateCreationCount", NAPI_AUTO_LENGTH, napi_test_isolate_creation_count, NULL, &fn); + napi_set_named_property(env, exports, "__test_isolateCreationCount", fn); + napi_create_function(env, "__test_teardownCallCount", NAPI_AUTO_LENGTH, napi_test_teardown_call_count, NULL, &fn); + napi_set_named_property(env, exports, "__test_teardownCallCount", fn); + napi_create_function(env, "__test_abandonedIsolateCount", NAPI_AUTO_LENGTH, napi_test_abandoned_isolate_count, NULL, &fn); + napi_set_named_property(env, exports, "__test_abandonedIsolateCount", fn); + napi_create_function(env, "__test_forcedDetachFailureCount", NAPI_AUTO_LENGTH, napi_test_forced_detach_failure_count, NULL, &fn); + napi_set_named_property(env, exports, "__test_forcedDetachFailureCount", fn); napi_create_function(env, "__test_holdNextAsyncOp", NAPI_AUTO_LENGTH, napi_test_hold_next_async_op, NULL, &fn); napi_set_named_property(env, exports, "__test_holdNextAsyncOp", fn); napi_create_function(env, "__test_asyncOpHeld", NAPI_AUTO_LENGTH, napi_test_async_op_held, NULL, &fn); diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts new file mode 100644 index 00000000..16f94484 --- /dev/null +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -0,0 +1,121 @@ +import { describe, expect, it } from "vitest"; +import { spawnSync } from "node:child_process"; +import { join } from "node:path"; +import { findLibrary } from "../../src/utils"; + +const ADDON_PATH = join(__dirname, "..", "..", "build", "Release", "dwlib_addon.node"); +const LIB_PATH = findLibrary(); +const SYNC_FIXTURE = join(__dirname, "fixtures", "detach-poison-sync.cjs"); +const TRANSFORM_FIXTURE = join(__dirname, "fixtures", "detach-poison-transform.cjs"); +const FATAL_STDERR = + /Must either be at a safepoint or in native mode|fatal error|SIGSEGV|segmentation fault|SIGABRT|\babort(?:ed|ing)?\b/i; + +const DETACH_SITES = [ + "bridge-finalize", + "stream-worker", + "transform-worker", + "create-engine", + "create-rollback", + "resolver-create", + "unknown-destroy", + "synchronous-run", +] as const; + +function runFixture( + fixture: string, + args: string[], + hooks = true +): Record { + const env = { ...process.env }; + if (hooks) env.DATAWEAVE_TEST_HOOKS = "1"; + else delete env.DATAWEAVE_TEST_HOOKS; + + const child = spawnSync( + process.execPath, + [fixture, ADDON_PATH, LIB_PATH, ...args], + { + cwd: __dirname, + encoding: "utf-8", + timeout: 30_000, + env, + } + ); + + expect(child.error, child.error?.message).toBeUndefined(); + expect(child.signal, child.stderr).toBeNull(); + expect(child.status, child.stderr).not.toBe(99); + expect(child.status, child.stderr).toBe(0); + expect(child.stderr, child.stderr).not.toMatch(FATAL_STDERR); + expect(child.stdout.trim(), child.stderr).not.toBe(""); + return JSON.parse(child.stdout.trim()) as Record; +} + +describe("detach failure poisoning and recovery", () => { + it("preserves a completed synchronous result, fails later admission closed, and recovers with a fresh isolate", () => { + expect(runFixture(SYNC_FIXTURE, ["recovery"])).toMatchObject({ + firstResult: "42", + rejectedAdmissions: 6, + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + + it("poisons a transform worker during an active final cleanup without hanging or tearing down the old isolate", () => { + expect(runFixture(TRANSFORM_FIXTURE, [])).toMatchObject({ + transformResult: "detach poison transform", + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + + it.each(DETACH_SITES)("accepts the exact detach site name %s", (site) => { + expect(runFixture(SYNC_FIXTURE, ["validate-site", site])).toEqual({ site }); + }); + + it.each([ + "bridge-finalize", + "stream-worker", + "create-engine", + "resolver-create", + "unknown-destroy", + ] as const)("forces and recovers from the ordinary detach site %s", (site) => { + expect(runFixture(SYNC_FIXTURE, ["exercise-site", site])).toMatchObject({ + site, + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + + it("forces a synchronous-run detach only after the successful result is copied", () => { + expect(runFixture(SYNC_FIXTURE, ["exercise-site", "synchronous-run"])).toMatchObject({ + site: "synchronous-run", + forcedFailures: 1, + abandoned: 1, + triggeringResult: "42", + freshResult: "42", + }); + }); + + it("rejects invalid sites and refuses to silently replace an armed failure", () => { + expect(runFixture(SYNC_FIXTURE, ["invalid-arguments"])).toEqual({ + invalidArguments: 4, + duplicateArmRejected: true, + }); + }); + + it("consumes only the selected site once while poison remains set", () => { + expect(runFixture(SYNC_FIXTURE, ["one-shot-site"])).toEqual({ + forcedFailures: 1, + poisonPersisted: true, + }); + }); + + it("does not export detach-poison hooks without DATAWEAVE_TEST_HOOKS", () => { + expect(runFixture(SYNC_FIXTURE, ["hooks-absent"], false)).toEqual({ + hooksAbsent: true, + }); + }); +}); diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs new file mode 100644 index 00000000..c961f65f --- /dev/null +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -0,0 +1,257 @@ +"use strict"; + +const path = require("node:path"); + +const addonPath = path.resolve(process.cwd(), process.argv[2]); +const libPath = process.argv[3]; +const mode = process.argv[4]; +const addon = require(addonPath); + +const POISONED_MESSAGE = + "DataWeave isolate is unavailable after a thread detach failure; clean up and initialize again."; +const DETACH_HOOKS = [ + "__test_forceDetachFailureOnce", + "__test_isolatePoisoned", + "__test_isolateCreationCount", + "__test_teardownCallCount", + "__test_abandonedIsolateCount", + "__test_forcedDetachFailureCount", +]; + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function expectThrow(invoke, message) { + try { + invoke(); + } catch (error) { + assert(error instanceof Error, "expected an Error"); + assert(error.message === message, `expected ${JSON.stringify(message)}, got ${JSON.stringify(error.message)}`); + return; + } + throw new Error(`expected synchronous throw: ${message}`); +} + +function count(name) { + const value = addon[name](); + assert(typeof value === "bigint", `${name} must return a lossless bigint`); + return value; +} + +function successfulResult(raw) { + const result = JSON.parse(raw); + assert(result.success === true, `expected successful result, got ${raw}`); + return Buffer.from(result.result, "base64").toString("utf8"); +} + +function withTimeout(promise, label) { + let timer; + return Promise.race([ + promise, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${label} timed out`)), 10_000); + }), + ]).finally(() => clearTimeout(timer)); +} + +async function recovery() { + for (const hook of DETACH_HOOKS) { + assert(typeof addon[hook] === "function", `${hook} is undefined`); + } + + addon.initialize(libPath); + const firstHandle = addon.createEngine(); + const creationBefore = count("__test_isolateCreationCount"); + const teardownBefore = count("__test_teardownCallCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + const forcedBefore = count("__test_forcedDetachFailureCount"); + + addon.__test_forceDetachFailureOnce("synchronous-run"); + const otherHandle = addon.createEngine(); + assert(addon.__test_isolatePoisoned() === false, "a different detach site consumed the arm"); + assert(count("__test_forcedDetachFailureCount") === forcedBefore, "forced count changed at a different site"); + + const firstResult = successfulResult( + addon.runScriptEngine(firstHandle, "output application/json --- 6 * 7", "{}") + ); + assert(firstResult === "42", `unexpected first result: ${firstResult}`); + assert(addon.__test_isolatePoisoned() === true, "the isolate was not poisoned"); + assert(count("__test_forcedDetachFailureCount") === forcedBefore + 1n, "forced failure was not one-shot"); + + let rejectedAdmissions = 0; + const liveFlowsBefore = addon.__test_outputStats().liveFlows; + const reject = (invoke) => { + expectThrow(invoke, POISONED_MESSAGE); + rejectedAdmissions++; + }; + reject(() => addon.runScriptEngine(firstHandle, "output application/json --- 1", "{}")); + reject(() => addon.createEngine()); + reject(() => addon.createEngineWithResolver(() => null)); + reject(() => addon.runScriptStreamingEngine(firstHandle, "output application/json --- []", "{}", () => {})); + reject(() => addon.runScriptTransformEngine( + firstHandle, + "output application/json --- payload", + "{}", + "payload", + "application/json", + "UTF-8", + () => null, + () => {} + )); + reject(() => addon.initialize(libPath)); + assert(addon.__test_outputStats().liveFlows === liveFlowsBefore, "rejected async work allocated an output flow"); + + // A poisoned generation is abandoned as a unit. destroyEngine is safe but + // intentionally performs no further Graal attachment on it. + addon.destroyEngine(otherHandle); + addon.destroyEngine(firstHandle); + await withTimeout(addon.cleanup(), "poisoned isolate cleanup"); + assert(count("__test_teardownCallCount") === teardownBefore, "poisoned cleanup invoked teardown"); + assert(count("__test_abandonedIsolateCount") === abandonedBefore + 1n, "poisoned cleanup did not abandon once"); + + addon.initialize(libPath); + assert(count("__test_isolateCreationCount") === creationBefore + 1n, "recovery did not create one fresh isolate"); + const freshHandle = addon.createEngine(); + const freshResult = successfulResult( + addon.runScriptEngine(freshHandle, "output application/json --- 6 * 7", "{}") + ); + assert(freshResult === "42", `unexpected fresh result: ${freshResult}`); + assert(addon.__test_isolatePoisoned() === false, "fresh isolate inherited poison"); + addon.destroyEngine(freshHandle); + await withTimeout(addon.cleanup(), "fresh isolate cleanup"); + assert(count("__test_teardownCallCount") === teardownBefore + 1n, "fresh isolate was not torn down once"); + assert(count("__test_abandonedIsolateCount") === abandonedBefore + 1n, "fresh cleanup changed abandon count"); + + return { + firstResult, + rejectedAdmissions, + forcedFailures: Number(count("__test_forcedDetachFailureCount") - forcedBefore), + abandoned: Number(count("__test_abandonedIsolateCount") - abandonedBefore), + freshResult, + }; +} + +function validateSite(site) { + assert(typeof addon.__test_forceDetachFailureOnce === "function", "detach failure hook is undefined"); + addon.__test_forceDetachFailureOnce(site); + return { site }; +} + +function invalidArguments() { + let invalidArguments = 0; + for (const value of [undefined, null, 42, "not-a-detach-site"]) { + try { + if (value === undefined) addon.__test_forceDetachFailureOnce(); + else addon.__test_forceDetachFailureOnce(value); + } catch (error) { + assert(error instanceof Error, "invalid detach site did not throw Error"); + invalidArguments++; + } + } + addon.__test_forceDetachFailureOnce("synchronous-run"); + let duplicateArmRejected = false; + try { + addon.__test_forceDetachFailureOnce("create-engine"); + } catch (error) { + duplicateArmRejected = error instanceof Error; + } + assert(duplicateArmRejected, "a second arm silently replaced the first"); + return { invalidArguments, duplicateArmRejected }; +} + +async function oneShotSite() { + addon.initialize(libPath); + const first = addon.createEngine(); + const second = addon.createEngine(); + const forcedBefore = count("__test_forcedDetachFailureCount"); + addon.__test_forceDetachFailureOnce("bridge-finalize"); + addon.destroyEngine(first); + assert(addon.__test_isolatePoisoned() === true, "selected bridge detach did not poison"); + addon.destroyEngine(second); + const forcedFailures = Number(count("__test_forcedDetachFailureCount") - forcedBefore); + const poisonPersisted = addon.__test_isolatePoisoned(); + assert(forcedFailures === 1, "the same detach site consumed more than once"); + assert(poisonPersisted === true, "a later successful detach cleared poison early"); + await withTimeout(addon.cleanup(), "one-shot isolate cleanup"); + return { forcedFailures, poisonPersisted }; +} + +async function exerciseSite(site) { + addon.initialize(libPath); + const handle = addon.createEngine(); + const forcedBefore = count("__test_forcedDetachFailureCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + addon.__test_forceDetachFailureOnce(site); + let triggeringResult; + + if (site === "bridge-finalize") { + addon.destroyEngine(handle); + } else if (site === "stream-worker") { + const operation = addon.runScriptStreamingEngine( + handle, + "output application/json --- []", + "{}", + (_chunk, sequence) => operation.acknowledge(sequence, _chunk.length) + ); + const result = JSON.parse(await withTimeout(operation.completion, "stream detach completion")); + assert(result.success === true, `stream detach operation failed: ${JSON.stringify(result)}`); + addon.destroyEngine(handle); + } else if (site === "create-engine") { + const created = addon.createEngine(); + addon.destroyEngine(created); + addon.destroyEngine(handle); + } else if (site === "resolver-create") { + const created = addon.createEngineWithResolver(() => null); + addon.destroyEngine(created); + addon.destroyEngine(handle); + } else if (site === "unknown-destroy") { + addon.destroyEngine(handle + 1_000_000); + addon.destroyEngine(handle); + } else if (site === "synchronous-run") { + triggeringResult = successfulResult( + addon.runScriptEngine(handle, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(handle); + } else { + throw new Error(`exercise-site does not support ${site}`); + } + + assert(addon.__test_isolatePoisoned() === true, `${site} did not poison isolate`); + await withTimeout(addon.cleanup(), `${site} cleanup`); + addon.initialize(libPath); + const fresh = addon.createEngine(); + const freshResult = successfulResult( + addon.runScriptEngine(fresh, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(fresh); + await withTimeout(addon.cleanup(), `${site} fresh cleanup`); + return { + site, + forcedFailures: Number(count("__test_forcedDetachFailureCount") - forcedBefore), + abandoned: Number(count("__test_abandonedIsolateCount") - abandonedBefore), + triggeringResult, + freshResult, + }; +} + +function hooksAbsent() { + return { hooksAbsent: DETACH_HOOKS.every((hook) => addon[hook] === undefined) }; +} + +async function main() { + let result; + if (mode === "recovery") result = await recovery(); + else if (mode === "validate-site") result = validateSite(process.argv[5]); + else if (mode === "invalid-arguments") result = invalidArguments(); + else if (mode === "one-shot-site") result = await oneShotSite(); + else if (mode === "exercise-site") result = await exerciseSite(process.argv[5]); + else if (mode === "hooks-absent") result = hooksAbsent(); + else throw new Error(`unknown fixture mode: ${mode}`); + process.stdout.write(`${JSON.stringify(result)}\n`); +} + +main().catch((error) => { + process.stderr.write(`${error.stack ?? error}\n`); + process.exitCode = 99; +}); diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs new file mode 100644 index 00000000..91d18580 --- /dev/null +++ b/native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs @@ -0,0 +1,124 @@ +"use strict"; + +const path = require("node:path"); + +const addonPath = path.resolve(process.cwd(), process.argv[2]); +const libPath = process.argv[3]; +const addon = require(addonPath); +const INPUT = Buffer.from("detach poison transform"); + +function assert(condition, message) { + if (!condition) throw new Error(message); +} + +function count(name) { + const value = addon[name](); + assert(typeof value === "bigint", `${name} must return a lossless bigint`); + return value; +} + +function successfulResult(raw) { + const result = JSON.parse(raw); + assert(result.success === true, `expected successful result, got ${raw}`); + return result; +} + +function immediate() { + return new Promise((resolve) => setImmediate(resolve)); +} + +async function waitFor(predicate, label) { + const deadline = Date.now() + 10_000; + while (!predicate()) { + if (Date.now() >= deadline) throw new Error(`${label} timed out`); + await immediate(); + } +} + +function withTimeout(promise, label) { + let timer; + return Promise.race([ + promise, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${label} timed out`)), 10_000); + }), + ]).finally(() => clearTimeout(timer)); +} + +async function main() { + addon.initialize(libPath); + const handle = addon.createEngine(); + const creationBefore = count("__test_isolateCreationCount"); + const teardownBefore = count("__test_teardownCallCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + const forcedBefore = count("__test_forcedDetachFailureCount"); + + addon.__test_forceDetachFailureOnce("transform-worker"); + addon.__test_holdNextAsyncOp(); + const chunks = []; + let read = false; + let operation; + operation = addon.runScriptTransformEngine( + handle, + "output application/octet-stream deferred=true --- payload", + "{}", + "payload", + "application/octet-stream", + null, + () => { + if (read) return null; + read = true; + return INPUT; + }, + (chunk, sequence) => { + chunks.push(chunk); + operation.acknowledge(sequence, chunk.length); + } + ); + + await waitFor(() => addon.__test_asyncOpHeld(), "transform admission barrier"); + addon.destroyEngine(handle); + const cleanup = addon.cleanup(); + addon.__test_releaseAsyncOp(); + + const [raw] = await withTimeout( + Promise.all([operation.completion, cleanup]), + "transform completion and poisoned cleanup" + ); + successfulResult(raw); + const transformResult = Buffer.concat(chunks).toString("utf8"); + assert(transformResult === INPUT.toString("utf8"), `unexpected transform output: ${transformResult}`); + assert(count("__test_forcedDetachFailureCount") === forcedBefore + 1n, "transform detach was not forced once"); + assert(count("__test_teardownCallCount") === teardownBefore, "poisoned transform cleanup invoked teardown"); + assert(count("__test_abandonedIsolateCount") === abandonedBefore + 1n, "poisoned transform cleanup did not abandon once"); + + addon.initialize(libPath); + assert(count("__test_isolateCreationCount") === creationBefore + 1n, "transform recovery did not create a fresh isolate"); + const freshHandle = addon.createEngine(); + const freshResultRaw = addon.runScriptEngine( + freshHandle, + "output application/json --- 6 * 7", + "{}" + ); + const freshEnvelope = successfulResult(freshResultRaw); + const freshResult = Buffer.from(freshEnvelope.result, "base64").toString("utf8"); + assert(freshResult === "42", `unexpected fresh result: ${freshResult}`); + assert(addon.__test_isolatePoisoned() === false, "fresh isolate inherited poison"); + addon.destroyEngine(freshHandle); + await withTimeout(addon.cleanup(), "fresh transform isolate cleanup"); + assert(count("__test_teardownCallCount") === teardownBefore + 1n, "fresh transform isolate was not torn down"); + assert(count("__test_abandonedIsolateCount") === abandonedBefore + 1n, "fresh cleanup changed abandon count"); + + process.stdout.write(`${JSON.stringify({ + transformResult, + forcedFailures: Number(count("__test_forcedDetachFailureCount") - forcedBefore), + abandoned: Number(count("__test_abandonedIsolateCount") - abandonedBefore), + freshResult, + })}\n`); +} + +main().catch((error) => { + addon.__test_releaseAsyncOp?.(); + process.stderr.write(`${error.stack ?? error}\n`); + process.exitCode = 99; +}); diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index 9d0b7fc2..f14e6362 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -26,8 +26,9 @@ export default defineConfig({ testTimeout: 30000, // Opt the integration lane into the addon's test-only entrypoints // (__test_forceStrandOnce / __test_strandedCount / - // __test_resolverRefDeleteCount). Set before any integration worker - // loads the addon, so its Init() getenv() sees it and registers them; + // __test_resolverRefDeleteCount and detach-poison fault/counter hooks, + // including __test_forcedDetachFailureCount). Set before any + // integration worker loads the addon, so its Init() getenv() sees it; // inert in every other lane and in production. Workers spawned by a // test inherit this env, so the addon Init() in a worker sees it too. env: { DATAWEAVE_TEST_HOOKS: "1" }, From d29168ec5c1699e843df21f8eea9ab337d50ed75 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 11:43:43 -0300 Subject: [PATCH 40/48] fix(node): isolate stale engine generations --- native-lib/node/src/addon.c | 293 ++++++++++++------ .../integration/detach-poison-hook.test.ts | 24 +- .../fixtures/detach-poison-sync.cjs | 106 ++++++- native-lib/node/vitest.config.ts | 3 +- 4 files changed, 317 insertions(+), 109 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 4605874f..fdfdb005 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -5,6 +5,7 @@ #include #include #include +#include #ifndef _WIN32 #include #endif @@ -132,7 +133,11 @@ typedef struct resolver_result_node { // streamed/transform custom-module lookup arriving on the background uv_thread // — and fail closed (return "not found") instead of crashing. typedef struct engine_bridge { + // Public handles stay unique for the process lifetime. native_handle is + // isolate-local and may be reused after an abandoned isolate is replaced. long long handle; + long long native_handle; + uint64_t isolate_generation; napi_env env; napi_ref resolver_js; // NULL => resolver-less engine (no bridge created) uv_thread_t owner; // JS thread that created and must run this engine @@ -164,6 +169,8 @@ typedef struct engine_bridge { struct engine_bridge* next; } engine_bridge_t; static engine_bridge_t* g_bridges = NULL; // linked list, guarded by g_mutex +static uint64_t g_isolate_generation = 0; // incremented for each fresh isolate +static long long g_next_engine_handle = 1; // process-unique public handle // Round-15 (svacas P1): bridges whose engine destroy was SKIPPED because // fn_attach_thread failed while the isolate was STILL LIVE. Such a bridge must @@ -201,6 +208,7 @@ static long long g_test_resolver_ref_deletes = 0; static bool g_test_hold_next_async_op = false; static bool g_test_async_op_held = false; static bool g_test_release_async_op = false; +static bool g_test_fail_next_engine_record_allocation = false; typedef enum { OUTPUT_SETTLEMENT_FAULT_NONE = 0, @@ -471,14 +479,33 @@ static void resolver_results_free_all(engine_bridge_t* b) { b->results = NULL; } -// Call under g_mutex. -static engine_bridge_t* bridge_find(long long handle) { +// Call under g_mutex. Stale bridges stay owned by their env cleanup hooks, but +// only records from the currently published isolate may admit operations. +static engine_bridge_t* bridge_find_current(long long handle) { + for (engine_bridge_t* b = g_bridges; b != NULL; b = b->next) { + if (b->handle == handle && b->isolate_generation == g_isolate_generation) return b; + } + return NULL; +} + +// Destruction may reclaim a stale bridge, provided finalization does not touch +// the replacement isolate. Call under g_mutex. +static engine_bridge_t* bridge_find_any(long long handle) { for (engine_bridge_t* b = g_bridges; b != NULL; b = b->next) { if (b->handle == handle) return b; } return NULL; } +// Allocate the public handle while g_mutex is held. A zero result means the +// signed handle space is exhausted. +static long long next_engine_handle_locked(void) { + if (g_next_engine_handle <= 0) return 0; + long long handle = g_next_engine_handle; + g_next_engine_handle = handle == LLONG_MAX ? 0 : handle + 1; + return handle; +} + // Round-15 (svacas P1): retain a bridge whose engine destroy was skipped while // the isolate was still live (see g_stranded_bridges). The ctx word Java holds // stays valid until a later drain retries the destroy and frees it. Takes @@ -493,7 +520,7 @@ static engine_bridge_t* bridge_find(long long handle) { // both of those already require in_flight == 0 to have run at all (see the // deferred-destroy comment above bridge_finalize_registry) -- so in_flight is // already drained to zero by construction before a bridge is ever stranded, and -// bridge_find() can no longer look it up by handle (it's unlinked from +// bridge_find_current() can no longer look it up by handle (it's unlinked from // g_bridges), so no new op can be admitted against it. The only way a // drained-then-freed stranded bridge could still be dereferenced is unsupported // cross-Worker handle sharing or other API misuse that starts a background @@ -589,6 +616,15 @@ static int env_init_refs_total_locked(void) { // the bridge (bridge_retain_stranded) and retry later (round-15, svacas P1). static bool bridge_finalize_registry(engine_bridge_t* b) { if (b == NULL || fn_destroy_engine == NULL) return true; + uv_mutex_lock(&g_mutex); + bool stale_generation = g_isolate == NULL || + b->isolate_generation != g_isolate_generation; + uv_mutex_unlock(&g_mutex); + if (stale_generation) { + // The bridge's Java registry died with its old isolate. Never attach to + // a replacement isolate, where native_handle may identify a new engine. + return true; + } // Test-only: force ONE live-isolate strand (simulate fn_attach_thread failing // while the isolate is live -> destroy SKIPPED). Inert unless a test both // enabled the hooks (DATAWEAVE_TEST_HOOKS) and armed it via @@ -611,7 +647,8 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { // publishes TEARING_DOWN (and Case 4 holds g_mutex across its g_active_ops==0 // check + teardown) under this same lock, this check plus the increment below // cannot be split by a teardown. - if (g_teardown_state == TEARDOWN_TEARING_DOWN || g_isolate == NULL) { + if (g_teardown_state == TEARDOWN_TEARING_DOWN || g_isolate == NULL || + b->isolate_generation != g_isolate_generation) { uv_mutex_unlock(&g_mutex); return true; } @@ -621,7 +658,7 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { void* thread = NULL; bool destroyed = false; if (fn_attach_thread(g_isolate, &thread) == 0 && thread != NULL) { - fn_destroy_engine(thread, b->handle); + fn_destroy_engine(thread, b->native_handle); detach_thread_checked(DETACH_SITE_BRIDGE_FINALIZE, thread); destroyed = true; // registry entry removed -> resolver ctx is now dead } @@ -824,7 +861,7 @@ static void bridge_env_cleanup(void* arg) { // record, or NULL for an unknown handle (nothing to pin; the worker/native call // surfaces "Unknown engine handle"). Caller MUST hold g_mutex. static engine_bridge_t* bridge_begin_op_locked(long long handle) { - engine_bridge_t* b = bridge_find(handle); + engine_bridge_t* b = bridge_find_current(handle); if (b != NULL) b->in_flight++; return b; } @@ -946,6 +983,7 @@ static void init_thread_fn(void* arg) { return; } + g_isolate_generation++; if (g_test_hooks) g_test_isolate_creations++; // review #21 #1: a brand-new isolate starts un-poisoned. Any poison flag left @@ -2537,7 +2575,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i napi_throw_error(env, NULL, "OOM"); return NULL; } - w->handle = (long long)handle64; + w->handle = pinned != NULL ? pinned->native_handle : 0; w->script = malloc(script_len + 1); w->inputs_json = malloc(inputs_len + 1); if (w->script == NULL || w->inputs_json == NULL) { @@ -3144,7 +3182,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i return NULL; } size_t len; - w->handle = (long long)handle64; + w->handle = pinned != NULL ? pinned->native_handle : 0; #define TRANSFORM_FAIL(msg) do { \ bridge_end_op(pinned, /*env_still_alive=*/true); \ @@ -3544,7 +3582,16 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { // owner is recorded for symmetry but is NOT used to restrict destruction based // on resolver state (see the owner guard in napi_destroy_engine, which now // fires for any record). - engine_bridge_t* rec = (engine_bridge_t*)calloc(1, sizeof(engine_bridge_t)); + bool fail_record_allocation = false; + if (g_test_hooks) { + uv_mutex_lock(&g_mutex); + fail_record_allocation = g_test_fail_next_engine_record_allocation; + g_test_fail_next_engine_record_allocation = false; + uv_mutex_unlock(&g_mutex); + } + engine_bridge_t* rec = fail_record_allocation + ? NULL + : (engine_bridge_t*)calloc(1, sizeof(engine_bridge_t)); if (rec == NULL) { // Roll back the engine we just created so we don't leak a registered but // unrecorded handle. fn_destroy_engine attaches its own thread. @@ -3564,10 +3611,23 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { napi_throw_error(env, NULL, "Failed to allocate engine record"); return NULL; } - rec->handle = handle; + rec->native_handle = handle; rec->owner = uv_thread_self(); rec->env = env; - uv_mutex_lock(&g_mutex); rec->next = g_bridges; g_bridges = rec; uv_mutex_unlock(&g_mutex); + uv_mutex_lock(&g_mutex); + rec->handle = next_engine_handle_locked(); + rec->isolate_generation = g_isolate_generation; + if (rec->handle > 0) { + rec->next = g_bridges; + g_bridges = rec; + } + uv_mutex_unlock(&g_mutex); + if (rec->handle <= 0) { + bridge_finalize(rec, /*env_still_alive=*/true, /*do_registry_remove=*/true, /*may_rehook=*/false); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "Engine handle space exhausted"); + return NULL; + } // Round-11 (#1): register an env cleanup hook for EVERY engine, not just // resolver-backed ones. Without it, a Worker that creates a resolver-less // engine and exits without destroyEngine() would strand this record, the Java @@ -3616,7 +3676,7 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { return NULL; } - napi_value out; napi_create_int64(env, (int64_t)handle, &out); + napi_value out; napi_create_int64(env, (int64_t)rec->handle, &out); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); return out; } @@ -3687,8 +3747,21 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i return NULL; } - bridge->handle = handle; - uv_mutex_lock(&g_mutex); bridge->next = g_bridges; g_bridges = bridge; uv_mutex_unlock(&g_mutex); + bridge->native_handle = handle; + uv_mutex_lock(&g_mutex); + bridge->handle = next_engine_handle_locked(); + bridge->isolate_generation = g_isolate_generation; + if (bridge->handle > 0) { + bridge->next = g_bridges; + g_bridges = bridge; + } + uv_mutex_unlock(&g_mutex); + if (bridge->handle <= 0) { + bridge_finalize(bridge, /*env_still_alive=*/true, /*do_registry_remove=*/true, /*may_rehook=*/false); + uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "Engine handle space exhausted"); + return NULL; + } // Register a per-env cleanup hook so THIS Worker/main thread disposes this // bridge's napi_ref on its own thread when its env tears down (F2). napi_cleanup // no longer touches bridge refs. destroyEngine removes this hook before an @@ -3728,7 +3801,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i napi_throw_error(env, NULL, "Failed to register engine cleanup hook"); return NULL; } - napi_value out; napi_create_int64(env, (int64_t)handle, &out); + napi_value out; napi_create_int64(env, (int64_t)bridge->handle, &out); uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); return out; } @@ -3764,7 +3837,7 @@ static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { // ones. bridge_finalize's napi_ref deletion stays resolver-gated // (resolver_js != NULL && env != NULL) -- that part is unchanged. uv_mutex_lock(&g_mutex); - engine_bridge_t* owned = bridge_find(handle); + engine_bridge_t* owned = bridge_find_any(handle); if (owned != NULL) { uv_thread_t self = uv_thread_self(); if (!uv_thread_equal(&self, &owned->owner)) { @@ -3855,7 +3928,11 @@ static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { uv_mutex_unlock(&g_mutex); void* thread = NULL; if (fn_attach_thread(g_isolate, &thread) == 0 && thread != NULL) { - fn_destroy_engine(thread, handle); + // Public handles are addon-local and cannot be passed to + // the isolate registry when no current bridge maps them. + // Preserve the native unknown-destroy call with the ABI's + // guaranteed-invalid zero handle. + fn_destroy_engine(thread, 0); detach_thread_checked(DETACH_SITE_UNKNOWN_DESTROY, thread); } // Verbatim g_active_ops release pattern. @@ -3952,7 +4029,8 @@ static napi_value napi_run_script_engine(napi_env env, napi_callback_info info) return NULL; } - char* result = (char*)fn_run_script_engine(thread, handle, script, inputs); + long long native_handle = bridge != NULL ? bridge->native_handle : 0; + char* result = (char*)fn_run_script_engine(thread, native_handle, script, inputs); // The pin taken at admission kept this record alive across the run, so no // second lookup is needed. resolver_results_free_all is a no-op for a @@ -4693,15 +4771,18 @@ static napi_value napi_test_resolver_ref_delete_count(napi_env env, napi_callbac return out; } -static detach_site_t detach_site_from_name(const char* name) { - if (strcmp(name, "bridge-finalize") == 0) return DETACH_SITE_BRIDGE_FINALIZE; - if (strcmp(name, "stream-worker") == 0) return DETACH_SITE_STREAM_WORKER; - if (strcmp(name, "transform-worker") == 0) return DETACH_SITE_TRANSFORM_WORKER; - if (strcmp(name, "create-engine") == 0) return DETACH_SITE_CREATE_ENGINE; - if (strcmp(name, "create-rollback") == 0) return DETACH_SITE_CREATE_ROLLBACK; - if (strcmp(name, "resolver-create") == 0) return DETACH_SITE_RESOLVER_CREATE; - if (strcmp(name, "unknown-destroy") == 0) return DETACH_SITE_UNKNOWN_DESTROY; - if (strcmp(name, "synchronous-run") == 0) return DETACH_SITE_SYNCHRONOUS_RUN; +static detach_site_t detach_site_from_name(const char* name, size_t length) { + #define DETACH_SITE_MATCH(value, site) \ + if (length == sizeof(value) - 1 && memcmp(name, value, sizeof(value) - 1) == 0) return site + DETACH_SITE_MATCH("bridge-finalize", DETACH_SITE_BRIDGE_FINALIZE); + DETACH_SITE_MATCH("stream-worker", DETACH_SITE_STREAM_WORKER); + DETACH_SITE_MATCH("transform-worker", DETACH_SITE_TRANSFORM_WORKER); + DETACH_SITE_MATCH("create-engine", DETACH_SITE_CREATE_ENGINE); + DETACH_SITE_MATCH("create-rollback", DETACH_SITE_CREATE_ROLLBACK); + DETACH_SITE_MATCH("resolver-create", DETACH_SITE_RESOLVER_CREATE); + DETACH_SITE_MATCH("unknown-destroy", DETACH_SITE_UNKNOWN_DESTROY); + DETACH_SITE_MATCH("synchronous-run", DETACH_SITE_SYNCHRONOUS_RUN); + #undef DETACH_SITE_MATCH return DETACH_SITE_NONE; } @@ -4711,15 +4792,23 @@ static napi_value napi_test_force_detach_failure_once( napi_value argv[1]; napi_valuetype type; char site_name[64]; - size_t length; + size_t length = 0; if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || napi_typeof(env, argv[0], &type) != napi_ok || type != napi_string || + napi_get_value_string_utf8(env, argv[0], NULL, 0, &length) != napi_ok || + length >= sizeof(site_name)) { + napi_throw_type_error(env, NULL, "A detach site string is required"); + return NULL; + } + size_t copied = 0; + if ( napi_get_value_string_utf8( - env, argv[0], site_name, sizeof(site_name), &length) != napi_ok) { + env, argv[0], site_name, sizeof(site_name), &copied) != napi_ok || + copied != length || memchr(site_name, '\0', length) != NULL) { napi_throw_type_error(env, NULL, "A detach site string is required"); return NULL; } - detach_site_t site = detach_site_from_name(site_name); + detach_site_t site = detach_site_from_name(site_name, length); if (site == DETACH_SITE_NONE) { napi_throw_range_error(env, NULL, "Unknown detach site"); return NULL; @@ -4747,7 +4836,10 @@ static napi_value napi_test_isolate_poisoned(napi_env env, napi_callback_info in static napi_value test_uint64_counter(napi_env env, uint64_t value) { napi_value out; - napi_create_bigint_uint64(env, value, &out); + if (napi_create_bigint_uint64(env, value, &out) != napi_ok) { + napi_throw_error(env, NULL, "Failed to create test counter"); + return NULL; + } return out; } @@ -4818,6 +4910,20 @@ static napi_value napi_test_release_async_op(napi_env env, napi_callback_info in return NULL; } +static napi_value napi_test_fail_next_engine_record_allocation( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + if (g_test_fail_next_engine_record_allocation) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "An engine record allocation failure is already armed"); + return NULL; + } + g_test_fail_next_engine_record_allocation = true; + uv_mutex_unlock(&g_mutex); + return NULL; +} + typedef struct foreign_wrapped_value { uint64_t marker; } foreign_wrapped_value_t; @@ -4946,13 +5052,16 @@ static napi_value napi_test_held_output_delivery( napi_value out; napi_value value; - napi_create_object(env, &out); - napi_get_boolean(env, held, &value); - napi_set_named_property(env, out, "held", value); - napi_create_bigint_uint64(env, sequence, &value); - napi_set_named_property(env, out, "sequence", value); - napi_create_double(env, (double)bytes, &value); - napi_set_named_property(env, out, "bytes", value); + if (napi_create_object(env, &out) != napi_ok || + napi_get_boolean(env, held, &value) != napi_ok || + napi_set_named_property(env, out, "held", value) != napi_ok || + napi_create_bigint_uint64(env, sequence, &value) != napi_ok || + napi_set_named_property(env, out, "sequence", value) != napi_ok || + napi_create_double(env, (double)bytes, &value) != napi_ok || + napi_set_named_property(env, out, "bytes", value) != napi_ok) { + napi_throw_error(env, NULL, "Failed to create held output delivery state"); + return NULL; + } return out; } @@ -5054,6 +5163,14 @@ static napi_value napi_test_output_operation_id(napi_env env, napi_callback_info return out; } +static bool export_function(napi_env env, napi_value exports, const char* name, + napi_callback callback) { + napi_value fn; + return napi_create_function( + env, name, NAPI_AUTO_LENGTH, callback, NULL, &fn) == napi_ok && + napi_set_named_property(env, exports, name, fn) == napi_ok; +} + static napi_value Init(napi_env env, napi_value exports) { uv_once(&g_mutex_once, init_g_mutex); @@ -5066,31 +5183,17 @@ static napi_value Init(napi_env env, napi_value exports) { return NULL; } - napi_value fn; - - napi_create_function(env, "initialize", NAPI_AUTO_LENGTH, napi_initialize, NULL, &fn); - napi_set_named_property(env, exports, "initialize", fn); - - napi_create_function(env, "createEngine", NAPI_AUTO_LENGTH, napi_create_engine, NULL, &fn); - napi_set_named_property(env, exports, "createEngine", fn); - - napi_create_function(env, "createEngineWithResolver", NAPI_AUTO_LENGTH, napi_create_engine_with_resolver, NULL, &fn); - napi_set_named_property(env, exports, "createEngineWithResolver", fn); - - napi_create_function(env, "destroyEngine", NAPI_AUTO_LENGTH, napi_destroy_engine, NULL, &fn); - napi_set_named_property(env, exports, "destroyEngine", fn); - - napi_create_function(env, "runScriptEngine", NAPI_AUTO_LENGTH, napi_run_script_engine, NULL, &fn); - napi_set_named_property(env, exports, "runScriptEngine", fn); - - napi_create_function(env, "runScriptStreamingEngine", NAPI_AUTO_LENGTH, napi_run_script_streaming_engine, NULL, &fn); - napi_set_named_property(env, exports, "runScriptStreamingEngine", fn); - - napi_create_function(env, "runScriptTransformEngine", NAPI_AUTO_LENGTH, napi_run_script_transform_engine, NULL, &fn); - napi_set_named_property(env, exports, "runScriptTransformEngine", fn); - - napi_create_function(env, "cleanup", NAPI_AUTO_LENGTH, napi_cleanup, NULL, &fn); - napi_set_named_property(env, exports, "cleanup", fn); + if (!export_function(env, exports, "initialize", napi_initialize) || + !export_function(env, exports, "createEngine", napi_create_engine) || + !export_function(env, exports, "createEngineWithResolver", napi_create_engine_with_resolver) || + !export_function(env, exports, "destroyEngine", napi_destroy_engine) || + !export_function(env, exports, "runScriptEngine", napi_run_script_engine) || + !export_function(env, exports, "runScriptStreamingEngine", napi_run_script_streaming_engine) || + !export_function(env, exports, "runScriptTransformEngine", napi_run_script_transform_engine) || + !export_function(env, exports, "cleanup", napi_cleanup)) { + napi_throw_error(env, NULL, "Failed to register DataWeave native exports"); + return NULL; + } // Test-only entrypoints, registered only when the process opts in via // DATAWEAVE_TEST_HOOKS (non-empty). getenv() is safe here: Init runs once per @@ -5099,46 +5202,30 @@ static napi_value Init(napi_env env, napi_value exports) { const char* test_hooks = getenv("DATAWEAVE_TEST_HOOKS"); if (test_hooks != NULL && test_hooks[0] != '\0') { g_test_hooks = true; - napi_create_function(env, "__test_forceStrandOnce", NAPI_AUTO_LENGTH, napi_test_force_strand_once, NULL, &fn); - napi_set_named_property(env, exports, "__test_forceStrandOnce", fn); - napi_create_function(env, "__test_strandedCount", NAPI_AUTO_LENGTH, napi_test_stranded_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_strandedCount", fn); - napi_create_function(env, "__test_resolverRefDeleteCount", NAPI_AUTO_LENGTH, napi_test_resolver_ref_delete_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_resolverRefDeleteCount", fn); - napi_create_function(env, "__test_forceDetachFailureOnce", NAPI_AUTO_LENGTH, napi_test_force_detach_failure_once, NULL, &fn); - napi_set_named_property(env, exports, "__test_forceDetachFailureOnce", fn); - napi_create_function(env, "__test_isolatePoisoned", NAPI_AUTO_LENGTH, napi_test_isolate_poisoned, NULL, &fn); - napi_set_named_property(env, exports, "__test_isolatePoisoned", fn); - napi_create_function(env, "__test_isolateCreationCount", NAPI_AUTO_LENGTH, napi_test_isolate_creation_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_isolateCreationCount", fn); - napi_create_function(env, "__test_teardownCallCount", NAPI_AUTO_LENGTH, napi_test_teardown_call_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_teardownCallCount", fn); - napi_create_function(env, "__test_abandonedIsolateCount", NAPI_AUTO_LENGTH, napi_test_abandoned_isolate_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_abandonedIsolateCount", fn); - napi_create_function(env, "__test_forcedDetachFailureCount", NAPI_AUTO_LENGTH, napi_test_forced_detach_failure_count, NULL, &fn); - napi_set_named_property(env, exports, "__test_forcedDetachFailureCount", fn); - napi_create_function(env, "__test_holdNextAsyncOp", NAPI_AUTO_LENGTH, napi_test_hold_next_async_op, NULL, &fn); - napi_set_named_property(env, exports, "__test_holdNextAsyncOp", fn); - napi_create_function(env, "__test_asyncOpHeld", NAPI_AUTO_LENGTH, napi_test_async_op_held, NULL, &fn); - napi_set_named_property(env, exports, "__test_asyncOpHeld", fn); - napi_create_function(env, "__test_releaseAsyncOp", NAPI_AUTO_LENGTH, napi_test_release_async_op, NULL, &fn); - napi_set_named_property(env, exports, "__test_releaseAsyncOp", fn); - napi_create_function(env, "__test_outputStats", NAPI_AUTO_LENGTH, napi_test_output_stats, NULL, &fn); - napi_set_named_property(env, exports, "__test_outputStats", fn); - napi_create_function(env, "__test_outputOperationId", NAPI_AUTO_LENGTH, napi_test_output_operation_id, NULL, &fn); - napi_set_named_property(env, exports, "__test_outputOperationId", fn); - napi_create_function(env, "__test_createForeignWrappedObject", NAPI_AUTO_LENGTH, napi_test_create_foreign_wrapped_object, NULL, &fn); - napi_set_named_property(env, exports, "__test_createForeignWrappedObject", fn); - napi_create_function(env, "__test_failNextOutputSettlement", NAPI_AUTO_LENGTH, napi_test_fail_next_output_settlement, NULL, &fn); - napi_set_named_property(env, exports, "__test_failNextOutputSettlement", fn); - napi_create_function(env, "__test_failNextOutputExceptionClear", NAPI_AUTO_LENGTH, napi_test_fail_next_output_exception_clear, NULL, &fn); - napi_set_named_property(env, exports, "__test_failNextOutputExceptionClear", fn); - napi_create_function(env, "__test_holdNextOutputDelivery", NAPI_AUTO_LENGTH, napi_test_hold_next_output_delivery, NULL, &fn); - napi_set_named_property(env, exports, "__test_holdNextOutputDelivery", fn); - napi_create_function(env, "__test_heldOutputDelivery", NAPI_AUTO_LENGTH, napi_test_held_output_delivery, NULL, &fn); - napi_set_named_property(env, exports, "__test_heldOutputDelivery", fn); - napi_create_function(env, "__test_releaseOutputDelivery", NAPI_AUTO_LENGTH, napi_test_release_output_delivery, NULL, &fn); - napi_set_named_property(env, exports, "__test_releaseOutputDelivery", fn); + if (!export_function(env, exports, "__test_forceStrandOnce", napi_test_force_strand_once) || + !export_function(env, exports, "__test_strandedCount", napi_test_stranded_count) || + !export_function(env, exports, "__test_resolverRefDeleteCount", napi_test_resolver_ref_delete_count) || + !export_function(env, exports, "__test_forceDetachFailureOnce", napi_test_force_detach_failure_once) || + !export_function(env, exports, "__test_isolatePoisoned", napi_test_isolate_poisoned) || + !export_function(env, exports, "__test_isolateCreationCount", napi_test_isolate_creation_count) || + !export_function(env, exports, "__test_teardownCallCount", napi_test_teardown_call_count) || + !export_function(env, exports, "__test_abandonedIsolateCount", napi_test_abandoned_isolate_count) || + !export_function(env, exports, "__test_forcedDetachFailureCount", napi_test_forced_detach_failure_count) || + !export_function(env, exports, "__test_failNextEngineRecordAllocation", napi_test_fail_next_engine_record_allocation) || + !export_function(env, exports, "__test_holdNextAsyncOp", napi_test_hold_next_async_op) || + !export_function(env, exports, "__test_asyncOpHeld", napi_test_async_op_held) || + !export_function(env, exports, "__test_releaseAsyncOp", napi_test_release_async_op) || + !export_function(env, exports, "__test_outputStats", napi_test_output_stats) || + !export_function(env, exports, "__test_outputOperationId", napi_test_output_operation_id) || + !export_function(env, exports, "__test_createForeignWrappedObject", napi_test_create_foreign_wrapped_object) || + !export_function(env, exports, "__test_failNextOutputSettlement", napi_test_fail_next_output_settlement) || + !export_function(env, exports, "__test_failNextOutputExceptionClear", napi_test_fail_next_output_exception_clear) || + !export_function(env, exports, "__test_holdNextOutputDelivery", napi_test_hold_next_output_delivery) || + !export_function(env, exports, "__test_heldOutputDelivery", napi_test_held_output_delivery) || + !export_function(env, exports, "__test_releaseOutputDelivery", napi_test_release_output_delivery)) { + napi_throw_error(env, NULL, "Failed to register DataWeave native test exports"); + return NULL; + } } return exports; diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts index 16f94484..6123569f 100644 --- a/native-lib/node/tests/integration/detach-poison-hook.test.ts +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -61,6 +61,20 @@ describe("detach failure poisoning and recovery", () => { }); }); + it("prevents an old-generation handle from destroying a fresh engine with the same native handle", () => { + expect(runFixture(SYNC_FIXTURE, ["stale-handle"])).toMatchObject({ + handlesDiffer: true, + freshResult: "42", + }); + }); + + it("prevents an old-generation env finalizer from destroying a fresh engine", () => { + expect(runFixture(SYNC_FIXTURE, ["stale-finalizer"])).toMatchObject({ + handlesDiffer: true, + freshResult: "42", + }); + }); + it("poisons a transform worker during an active final cleanup without hanging or tearing down the old isolate", () => { expect(runFixture(TRANSFORM_FIXTURE, [])).toMatchObject({ transformResult: "detach poison transform", @@ -99,9 +113,17 @@ describe("detach failure poisoning and recovery", () => { }); }); + it("executes and recovers from the create-rollback detach site", () => { + expect(runFixture(SYNC_FIXTURE, ["exercise-create-rollback"])).toMatchObject({ + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + it("rejects invalid sites and refuses to silently replace an armed failure", () => { expect(runFixture(SYNC_FIXTURE, ["invalid-arguments"])).toEqual({ - invalidArguments: 4, + invalidArguments: 6, duplicateArmRejected: true, }); }); diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs index c961f65f..5d2ea35a 100644 --- a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -1,6 +1,8 @@ "use strict"; const path = require("node:path"); +const { once } = require("node:events"); +const { Worker } = require("node:worker_threads"); const addonPath = path.resolve(process.cwd(), process.argv[2]); const libPath = process.argv[3]; @@ -17,6 +19,7 @@ const DETACH_HOOKS = [ "__test_abandonedIsolateCount", "__test_forcedDetachFailureCount", ]; +const TEST_HOOKS = [...DETACH_HOOKS, "__test_failNextEngineRecordAllocation"]; function assert(condition, message) { if (!condition) throw new Error(message); @@ -102,8 +105,8 @@ async function recovery() { reject(() => addon.initialize(libPath)); assert(addon.__test_outputStats().liveFlows === liveFlowsBefore, "rejected async work allocated an output flow"); - // A poisoned generation is abandoned as a unit. destroyEngine is safe but - // intentionally performs no further Graal attachment on it. + // Engine destruction remains permitted after poison. Final cleanup abandons + // the generation without attempting isolate teardown. addon.destroyEngine(otherHandle); addon.destroyEngine(firstHandle); await withTimeout(addon.cleanup(), "poisoned isolate cleanup"); @@ -132,6 +135,63 @@ async function recovery() { }; } +async function staleHandle() { + addon.initialize(libPath); + const oldHandle = addon.createEngine(); + addon.__test_forceDetachFailureOnce("synchronous-run"); + successfulResult(addon.runScriptEngine(oldHandle, "output application/json --- 6 * 7", "{}")); + await withTimeout(addon.cleanup(), "old isolate cleanup"); + + addon.initialize(libPath); + const freshHandle = addon.createEngine(); + addon.destroyEngine(oldHandle); + const freshResult = successfulResult( + addon.runScriptEngine(freshHandle, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(freshHandle); + await withTimeout(addon.cleanup(), "fresh isolate cleanup"); + return { handlesDiffer: oldHandle !== freshHandle, freshResult }; +} + +async function staleFinalizer() { + const worker = new Worker(` + "use strict"; + const { parentPort, workerData } = require("node:worker_threads"); + const addon = require(workerData.addonPath); + function successful(raw) { + const result = JSON.parse(raw); + if (result.success !== true) throw new Error(\`worker run failed: \${raw}\`); + } + (async () => { + addon.initialize(workerData.libPath); + const handle = addon.createEngine(); + addon.__test_forceDetachFailureOnce("synchronous-run"); + successful(addon.runScriptEngine(handle, "output application/json --- 6 * 7", "{}")); + await addon.cleanup(); + parentPort.postMessage({ handle }); + parentPort.once("message", () => parentPort.close()); + })().catch((error) => { + parentPort.postMessage({ error: error.stack || String(error) }); + }); + `, { eval: true, workerData: { addonPath, libPath } }); + + const [message] = await once(worker, "message"); + assert(message.error === undefined, message.error ?? "worker failed"); + addon.initialize(libPath); + const freshHandle = addon.createEngine(); + const exit = once(worker, "exit"); + worker.postMessage("exit"); + const [exitCode] = await exit; + assert(exitCode === 0, `worker exited with ${exitCode}`); + + const freshResult = successfulResult( + addon.runScriptEngine(freshHandle, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(freshHandle); + await withTimeout(addon.cleanup(), "fresh isolate cleanup after stale finalizer"); + return { handlesDiffer: message.handle !== freshHandle, freshResult }; +} + function validateSite(site) { assert(typeof addon.__test_forceDetachFailureOnce === "function", "detach failure hook is undefined"); addon.__test_forceDetachFailureOnce(site); @@ -140,7 +200,14 @@ function validateSite(site) { function invalidArguments() { let invalidArguments = 0; - for (const value of [undefined, null, 42, "not-a-detach-site"]) { + for (const value of [ + undefined, + null, + 42, + "not-a-detach-site", + "synchronous-run\0unknown", + "x".repeat(64), + ]) { try { if (value === undefined) addon.__test_forceDetachFailureOnce(); else addon.__test_forceDetachFailureOnce(value); @@ -160,6 +227,34 @@ function invalidArguments() { return { invalidArguments, duplicateArmRejected }; } +async function exerciseCreateRollback() { + assert( + typeof addon.__test_failNextEngineRecordAllocation === "function", + "engine record allocation hook is undefined" + ); + addon.initialize(libPath); + const forcedBefore = count("__test_forcedDetachFailureCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + addon.__test_forceDetachFailureOnce("create-rollback"); + addon.__test_failNextEngineRecordAllocation(); + expectThrow(() => addon.createEngine(), "Failed to allocate engine record"); + assert(addon.__test_isolatePoisoned() === true, "create rollback did not poison isolate"); + await withTimeout(addon.cleanup(), "create rollback cleanup"); + + addon.initialize(libPath); + const fresh = addon.createEngine(); + const freshResult = successfulResult( + addon.runScriptEngine(fresh, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(fresh); + await withTimeout(addon.cleanup(), "create rollback fresh cleanup"); + return { + forcedFailures: Number(count("__test_forcedDetachFailureCount") - forcedBefore), + abandoned: Number(count("__test_abandonedIsolateCount") - abandonedBefore), + freshResult, + }; +} + async function oneShotSite() { addon.initialize(libPath); const first = addon.createEngine(); @@ -236,16 +331,19 @@ async function exerciseSite(site) { } function hooksAbsent() { - return { hooksAbsent: DETACH_HOOKS.every((hook) => addon[hook] === undefined) }; + return { hooksAbsent: TEST_HOOKS.every((hook) => addon[hook] === undefined) }; } async function main() { let result; if (mode === "recovery") result = await recovery(); + else if (mode === "stale-handle") result = await staleHandle(); + else if (mode === "stale-finalizer") result = await staleFinalizer(); else if (mode === "validate-site") result = validateSite(process.argv[5]); else if (mode === "invalid-arguments") result = invalidArguments(); else if (mode === "one-shot-site") result = await oneShotSite(); else if (mode === "exercise-site") result = await exerciseSite(process.argv[5]); + else if (mode === "exercise-create-rollback") result = await exerciseCreateRollback(); else if (mode === "hooks-absent") result = hooksAbsent(); else throw new Error(`unknown fixture mode: ${mode}`); process.stdout.write(`${JSON.stringify(result)}\n`); diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index f14e6362..0bb95b72 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -27,7 +27,8 @@ export default defineConfig({ // Opt the integration lane into the addon's test-only entrypoints // (__test_forceStrandOnce / __test_strandedCount / // __test_resolverRefDeleteCount and detach-poison fault/counter hooks, - // including __test_forcedDetachFailureCount). Set before any + // including __test_forcedDetachFailureCount and the engine-record + // allocation failure hook). Set before any // integration worker loads the addon, so its Init() getenv() sees it; // inert in every other lane and in production. Workers spawned by a // test inherit this env, so the addon Init() in a worker sees it too. From 8cf1f0994ccdaeae4973c76c43028aea7e306d73 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 12:14:09 -0300 Subject: [PATCH 41/48] fix(node): bound engine identity lifetime --- native-lib/node/src/addon.c | 160 ++++++++++++-- .../integration/detach-poison-hook.test.ts | 55 +++++ .../fixtures/detach-poison-sync.cjs | 195 +++++++++++++++++- native-lib/node/vitest.config.ts | 4 +- 4 files changed, 397 insertions(+), 17 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index fdfdb005..d58f9f5c 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -5,7 +5,6 @@ #include #include #include -#include #ifndef _WIN32 #include #endif @@ -52,6 +51,7 @@ static uv_once_t g_mutex_once = UV_ONCE_INIT; #define CALLBACK_REENTRANCY_CODE "ERR_DATAWEAVE_CALLBACK_REENTRANCY" #define ISOLATE_POISONED_MESSAGE \ "DataWeave isolate is unavailable after a thread detach failure; clean up and initialize again." +#define MAX_SAFE_ENGINE_HANDLE 9007199254740991LL static unsigned native_callback_depth(void) { return (unsigned)(uintptr_t)uv_key_get(&g_native_callback_depth); @@ -208,7 +208,7 @@ static long long g_test_resolver_ref_deletes = 0; static bool g_test_hold_next_async_op = false; static bool g_test_async_op_held = false; static bool g_test_release_async_op = false; -static bool g_test_fail_next_engine_record_allocation = false; +static uint64_t g_test_engine_record_allocation_failure_generation = 0; typedef enum { OUTPUT_SETTLEMENT_FAULT_NONE = 0, @@ -389,6 +389,9 @@ static void test_hold_async_op_if_armed(void) { // (_release_isolate / _retry_pending_teardown_locked). Caller holds g_mutex. static void abandon_unrecoverable_isolate_locked(void) { if (g_test_hooks && g_isolate != NULL) g_test_abandoned_isolates++; + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -497,12 +500,12 @@ static engine_bridge_t* bridge_find_any(long long handle) { return NULL; } -// Allocate the public handle while g_mutex is held. A zero result means the -// signed handle space is exhausted. +// Allocate the public handle while g_mutex is held. A zero result means the JS +// safe-integer handle space is exhausted. static long long next_engine_handle_locked(void) { - if (g_next_engine_handle <= 0) return 0; + if (g_next_engine_handle <= 0 || g_next_engine_handle > MAX_SAFE_ENGINE_HANDLE) return 0; long long handle = g_next_engine_handle; - g_next_engine_handle = handle == LLONG_MAX ? 0 : handle + 1; + g_next_engine_handle = handle == MAX_SAFE_ENGINE_HANDLE ? 0 : handle + 1; return handle; } @@ -606,6 +609,10 @@ static int env_init_refs_total_locked(void) { // section: no teardown path can interleave between "isolate is live" and // "reservation taken". Callable from any thread NOT holding g_mutex. // +// `detach_site` identifies the operation whose successful attach/destroy is +// being detached; normal finalization uses bridge-finalize, while creation +// rollback keeps the distinct create-rollback fault-injection contract. +// // Returns TRUE when the caller may safely free the bridge: the engine was // actually destroyed (registry entry removed), OR the whole isolate is going // away (TEARING_DOWN / g_isolate == NULL) so the Java registry -- and the @@ -614,7 +621,7 @@ static int env_init_refs_total_locked(void) { // live (fn_attach_thread failed): the Java registry still holds this bridge as a // resolver ctx, so freeing it now would be a UAF. The caller must instead retain // the bridge (bridge_retain_stranded) and retry later (round-15, svacas P1). -static bool bridge_finalize_registry(engine_bridge_t* b) { +static bool bridge_finalize_registry_at_site(engine_bridge_t* b, detach_site_t detach_site) { if (b == NULL || fn_destroy_engine == NULL) return true; uv_mutex_lock(&g_mutex); bool stale_generation = g_isolate == NULL || @@ -659,7 +666,7 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { bool destroyed = false; if (fn_attach_thread(g_isolate, &thread) == 0 && thread != NULL) { fn_destroy_engine(thread, b->native_handle); - detach_thread_checked(DETACH_SITE_BRIDGE_FINALIZE, thread); + detach_thread_checked(detach_site, thread); destroyed = true; // registry entry removed -> resolver ctx is now dead } // else: attach failed while the isolate is STILL LIVE -- destroy was skipped, @@ -675,6 +682,10 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { return destroyed; } +static bool bridge_finalize_registry(engine_bridge_t* b) { + return bridge_finalize_registry_at_site(b, DETACH_SITE_BRIDGE_FINALIZE); +} + // Forward declaration: the env cleanup hook. bridge_finalize re-registers/keeps // it on an owner-thread live-isolate strand (may_rehook) and removes it on the // owner-thread free path; the definition is below (after drain_stranded_bridges). @@ -983,6 +994,25 @@ static void init_thread_fn(void* arg) { return; } + if (g_isolate_generation == UINT64_MAX) { + if (g_test_hooks) { + g_test_isolate_creations++; + g_test_teardown_calls++; + } + int td_rc = fn_tear_down_isolate(boot_thread); + g_isolate = NULL; + g_thread = NULL; + if (td_rc != 0) { + fprintf(stderr, + "[DataWeave Node addon] isolate generation space was exhausted and " + "the unpublishable isolate could not be torn down; it is being leaked " + "for the process lifetime.\n"); + } + snprintf(args->error, sizeof(args->error), + "DataWeave isolate generation space exhausted"); + args->result = -4; + return; + } g_isolate_generation++; if (g_test_hooks) g_test_isolate_creations++; @@ -1279,6 +1309,9 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { if (result.outcome == CLEANUP_TORN_DOWN) { // Teardown ran (or there was nothing to tear down) -- clear the globals // so the next initialize() sees a clean slate. g_ref_count is already 0. + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -3585,8 +3618,11 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { bool fail_record_allocation = false; if (g_test_hooks) { uv_mutex_lock(&g_mutex); - fail_record_allocation = g_test_fail_next_engine_record_allocation; - g_test_fail_next_engine_record_allocation = false; + fail_record_allocation = + g_test_engine_record_allocation_failure_generation == g_isolate_generation; + if (fail_record_allocation) { + g_test_engine_record_allocation_failure_generation = 0; + } uv_mutex_unlock(&g_mutex); } engine_bridge_t* rec = fail_record_allocation @@ -3623,7 +3659,15 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { } uv_mutex_unlock(&g_mutex); if (rec->handle <= 0) { - bridge_finalize(rec, /*env_still_alive=*/true, /*do_registry_remove=*/true, /*may_rehook=*/false); + // No public handle or cleanup hook was published. Use the normal + // generation-aware finalizer so a transient attach failure retains the + // native registry record instead of freeing a resolver ctx still held + // by Java. may_rehook=false because no cleanup hook was registered. + if (bridge_finalize_registry_at_site(rec, DETACH_SITE_CREATE_ROLLBACK)) { + bridge_finalize_free(rec, /*env_still_alive=*/true); + } else { + bridge_retain_stranded(rec); + } uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Engine handle space exhausted"); return NULL; @@ -3757,7 +3801,11 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i } uv_mutex_unlock(&g_mutex); if (bridge->handle <= 0) { - bridge_finalize(bridge, /*env_still_alive=*/true, /*do_registry_remove=*/true, /*may_rehook=*/false); + if (bridge_finalize_registry_at_site(bridge, DETACH_SITE_CREATE_ROLLBACK)) { + bridge_finalize_free(bridge, /*env_still_alive=*/true); + } else { + bridge_retain_stranded(bridge); + } uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "Engine handle space exhausted"); return NULL; @@ -4220,6 +4268,9 @@ static void teardown_waiter_thread_fn(void* arg) { uv_mutex_lock(&g_mutex); if (!cancelled && result == CLEANUP_TORN_DOWN) { + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -4371,6 +4422,9 @@ static void retry_stranded_teardown_locked(void) { if (spawn_rc == 0) uv_thread_join(&tid); if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; if (result.outcome == CLEANUP_TORN_DOWN) { + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -4403,6 +4457,9 @@ static void isolate_ref_release_n_locked(int n) { } if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; if (result.outcome == CLEANUP_TORN_DOWN) { + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -4601,6 +4658,9 @@ static napi_value release_isolate_ref_locked(napi_env env) { // attach-failure path). if (g_test_hooks && result.teardown_callable) g_test_teardown_calls++; if (result.outcome == CLEANUP_TORN_DOWN) { + if (g_test_engine_record_allocation_failure_generation == g_isolate_generation) { + g_test_engine_record_allocation_failure_generation = 0; + } g_thread = NULL; g_isolate = NULL; g_initialized = 0; @@ -4914,16 +4974,85 @@ static napi_value napi_test_fail_next_engine_record_allocation( napi_env env, napi_callback_info info) { (void)info; uv_mutex_lock(&g_mutex); - if (g_test_fail_next_engine_record_allocation) { + if (!g_initialized || g_isolate == NULL || g_isolate_poisoned || + g_teardown_state != TEARDOWN_NONE || g_teardown_needed || + g_active_ops != 0 || g_isolate_generation == 0) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "A healthy initialized isolate generation is required"); + return NULL; + } + if (g_test_engine_record_allocation_failure_generation != 0) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, "An engine record allocation failure is already armed"); return NULL; } - g_test_fail_next_engine_record_allocation = true; + g_test_engine_record_allocation_failure_generation = g_isolate_generation; uv_mutex_unlock(&g_mutex); return NULL; } +static napi_value napi_test_set_next_engine_handle( + napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + napi_valuetype type; + double next_handle; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || + napi_typeof(env, argv[0], &type) != napi_ok || type != napi_number || + napi_get_value_double(env, argv[0], &next_handle) != napi_ok || + next_handle <= 0 || next_handle > (double)MAX_SAFE_ENGINE_HANDLE || + next_handle != (double)(long long)next_handle) { + napi_throw_range_error(env, NULL, "Next engine handle must be a positive safe integer"); + return NULL; + } + uv_mutex_lock(&g_mutex); + if (g_bridges != NULL || g_stranded_bridges != NULL || g_active_ops != 0 || + g_next_engine_handle == 0 || next_handle < g_next_engine_handle) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "Next engine handle cannot be changed in the current state"); + return NULL; + } + g_next_engine_handle = (long long)next_handle; + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_set_isolate_generation( + napi_env env, napi_callback_info info) { + size_t argc = 1; + napi_value argv[1]; + uint64_t generation; + bool lossless = false; + if (napi_get_cb_info(env, info, &argc, argv, NULL, NULL) != napi_ok || argc < 1 || + napi_get_value_bigint_uint64(env, argv[0], &generation, &lossless) != napi_ok || + !lossless) { + napi_throw_range_error(env, NULL, "Isolate generation must be a non-decreasing uint64 BigInt"); + return NULL; + } + uv_mutex_lock(&g_mutex); + if (generation < g_isolate_generation || + !g_initialized || g_isolate == NULL || g_isolate_poisoned || + g_teardown_state != TEARDOWN_NONE || g_active_ops != 0 || + g_bridges != NULL || g_stranded_bridges != NULL || + g_test_engine_record_allocation_failure_generation != 0) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "Isolate generation cannot be changed in the current state"); + return NULL; + } + g_isolate_generation = generation; + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_isolate_generation( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t generation = g_isolate_generation; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, generation); +} + typedef struct foreign_wrapped_value { uint64_t marker; } foreign_wrapped_value_t; @@ -5212,6 +5341,9 @@ static napi_value Init(napi_env env, napi_value exports) { !export_function(env, exports, "__test_abandonedIsolateCount", napi_test_abandoned_isolate_count) || !export_function(env, exports, "__test_forcedDetachFailureCount", napi_test_forced_detach_failure_count) || !export_function(env, exports, "__test_failNextEngineRecordAllocation", napi_test_fail_next_engine_record_allocation) || + !export_function(env, exports, "__test_setNextEngineHandle", napi_test_set_next_engine_handle) || + !export_function(env, exports, "__test_setIsolateGeneration", napi_test_set_isolate_generation) || + !export_function(env, exports, "__test_isolateGeneration", napi_test_isolate_generation) || !export_function(env, exports, "__test_holdNextAsyncOp", napi_test_hold_next_async_op) || !export_function(env, exports, "__test_asyncOpHeld", napi_test_async_op_held) || !export_function(env, exports, "__test_releaseAsyncOp", napi_test_release_async_op) || diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts index 6123569f..fb827ec1 100644 --- a/native-lib/node/tests/integration/detach-poison-hook.test.ts +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -121,6 +121,61 @@ describe("detach failure poisoning and recovery", () => { }); }); + it("allocates the final two exact JS handles and rolls back the next native engine", () => { + expect(runFixture(SYNC_FIXTURE, ["handle-exhaustion"])).toEqual({ + firstHandle: Number.MAX_SAFE_INTEGER - 1, + secondHandle: Number.MAX_SAFE_INTEGER, + firstResult: "42", + secondResult: "42", + exhaustionRejected: true, + forcedRollbackDetach: 1, + isolatePoisoned: true, + abandoned: 1, + }); + }); + + it("clears an armed record-allocation fault after normal generation teardown", () => { + expect(runFixture(SYNC_FIXTURE, ["allocation-fault-normal-reset"])).toEqual({ + freshResult: "42", + freshCreateFailed: false, + }); + }); + + it("clears an armed record-allocation fault after poisoned generation abandonment", () => { + expect(runFixture(SYNC_FIXTURE, ["allocation-fault-poison-reset"])).toEqual({ + freshResult: "42", + freshCreateFailed: false, + }); + }); + + it("tears down an unpublishable isolate rather than wrapping generation identity", () => { + expect(runFixture(SYNC_FIXTURE, ["generation-exhaustion"])).toEqual({ + exhaustionRejected: true, + creationDelta: 1, + teardownDelta: 2, + generation: "18446744073709551615", + }); + }); + + it("rejects identity and allocation fault hooks outside their safe mutation states", () => { + expect(runFixture(SYNC_FIXTURE, ["identity-hook-validation"])).toEqual({ + fractionalHandleRejected: true, + nanHandleRejected: true, + armBeforeInitializeRejected: true, + duplicateArmRejected: true, + handleMutationWithBridgeRejected: true, + generationMutationWithBridgeRejected: true, + armAfterPoisonRejected: true, + }); + }); + + it("retains a resolver bridge when handle-exhaustion rollback cannot attach", () => { + expect(runFixture(SYNC_FIXTURE, ["handle-exhaustion-rollback-strand"])).toEqual({ + exhaustionRejected: true, + strandedDelta: 1, + }); + }); + it("rejects invalid sites and refuses to silently replace an armed failure", () => { expect(runFixture(SYNC_FIXTURE, ["invalid-arguments"])).toEqual({ invalidArguments: 6, diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs index 5d2ea35a..df8bae9f 100644 --- a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -19,7 +19,15 @@ const DETACH_HOOKS = [ "__test_abandonedIsolateCount", "__test_forcedDetachFailureCount", ]; -const TEST_HOOKS = [...DETACH_HOOKS, "__test_failNextEngineRecordAllocation"]; +const TEST_HOOKS = [ + ...DETACH_HOOKS, + "__test_failNextEngineRecordAllocation", + "__test_setNextEngineHandle", + "__test_setIsolateGeneration", + "__test_isolateGeneration", +]; +const MAX_SAFE_HANDLE = Number.MAX_SAFE_INTEGER; +const UINT64_MAX = 18_446_744_073_709_551_615n; function assert(condition, message) { if (!condition) throw new Error(message); @@ -255,6 +263,185 @@ async function exerciseCreateRollback() { }; } +async function handleExhaustion() { + assert(typeof addon.__test_setNextEngineHandle === "function", "handle setter is undefined"); + addon.initialize(libPath); + addon.__test_setNextEngineHandle(MAX_SAFE_HANDLE - 1); + const firstHandle = addon.createEngine(); + const secondHandle = addon.createEngineWithResolver(() => null); + assert(firstHandle === MAX_SAFE_HANDLE - 1, `unexpected first handle: ${firstHandle}`); + assert(secondHandle === MAX_SAFE_HANDLE, `unexpected second handle: ${secondHandle}`); + assert(Number.isSafeInteger(firstHandle), "first handle is not a safe integer"); + assert(Number.isSafeInteger(secondHandle), "second handle is not a safe integer"); + const firstResult = successfulResult( + addon.runScriptEngine(firstHandle, "output application/json --- 6 * 7", "{}") + ); + const secondResult = successfulResult( + addon.runScriptEngine(secondHandle, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(firstHandle); + addon.destroyEngine(secondHandle); + + const forcedBefore = count("__test_forcedDetachFailureCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + addon.__test_forceDetachFailureOnce("create-rollback"); + let exhaustionRejected = false; + try { + addon.createEngine(); + } catch (error) { + exhaustionRejected = error instanceof Error && error.message === "Engine handle space exhausted"; + } + assert(exhaustionRejected, "handle exhaustion was not rejected"); + const forcedRollbackDetach = Number(count("__test_forcedDetachFailureCount") - forcedBefore); + const isolatePoisoned = addon.__test_isolatePoisoned(); + await withTimeout(addon.cleanup(), "handle exhaustion cleanup"); + const abandoned = Number(count("__test_abandonedIsolateCount") - abandonedBefore); + return { + firstHandle, + secondHandle, + firstResult, + secondResult, + exhaustionRejected, + forcedRollbackDetach, + isolatePoisoned, + abandoned, + }; +} + +async function allocationFaultReset(poison) { + addon.initialize(libPath); + addon.__test_failNextEngineRecordAllocation(); + if (poison) { + const handle = addon.createEngineWithResolver(() => null); + addon.__test_forceDetachFailureOnce("synchronous-run"); + successfulResult(addon.runScriptEngine(handle, "output application/json --- 6 * 7", "{}")); + } + await withTimeout(addon.cleanup(), poison ? "poisoned fault reset" : "normal fault reset"); + + addon.initialize(libPath); + let freshCreateFailed = false; + let fresh; + try { + fresh = addon.createEngine(); + } catch (error) { + freshCreateFailed = error instanceof Error && error.message === "Failed to allocate engine record"; + if (!freshCreateFailed) throw error; + } + assert(!freshCreateFailed, "fresh generation consumed the stale allocation fault"); + const freshResult = successfulResult( + addon.runScriptEngine(fresh, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(fresh); + await withTimeout(addon.cleanup(), "fresh allocation fault reset cleanup"); + return { freshResult, freshCreateFailed }; +} + +async function generationExhaustion() { + assert(typeof addon.__test_setIsolateGeneration === "function", "generation setter is undefined"); + assert(typeof addon.__test_isolateGeneration === "function", "generation counter is undefined"); + addon.initialize(libPath); + const creationBefore = count("__test_isolateCreationCount"); + const teardownBefore = count("__test_teardownCallCount"); + addon.__test_setIsolateGeneration(UINT64_MAX); + await withTimeout(addon.cleanup(), "generation max cleanup"); + + let exhaustionRejected = false; + try { + addon.initialize(libPath); + } catch (error) { + exhaustionRejected = error instanceof Error && error.message === "DataWeave isolate generation space exhausted"; + } + assert(exhaustionRejected, "generation exhaustion was not rejected"); + return { + exhaustionRejected, + creationDelta: Number(count("__test_isolateCreationCount") - creationBefore), + teardownDelta: Number(count("__test_teardownCallCount") - teardownBefore), + generation: addon.__test_isolateGeneration().toString(), + }; +} + +async function identityHookValidation() { + let fractionalHandleRejected = false; + try { + addon.__test_setNextEngineHandle(1.5); + } catch (error) { + fractionalHandleRejected = error instanceof Error; + } + let nanHandleRejected = false; + try { + addon.__test_setNextEngineHandle(Number.NaN); + } catch (error) { + nanHandleRejected = error instanceof Error; + } + let armBeforeInitializeRejected = false; + try { + addon.__test_failNextEngineRecordAllocation(); + } catch (error) { + armBeforeInitializeRejected = error instanceof Error; + } + addon.initialize(libPath); + addon.__test_failNextEngineRecordAllocation(); + let duplicateArmRejected = false; + try { + addon.__test_failNextEngineRecordAllocation(); + } catch (error) { + duplicateArmRejected = error instanceof Error; + } + expectThrow(() => addon.createEngine(), "Failed to allocate engine record"); + + const handle = addon.createEngine(); + let handleMutationWithBridgeRejected = false; + try { + addon.__test_setNextEngineHandle(MAX_SAFE_HANDLE - 1); + } catch (error) { + handleMutationWithBridgeRejected = error instanceof Error; + } + let generationMutationWithBridgeRejected = false; + try { + addon.__test_setIsolateGeneration(UINT64_MAX); + } catch (error) { + generationMutationWithBridgeRejected = error instanceof Error; + } + addon.__test_forceDetachFailureOnce("synchronous-run"); + successfulResult(addon.runScriptEngine(handle, "output application/json --- 6 * 7", "{}")); + let armAfterPoisonRejected = false; + try { + addon.__test_failNextEngineRecordAllocation(); + } catch (error) { + armAfterPoisonRejected = error instanceof Error; + } + await withTimeout(addon.cleanup(), "identity hook validation cleanup"); + return { + fractionalHandleRejected, + nanHandleRejected, + armBeforeInitializeRejected, + duplicateArmRejected, + handleMutationWithBridgeRejected, + generationMutationWithBridgeRejected, + armAfterPoisonRejected, + }; +} + +async function handleExhaustionRollbackStrand() { + addon.initialize(libPath); + addon.__test_setNextEngineHandle(MAX_SAFE_HANDLE); + const finalHandle = addon.createEngine(); + addon.destroyEngine(finalHandle); + + const strandedBefore = addon.__test_strandedCount(); + addon.__test_forceStrandOnce(); + let exhaustionRejected = false; + try { + addon.createEngineWithResolver(() => null); + } catch (error) { + exhaustionRejected = error instanceof Error && error.message === "Engine handle space exhausted"; + } + assert(exhaustionRejected, "handle exhaustion was not rejected"); + const strandedDelta = addon.__test_strandedCount() - strandedBefore; + await withTimeout(addon.cleanup(), "handle exhaustion rollback strand cleanup"); + return { exhaustionRejected, strandedDelta }; +} + async function oneShotSite() { addon.initialize(libPath); const first = addon.createEngine(); @@ -344,6 +531,12 @@ async function main() { else if (mode === "one-shot-site") result = await oneShotSite(); else if (mode === "exercise-site") result = await exerciseSite(process.argv[5]); else if (mode === "exercise-create-rollback") result = await exerciseCreateRollback(); + else if (mode === "handle-exhaustion") result = await handleExhaustion(); + else if (mode === "allocation-fault-normal-reset") result = await allocationFaultReset(false); + else if (mode === "allocation-fault-poison-reset") result = await allocationFaultReset(true); + else if (mode === "generation-exhaustion") result = await generationExhaustion(); + else if (mode === "identity-hook-validation") result = await identityHookValidation(); + else if (mode === "handle-exhaustion-rollback-strand") result = await handleExhaustionRollbackStrand(); else if (mode === "hooks-absent") result = hooksAbsent(); else throw new Error(`unknown fixture mode: ${mode}`); process.stdout.write(`${JSON.stringify(result)}\n`); diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index 0bb95b72..7ac00f44 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -27,8 +27,8 @@ export default defineConfig({ // Opt the integration lane into the addon's test-only entrypoints // (__test_forceStrandOnce / __test_strandedCount / // __test_resolverRefDeleteCount and detach-poison fault/counter hooks, - // including __test_forcedDetachFailureCount and the engine-record - // allocation failure hook). Set before any + // including __test_forcedDetachFailureCount, identity-boundary hooks, + // and the engine-record allocation failure hook). Set before any // integration worker loads the addon, so its Init() getenv() sees it; // inert in every other lane and in production. Workers spawned by a // test inherit this env, so the addon Init() in a worker sees it too. From 263bcd9e15b01c7ba02e8188d97560b33afdbc52 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 14:26:58 -0300 Subject: [PATCH 42/48] fix(node): publish detach failures before admission --- native-lib/node/src/addon.c | 329 ++++++++++++++++-- .../integration/detach-poison-hook.test.ts | 30 ++ .../fixtures/detach-poison-sync.cjs | 128 +++++++ native-lib/node/vitest.config.ts | 4 +- 4 files changed, 466 insertions(+), 25 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index d58f9f5c..46517e85 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -166,6 +166,18 @@ typedef struct engine_bridge { // owner thread (creation, destroyEngine, bridge_env_cleanup) under the usual // owner-thread-serialization contract. bool hook_registered; + // Resolver refs and bridge memory have two independent owners once native + // registry removal strands: the Java callback ctx and the owner Node env. + // The env hook drops owner_alive/resolver_js; the native drain drops + // native_alive. The bridge is freed only after both have released it. + bool native_alive; + bool owner_alive; + // Every resolver bridge owns an unreferenced TSFN. A native-side drain can + // queue its callback onto the owner env to delete resolver_js legally; the + // TSFN finalizer drops owner ownership when that env dies instead. + napi_threadsafe_function owner_cleanup_tsfn; + bool owner_cleanup_queued; + bool owner_cleanup_released; struct engine_bridge* next; } engine_bridge_t; static engine_bridge_t* g_bridges = NULL; // linked list, guarded by g_mutex @@ -209,6 +221,16 @@ static bool g_test_hold_next_async_op = false; static bool g_test_async_op_held = false; static bool g_test_release_async_op = false; static uint64_t g_test_engine_record_allocation_failure_generation = 0; +static uint64_t g_detach_in_progress_generation = 0; +static unsigned g_detach_in_progress = 0; +static bool g_test_hold_next_detach_publication = false; +static bool g_test_detach_publication_held = false; +static bool g_test_release_detach_publication = false; +static unsigned g_detach_publication_waiters = 0; +static uint64_t g_test_live_resolver_refs = 0; + +static void bridge_env_cleanup(void* arg); +static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive); typedef enum { OUTPUT_SETTLEMENT_FAULT_NONE = 0, @@ -398,6 +420,8 @@ static void abandon_unrecoverable_isolate_locked(void) { g_ref_count = 0; g_teardown_needed = false; g_isolate_poisoned = false; + g_detach_in_progress = 0; + g_detach_in_progress_generation = 0; fprintf(stderr, "[DataWeave Node addon] GraalVM isolate teardown AND worker detach both " "failed; the isolate can never be torn down and is being leaked for the " @@ -424,21 +448,68 @@ static void poison_isolate_detach_failure_locked(int detach_rc) { g_isolate_poisoned = true; } +// Wait while this generation has an ordinary detach whose result has not yet +// been published. Callers hold g_mutex. Each detach is covered by an active-op +// reservation, so waiting cannot let teardown overtake the detaching thread. +static void wait_for_detach_publication_locked(void) { + bool counted = false; + while (g_detach_in_progress > 0 && + g_detach_in_progress_generation == g_isolate_generation) { + if (g_test_hooks && !counted) { + g_detach_publication_waiters++; + counted = true; + uv_cond_broadcast(&g_teardown_cond); + } + uv_cond_wait(&g_teardown_cond, &g_mutex); + } + if (counted) { + g_detach_publication_waiters--; + uv_cond_broadcast(&g_teardown_cond); + } +} + // Centralized policy for every ordinary operation detach. The real Graal // detach always runs first. Test injection may turn only a successful detach // at the selected site into one synthetic failure; real failures never consume // the arm. Any nonzero result poisons this published isolate before later work // can be admitted. static int detach_thread_checked(detach_site_t site, void* thread) { + uv_mutex_lock(&g_mutex); + uint64_t generation = g_isolate_generation; + if (g_detach_in_progress == 0) { + g_detach_in_progress_generation = generation; + } + g_detach_in_progress++; + uv_mutex_unlock(&g_mutex); + int detach_rc = fn_detach_thread(thread); - if (detach_rc == 0 && !g_test_hooks) return 0; uv_mutex_lock(&g_mutex); - if (detach_rc == 0 && g_test_hooks && g_test_detach_failure_site == site) { + bool current_generation = generation == g_isolate_generation; + if (detach_rc == 0 && current_generation && g_test_hooks && + g_test_detach_failure_site == site) { g_test_detach_failure_site = DETACH_SITE_NONE; g_test_forced_detach_failures++; detach_rc = -1; } - if (detach_rc != 0) poison_isolate_detach_failure_locked(detach_rc); + if (current_generation && g_test_hooks && + g_test_hold_next_detach_publication) { + g_test_hold_next_detach_publication = false; + g_test_detach_publication_held = true; + uv_cond_broadcast(&g_teardown_cond); + while (!g_test_release_detach_publication) { + uv_cond_wait(&g_teardown_cond, &g_mutex); + } + g_test_release_detach_publication = false; + g_test_detach_publication_held = false; + } + if (detach_rc != 0 && current_generation) { + poison_isolate_detach_failure_locked(detach_rc); + } + if (generation == g_detach_in_progress_generation && g_detach_in_progress > 0) { + g_detach_in_progress--; + if (g_detach_in_progress == 0) g_detach_in_progress_generation = 0; + } + uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); return detach_rc; } @@ -482,6 +553,85 @@ static void resolver_results_free_all(engine_bridge_t* b) { b->results = NULL; } +static void bridge_release_if_unowned_locked(engine_bridge_t* b) { + if (b == NULL || b->native_alive || b->owner_alive || + b->owner_cleanup_tsfn != NULL) return; + resolver_results_free_all(b); + free(b); +} + +static void bridge_owner_cleanup_finalize( + napi_env env, void* finalize_data, void* finalize_hint) { + (void)env; + (void)finalize_hint; + engine_bridge_t* b = (engine_bridge_t*)finalize_data; + if (b == NULL) return; + uv_mutex_lock(&g_mutex); + b->owner_cleanup_tsfn = NULL; + if (b->owner_alive && b->resolver_js != NULL && g_test_hooks && + g_test_live_resolver_refs > 0) { + g_test_live_resolver_refs--; + } + b->owner_alive = false; + b->resolver_js = NULL; + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); +} + +static void call_js_bridge_owner_cleanup( + napi_env env, napi_value js_callback, void* context, void* data) { + (void)js_callback; + (void)data; + engine_bridge_t* b = (engine_bridge_t*)context; + if (b == NULL || env == NULL) return; + if (b->hook_registered) { + napi_remove_env_cleanup_hook(env, bridge_env_cleanup, b); + b->hook_registered = false; + } + bridge_finalize_free(b, /*env_still_alive=*/true); +} + +// Called on the bridge owner thread after its resolver reference is created. +// The unreferenced TSFN is an owner-env cleanup handoff: native drains may queue +// its JS callback from any thread, while env teardown invokes its finalizer. +static bool bridge_register_owner_cleanup(engine_bridge_t* b) { + if (b == NULL || b->resolver_js == NULL || b->env == NULL) return true; + napi_value resource_name; + if (napi_create_string_utf8( + b->env, "dwResolverOwnerCleanup", NAPI_AUTO_LENGTH, &resource_name) != napi_ok || + napi_create_threadsafe_function( + b->env, NULL, NULL, resource_name, 0, 1, b, + bridge_owner_cleanup_finalize, b, call_js_bridge_owner_cleanup, + &b->owner_cleanup_tsfn) != napi_ok) { + return false; + } + if (napi_unref_threadsafe_function(b->env, b->owner_cleanup_tsfn) != napi_ok) { + napi_release_threadsafe_function( + b->owner_cleanup_tsfn, napi_tsfn_abort); + b->owner_cleanup_released = true; + return false; + } + return true; +} + +static void bridge_release_native_and_handoff(engine_bridge_t* b) { + if (b == NULL) return; + uv_mutex_lock(&g_mutex); + if (b->owner_alive && b->resolver_js != NULL && + b->owner_cleanup_tsfn != NULL && !b->owner_cleanup_queued) { + b->owner_cleanup_queued = true; + // Queueing is thread-safe and never invokes the JS callback inline. Keep + // native ownership until the enqueue returns, so an env finalizer cannot + // free b between the ownership transition and this handoff. + napi_status status = napi_call_threadsafe_function( + b->owner_cleanup_tsfn, b, napi_tsfn_nonblocking); + if (status != napi_ok) b->owner_cleanup_queued = false; + } + b->native_alive = false; + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); +} + // Call under g_mutex. Stale bridges stay owned by their env cleanup hooks, but // only records from the currently published isolate may admit operations. static engine_bridge_t* bridge_find_current(long long handle) { @@ -646,6 +796,7 @@ static bool bridge_finalize_registry_at_site(engine_bridge_t* b, detach_site_t d uv_mutex_unlock(&g_mutex); } uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); // If the waiter already committed to physical teardown (TEARING_DOWN) or the // isolate is already gone, the Java registry died/dies with it -- nothing to // remove, and attaching would race graal_tear_down_isolate. Skip, but report @@ -686,11 +837,6 @@ static bool bridge_finalize_registry(engine_bridge_t* b) { return bridge_finalize_registry_at_site(b, DETACH_SITE_BRIDGE_FINALIZE); } -// Forward declaration: the env cleanup hook. bridge_finalize re-registers/keeps -// it on an owner-thread live-isolate strand (may_rehook) and removes it on the -// owner-thread free path; the definition is below (after drain_stranded_bridges). -static void bridge_env_cleanup(void* arg); - // The non-isolate finalize phase: delete the resolver napi_ref (owner JS thread // only, and only while its env is alive -- resolver-gated), free tracked result // buffers, free the record. Touches no GraalVM isolate state, so it is safe to @@ -705,11 +851,24 @@ static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive) { if (g_test_hooks) { uv_mutex_lock(&g_mutex); g_test_resolver_ref_deletes++; + if (g_test_live_resolver_refs > 0) g_test_live_resolver_refs--; uv_mutex_unlock(&g_mutex); } } - resolver_results_free_all(b); - free(b); + uv_mutex_lock(&g_mutex); + if (env_still_alive || b->env == NULL || !b->owner_alive) { + b->resolver_js = NULL; + b->owner_alive = false; + } + b->native_alive = false; + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); + if (env_still_alive && b->owner_cleanup_tsfn != NULL && + !b->owner_cleanup_released) { + b->owner_cleanup_released = true; + napi_release_threadsafe_function( + b->owner_cleanup_tsfn, napi_tsfn_release); + } } // Thin wrapper preserving the original signature and every call site. Registry @@ -785,10 +944,10 @@ static void drain_stranded_bridges(void) { list = list->next; // snapshot the link before b is freed or re-retained b->next = NULL; if (bridge_finalize_registry(b)) { - // Registry entry removed (or isolate gone): the resolver ctx is dead, - // so freeing is safe. Skip the napi_ref delete (env_still_alive=false) - // -- we may not be on the owner thread. - bridge_finalize_free(b, /*env_still_alive=*/false); + // Registry entry removed (or isolate gone): release native ownership. + // The owner env hook remains responsible for deleting resolver_js and + // releasing owner ownership on its Node thread. + bridge_release_native_and_handoff(b); } else { // Still could not attach (isolate live, transient failure): keep the // ctx valid and retry at the next drain. @@ -821,6 +980,15 @@ static void bridge_env_cleanup(void* arg) { if (*pp == b) { *pp = b->next; break; } pp = &(*pp)->next; } + engine_bridge_t** stranded_pp = &g_stranded_bridges; + while (*stranded_pp != NULL) { + if (*stranded_pp == b) { + *stranded_pp = b->next; + b->next = NULL; + break; + } + stranded_pp = &(*stranded_pp)->next; + } // An in-flight streaming/transform op holds a live threadsafe function that // keeps this env's event loop alive, so the env should never tear down while // in_flight > 0. Guard defensively anyway: mark destroy_pending and let the @@ -861,7 +1029,31 @@ static void bridge_env_cleanup(void* arg) { // may_rehook=false: the env is tearing down, so do NOT re-register the hook on // a strand -- a strand here falls back to g_stranded_bridges (Node reclaims the // ref at env teardown; the off-thread drain frees the record later). - bridge_finalize(b, /*env_still_alive=*/true, /*do_registry_remove=*/true, /*may_rehook=*/false); + if (b->resolver_js != NULL && b->env != NULL) { + napi_delete_reference(b->env, b->resolver_js); + b->resolver_js = NULL; + if (g_test_hooks) { + uv_mutex_lock(&g_mutex); + g_test_resolver_ref_deletes++; + if (g_test_live_resolver_refs > 0) g_test_live_resolver_refs--; + uv_mutex_unlock(&g_mutex); + } + } + b->owner_alive = false; + if (b->native_alive) { + if (bridge_finalize_registry(b)) { + uv_mutex_lock(&g_mutex); + b->native_alive = false; + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); + } else { + bridge_retain_stranded(b); + } + } else { + uv_mutex_lock(&g_mutex); + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); + } } // Increment this engine's in_flight while g_mutex is ALREADY held. Used by the @@ -1140,6 +1332,7 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { } uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); bool poisoned_before_drain = g_isolate_poisoned; uv_mutex_unlock(&g_mutex); if (poisoned_before_drain) { @@ -1156,6 +1349,7 @@ static napi_value napi_initialize(napi_env env, napi_callback_info info) { drain_stranded_bridges(); uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); // A detach failure is terminal for the currently published isolate. It may // not be adopted or reused; explicit cleanup must abandon it first. @@ -2561,6 +2755,7 @@ static napi_value napi_run_script_streaming_engine(napi_env env, napi_callback_i // genuine (non-cancelled) PENDING_WAIT or a committed TEARING_DOWN still // rejects. uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); if (g_isolate_poisoned) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); @@ -3179,6 +3374,7 @@ static napi_value napi_run_script_transform_engine(napi_env env, napi_callback_i // post-adoption op throws "Not initialized". A genuine (non-cancelled) // PENDING_WAIT or a committed TEARING_DOWN still rejects. uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); if (g_isolate_poisoned) { uv_mutex_unlock(&g_mutex); napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); @@ -3569,6 +3765,7 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { // check and the g_active_ops++ cannot be split by a teardown because every // teardown transition and the g_active_ops==0 fast path also hold g_mutex. uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); env_init_rec_t* self = env_init_rec_find_locked(env); if (g_isolate_poisoned) { uv_mutex_unlock(&g_mutex); @@ -3648,6 +3845,8 @@ static napi_value napi_create_engine(napi_env env, napi_callback_info info) { return NULL; } rec->native_handle = handle; + rec->native_alive = true; + rec->owner_alive = true; rec->owner = uv_thread_self(); rec->env = env; uv_mutex_lock(&g_mutex); @@ -3738,16 +3937,28 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i if (napi_create_reference(env, argv[0], 1, &bridge->resolver_js) != napi_ok) { free(bridge); napi_throw_error(env, NULL, "Failed to reference resolver callback"); return NULL; } + if (g_test_hooks) { + uv_mutex_lock(&g_mutex); + g_test_live_resolver_refs++; + uv_mutex_unlock(&g_mutex); + } bridge->env = env; bridge->owner = uv_thread_self(); bridge->results = NULL; + bridge->owner_alive = true; + if (!bridge_register_owner_cleanup(bridge)) { + bridge_finalize_free(bridge, /*env_still_alive=*/true); + napi_throw_error(env, NULL, "Failed to register resolver owner cleanup"); + return NULL; + } // Round-14 (#1): same admission block as napi_create_engine. Taken AFTER the // bridge/resolver-ref allocation (those failures touch no isolate state and // must not decrement a reservation not yet held) and BEFORE fn_attach_thread. uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); env_init_rec_t* self = env_init_rec_find_locked(env); if (g_isolate_poisoned) { uv_mutex_unlock(&g_mutex); - napi_delete_reference(env, bridge->resolver_js); free(bridge); + bridge_finalize_free(bridge, /*env_still_alive=*/true); napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); return NULL; } @@ -3755,7 +3966,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i g_teardown_state == TEARDOWN_TEARING_DOWN || self == NULL || self->init_refs == 0) { uv_mutex_unlock(&g_mutex); - napi_delete_reference(env, bridge->resolver_js); free(bridge); + bridge_finalize_free(bridge, /*env_still_alive=*/true); napi_throw_error(env, NULL, "Not initialized. Call initialize() first."); return NULL; } @@ -3765,7 +3976,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i void* thread = NULL; if (fn_attach_thread(g_isolate, &thread) != 0) { uv_mutex_lock(&g_mutex); g_active_ops--; uv_cond_broadcast(&g_teardown_cond); uv_mutex_unlock(&g_mutex); - napi_delete_reference(env, bridge->resolver_js); free(bridge); + bridge_finalize_free(bridge, /*env_still_alive=*/true); napi_throw_error(env, NULL, "Failed to attach thread"); return NULL; } long long handle = fn_create_engine_with_resolver(thread, resolve_module_callback, (void*)bridge); @@ -3792,6 +4003,7 @@ static napi_value napi_create_engine_with_resolver(napi_env env, napi_callback_i } bridge->native_handle = handle; + bridge->native_alive = true; uv_mutex_lock(&g_mutex); bridge->handle = next_engine_handle_locked(); bridge->isolate_generation = g_isolate_generation; @@ -3969,6 +4181,7 @@ static napi_value napi_destroy_engine(napi_env env, napi_callback_info info) { // window; only the g_mutex-guarded read here does. if (fn_destroy_engine && fn_attach_thread) { uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); if (g_teardown_state == TEARDOWN_TEARING_DOWN || g_isolate == NULL) { uv_mutex_unlock(&g_mutex); // isolate gone/tearing down -> nothing to remove } else { @@ -4045,8 +4258,9 @@ static napi_value napi_run_script_engine(napi_env env, napi_callback_info info) // merely-cancelled teardown must not reject here -- otherwise a valid // post-adoption op throws "Not initialized". A genuine (non-cancelled) // PENDING_WAIT or a committed TEARING_DOWN still rejects. - uv_mutex_lock(&g_mutex); - if (g_isolate_poisoned) { + uv_mutex_lock(&g_mutex); + wait_for_detach_publication_locked(); + if (g_isolate_poisoned) { uv_mutex_unlock(&g_mutex); free(script); free(inputs); napi_throw_error(env, NULL, ISOLATE_POISONED_MESSAGE); @@ -4215,7 +4429,10 @@ static void teardown_waiter_thread_fn(void* arg) { (void)arg; uv_mutex_lock(&g_mutex); - while (g_active_ops > 0 && !g_teardown_cancelled) { + while ((g_active_ops > 0 || + (g_detach_in_progress > 0 && + g_detach_in_progress_generation == g_isolate_generation)) && + !g_teardown_cancelled) { uv_cond_wait(&g_teardown_cond, &g_mutex); } bool cancelled = g_teardown_cancelled; @@ -4412,6 +4629,8 @@ static void retry_stranded_teardown_locked(void) { if (g_ref_count > 0) { g_teardown_needed = false; return; } // adopted -> keep if (g_teardown_state != TEARDOWN_NONE) return; // a teardown drives if (g_active_ops > 0) return; // wait for drain + if (g_detach_in_progress > 0 && + g_detach_in_progress_generation == g_isolate_generation) return; if (g_isolate == NULL) { g_teardown_needed = false; return; } // nothing to do uv_thread_t tid; uv_thread_options_t opts; @@ -4445,7 +4664,9 @@ static void isolate_ref_release_n_locked(int n) { if (g_ref_count > 0) return; // other envs still hold references if (g_teardown_state != TEARDOWN_NONE) return; // a teardown already drives - if (g_active_ops == 0) { + if (g_active_ops == 0 && + !(g_detach_in_progress > 0 && + g_detach_in_progress_generation == g_isolate_generation)) { uv_thread_t tid; uv_thread_options_t opts; opts.flags = UV_THREAD_HAS_STACK_SIZE; @@ -4629,7 +4850,9 @@ static napi_value release_isolate_ref_locked(napi_env env) { // Case 4: last release, no teardown pending, and nothing active -- the // original, unchanged synchronous fast path. - if (g_active_ops == 0) { + if (g_active_ops == 0 && + !(g_detach_in_progress > 0 && + g_detach_in_progress_generation == g_isolate_generation)) { uv_thread_t tid; uv_thread_options_t opts; opts.flags = UV_THREAD_HAS_STACK_SIZE; @@ -4970,6 +5193,61 @@ static napi_value napi_test_release_async_op(napi_env env, napi_callback_info in return NULL; } +static napi_value napi_test_hold_detach_publication( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + if (g_test_hold_next_detach_publication || g_test_detach_publication_held) { + uv_mutex_unlock(&g_mutex); + napi_throw_error(env, NULL, "A detach publication gate is already armed"); + return NULL; + } + g_test_hold_next_detach_publication = true; + g_test_release_detach_publication = false; + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_detach_publication_held( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + bool held = g_test_detach_publication_held; + uv_mutex_unlock(&g_mutex); + napi_value out; + napi_get_boolean(env, held, &out); + return out; +} + +static napi_value napi_test_detach_publication_waiters( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t waiters = g_detach_publication_waiters; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, waiters); +} + +static napi_value napi_test_release_detach_publication( + napi_env env, napi_callback_info info) { + (void)env; + (void)info; + uv_mutex_lock(&g_mutex); + g_test_release_detach_publication = true; + uv_cond_broadcast(&g_teardown_cond); + uv_mutex_unlock(&g_mutex); + return NULL; +} + +static napi_value napi_test_live_stranded_resolver_ref_count( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t count = g_test_live_resolver_refs; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, count); +} + static napi_value napi_test_fail_next_engine_record_allocation( napi_env env, napi_callback_info info) { (void)info; @@ -5347,6 +5625,11 @@ static napi_value Init(napi_env env, napi_value exports) { !export_function(env, exports, "__test_holdNextAsyncOp", napi_test_hold_next_async_op) || !export_function(env, exports, "__test_asyncOpHeld", napi_test_async_op_held) || !export_function(env, exports, "__test_releaseAsyncOp", napi_test_release_async_op) || + !export_function(env, exports, "__test_holdDetachPublication", napi_test_hold_detach_publication) || + !export_function(env, exports, "__test_detachPublicationHeld", napi_test_detach_publication_held) || + !export_function(env, exports, "__test_detachPublicationWaiters", napi_test_detach_publication_waiters) || + !export_function(env, exports, "__test_releaseDetachPublication", napi_test_release_detach_publication) || + !export_function(env, exports, "__test_liveStrandedResolverRefCount", napi_test_live_stranded_resolver_ref_count) || !export_function(env, exports, "__test_outputStats", napi_test_output_stats) || !export_function(env, exports, "__test_outputOperationId", napi_test_output_operation_id) || !export_function(env, exports, "__test_createForeignWrappedObject", napi_test_create_foreign_wrapped_object) || diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts index fb827ec1..7765d41c 100644 --- a/native-lib/node/tests/integration/detach-poison-hook.test.ts +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -176,6 +176,36 @@ describe("detach failure poisoning and recovery", () => { }); }); + it("blocks cross-worker admission until synchronous detach failure is published", () => { + expect(runFixture(SYNC_FIXTURE, ["detach-publication-race", "synchronous-run"])).toEqual({ + admittedBeforeRelease: false, + admissionRejectedPoison: true, + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + + it("blocks cross-worker admission until stream-worker detach failure is published", () => { + expect(runFixture(SYNC_FIXTURE, ["detach-publication-race", "stream-worker"])).toEqual({ + admittedBeforeRelease: false, + admissionRejectedPoison: true, + forcedFailures: 1, + abandoned: 1, + freshResult: "42", + }); + }); + + it("deletes a stranded rollback resolver reference on its live owner env", () => { + expect(runFixture(SYNC_FIXTURE, ["handle-exhaustion-owner-cleanup"])).toEqual({ + exhaustionRejected: true, + strandedBeforeCleanup: 1, + strandedAfterHandoff: 0, + resolverDeletes: 1, + liveStrandedResolverRefs: 0, + }); + }); + it("rejects invalid sites and refuses to silently replace an armed failure", () => { expect(runFixture(SYNC_FIXTURE, ["invalid-arguments"])).toEqual({ invalidArguments: 6, diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs index df8bae9f..897107b0 100644 --- a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -25,6 +25,11 @@ const TEST_HOOKS = [ "__test_setNextEngineHandle", "__test_setIsolateGeneration", "__test_isolateGeneration", + "__test_holdDetachPublication", + "__test_detachPublicationHeld", + "__test_detachPublicationWaiters", + "__test_releaseDetachPublication", + "__test_liveStrandedResolverRefCount", ]; const MAX_SAFE_HANDLE = Number.MAX_SAFE_INTEGER; const UINT64_MAX = 18_446_744_073_709_551_615n; @@ -442,6 +447,127 @@ async function handleExhaustionRollbackStrand() { return { exhaustionRejected, strandedDelta }; } +async function detachPublicationRace(site) { + addon.initialize(libPath); + const handle = addon.createEngine(); + const forcedBefore = count("__test_forcedDetachFailureCount"); + const abandonedBefore = count("__test_abandonedIsolateCount"); + addon.__test_forceDetachFailureOnce(site); + addon.__test_holdDetachPublication(); + + const trigger = new Worker(` + "use strict"; + const { parentPort, workerData } = require("node:worker_threads"); + const addon = require(workerData.addonPath); + (async () => { + addon.initialize(workerData.libPath); + if (workerData.site === "synchronous-run") { + addon.runScriptEngine(workerData.handle, "output application/json --- 6 * 7", "{}"); + } else { + const operation = addon.runScriptStreamingEngine( + workerData.handle, + "output application/json --- []", + "{}", + (chunk, sequence) => operation.acknowledge(sequence, chunk.length) + ); + await operation.completion; + } + parentPort.postMessage("trigger-returned"); + })().catch((error) => parentPort.postMessage({ error: error.stack || String(error) })); + `, { eval: true, workerData: { addonPath, libPath, site, handle } }); + const triggerMessagePromise = once(trigger, "message"); + const triggerExitPromise = once(trigger, "exit"); + + while (!addon.__test_detachPublicationHeld()) { + await new Promise((resolve) => setImmediate(resolve)); + } + + const admission = new Worker(` + "use strict"; + const { parentPort, workerData } = require("node:worker_threads"); + const addon = require(workerData.addonPath); + (async () => { + parentPort.postMessage({ ready: true }); + await new Promise((resolve) => parentPort.once("message", resolve)); + addon.initialize(workerData.libPath); + const handle = addon.createEngine(); + parentPort.postMessage({ admitted: true, handle }); + })().catch((error) => { + parentPort.postMessage({ admitted: false, message: error && error.message }); + }); + `, { eval: true, workerData: { addonPath, libPath } }); + const admissionExitPromise = once(admission, "exit"); + + const [ready] = await withTimeout(once(admission, "message"), "detach publication admission ready"); + assert(ready.ready === true, "admission worker did not become ready"); + const admissionResult = new Promise((resolve, reject) => { + admission.once("message", resolve); + admission.once("error", reject); + }); + admission.postMessage("admit"); + while (addon.__test_detachPublicationWaiters() === 0n) { + await new Promise((resolve) => setImmediate(resolve)); + } + const admittedBeforeRelease = await Promise.race([ + admissionResult.then(() => true), + new Promise((resolve) => setImmediate(() => resolve(false))), + ]); + addon.__test_releaseDetachPublication(); + const admissionMessage = await withTimeout(admissionResult, "detach publication admission"); + const admissionRejectedPoison = + admissionMessage.admitted === false && admissionMessage.message === POISONED_MESSAGE; + const [triggerMessage] = await withTimeout(triggerMessagePromise, "detach publication trigger"); + assert(triggerMessage.error === undefined, triggerMessage.error ?? "trigger worker failed"); + await withTimeout(Promise.all([triggerExitPromise, admissionExitPromise]), "detach publication workers"); + + addon.destroyEngine(handle); + await withTimeout(addon.cleanup(), "detach publication cleanup"); + addon.initialize(libPath); + const fresh = addon.createEngine(); + const freshResult = successfulResult( + addon.runScriptEngine(fresh, "output application/json --- 6 * 7", "{}") + ); + addon.destroyEngine(fresh); + await withTimeout(addon.cleanup(), "detach publication fresh cleanup"); + return { + admittedBeforeRelease, + admissionRejectedPoison, + forcedFailures: Number(count("__test_forcedDetachFailureCount") - forcedBefore), + abandoned: Number(count("__test_abandonedIsolateCount") - abandonedBefore), + freshResult, + }; +} + +async function handleExhaustionOwnerCleanup() { + addon.initialize(libPath); + addon.__test_setNextEngineHandle(MAX_SAFE_HANDLE); + const finalHandle = addon.createEngine(); + addon.destroyEngine(finalHandle); + const deletesBefore = addon.__test_resolverRefDeleteCount(); + addon.__test_forceStrandOnce(); + let exhaustionRejected = false; + try { + addon.createEngineWithResolver(() => null); + } catch (error) { + exhaustionRejected = error instanceof Error && error.message === "Engine handle space exhausted"; + } + const strandedBeforeCleanup = addon.__test_strandedCount(); + await withTimeout(addon.cleanup(), "owner cleanup old isolate"); + addon.initialize(libPath); + await new Promise((resolve) => setImmediate(resolve)); + const strandedAfterHandoff = addon.__test_strandedCount(); + const resolverDeletes = addon.__test_resolverRefDeleteCount() - deletesBefore; + const liveStrandedResolverRefs = Number(addon.__test_liveStrandedResolverRefCount()); + await withTimeout(addon.cleanup(), "owner cleanup fresh isolate"); + return { + exhaustionRejected, + strandedBeforeCleanup, + strandedAfterHandoff, + resolverDeletes, + liveStrandedResolverRefs, + }; +} + async function oneShotSite() { addon.initialize(libPath); const first = addon.createEngine(); @@ -537,6 +663,8 @@ async function main() { else if (mode === "generation-exhaustion") result = await generationExhaustion(); else if (mode === "identity-hook-validation") result = await identityHookValidation(); else if (mode === "handle-exhaustion-rollback-strand") result = await handleExhaustionRollbackStrand(); + else if (mode === "detach-publication-race") result = await detachPublicationRace(process.argv[5]); + else if (mode === "handle-exhaustion-owner-cleanup") result = await handleExhaustionOwnerCleanup(); else if (mode === "hooks-absent") result = hooksAbsent(); else throw new Error(`unknown fixture mode: ${mode}`); process.stdout.write(`${JSON.stringify(result)}\n`); diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index 7ac00f44..e2d8b5d1 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -27,8 +27,8 @@ export default defineConfig({ // Opt the integration lane into the addon's test-only entrypoints // (__test_forceStrandOnce / __test_strandedCount / // __test_resolverRefDeleteCount and detach-poison fault/counter hooks, - // including __test_forcedDetachFailureCount, identity-boundary hooks, - // and the engine-record allocation failure hook). Set before any + // including detach-publication barriers, identity-boundary hooks, and + // the engine-record allocation failure hook). Set before any // integration worker loads the addon, so its Init() getenv() sees it; // inert in every other lane and in production. Workers spawned by a // test inherit this env, so the addon Init() in a worker sees it too. From 770be50ed458b6cfed2273039b5723e8e8884f67 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 14:46:23 -0300 Subject: [PATCH 43/48] fix(node): finalize bridge ownership safely --- native-lib/node/src/addon.c | 34 ++++++++++++++++--- .../integration/detach-poison-hook.test.ts | 7 ++++ .../fixtures/detach-poison-sync.cjs | 31 +++++++++++++++++ 3 files changed, 68 insertions(+), 4 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index 46517e85..b6f171fc 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -228,6 +228,7 @@ static bool g_test_detach_publication_held = false; static bool g_test_release_detach_publication = false; static unsigned g_detach_publication_waiters = 0; static uint64_t g_test_live_resolver_refs = 0; +static uint64_t g_test_bridge_frees = 0; static void bridge_env_cleanup(void* arg); static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive); @@ -553,10 +554,13 @@ static void resolver_results_free_all(engine_bridge_t* b) { b->results = NULL; } +// Caller holds g_mutex. A call that can satisfy all three free predicates must +// be the caller's final operation through b. static void bridge_release_if_unowned_locked(engine_bridge_t* b) { if (b == NULL || b->native_alive || b->owner_alive || b->owner_cleanup_tsfn != NULL) return; resolver_results_free_all(b); + if (g_test_hooks) g_test_bridge_frees++; free(b); } @@ -855,19 +859,31 @@ static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive) { uv_mutex_unlock(&g_mutex); } } + napi_threadsafe_function owner_cleanup_tsfn = NULL; uv_mutex_lock(&g_mutex); if (env_still_alive || b->env == NULL || !b->owner_alive) { b->resolver_js = NULL; b->owner_alive = false; } - b->native_alive = false; - bridge_release_if_unowned_locked(b); - uv_mutex_unlock(&g_mutex); if (env_still_alive && b->owner_cleanup_tsfn != NULL && !b->owner_cleanup_released) { b->owner_cleanup_released = true; + owner_cleanup_tsfn = b->owner_cleanup_tsfn; + } + b->native_alive = false; + uv_mutex_unlock(&g_mutex); + + // Releasing the TSFN lets its finalizer clear owner_cleanup_tsfn and perform + // the final bridge free. If no TSFN exists (resolver-less), the ownership + // drop below frees directly. In both cases the finalization action is the + // final bridge operation: never dereference b afterward. + if (owner_cleanup_tsfn != NULL) { napi_release_threadsafe_function( - b->owner_cleanup_tsfn, napi_tsfn_release); + owner_cleanup_tsfn, napi_tsfn_release); + } else { + uv_mutex_lock(&g_mutex); + bridge_release_if_unowned_locked(b); + uv_mutex_unlock(&g_mutex); } } @@ -5248,6 +5264,15 @@ static napi_value napi_test_live_stranded_resolver_ref_count( return test_uint64_counter(env, count); } +static napi_value napi_test_bridge_free_count( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t count = g_test_bridge_frees; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, count); +} + static napi_value napi_test_fail_next_engine_record_allocation( napi_env env, napi_callback_info info) { (void)info; @@ -5630,6 +5655,7 @@ static napi_value Init(napi_env env, napi_value exports) { !export_function(env, exports, "__test_detachPublicationWaiters", napi_test_detach_publication_waiters) || !export_function(env, exports, "__test_releaseDetachPublication", napi_test_release_detach_publication) || !export_function(env, exports, "__test_liveStrandedResolverRefCount", napi_test_live_stranded_resolver_ref_count) || + !export_function(env, exports, "__test_bridgeFreeCount", napi_test_bridge_free_count) || !export_function(env, exports, "__test_outputStats", napi_test_output_stats) || !export_function(env, exports, "__test_outputOperationId", napi_test_output_operation_id) || !export_function(env, exports, "__test_createForeignWrappedObject", napi_test_create_foreign_wrapped_object) || diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts index 7765d41c..0c115d14 100644 --- a/native-lib/node/tests/integration/detach-poison-hook.test.ts +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -206,6 +206,13 @@ describe("detach failure poisoning and recovery", () => { }); }); + it("frees resolver-less bridges exactly once through explicit destroy and env finalization", () => { + expect(runFixture(SYNC_FIXTURE, ["resolverless-finalization"])).toEqual({ + expectedFrees: 200, + actualFrees: 200, + }); + }); + it("rejects invalid sites and refuses to silently replace an armed failure", () => { expect(runFixture(SYNC_FIXTURE, ["invalid-arguments"])).toEqual({ invalidArguments: 6, diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs index 897107b0..1eff7ca3 100644 --- a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -30,6 +30,7 @@ const TEST_HOOKS = [ "__test_detachPublicationWaiters", "__test_releaseDetachPublication", "__test_liveStrandedResolverRefCount", + "__test_bridgeFreeCount", ]; const MAX_SAFE_HANDLE = Number.MAX_SAFE_INTEGER; const UINT64_MAX = 18_446_744_073_709_551_615n; @@ -568,6 +569,35 @@ async function handleExhaustionOwnerCleanup() { }; } +async function resolverlessFinalization() { + const iterations = 100; + const freesBefore = count("__test_bridgeFreeCount"); + addon.initialize(libPath); + for (let i = 0; i < iterations; i++) { + const handle = addon.createEngine(); + addon.destroyEngine(handle); + } + + const worker = new Worker(` + "use strict"; + const { parentPort, workerData } = require("node:worker_threads"); + const addon = require(workerData.addonPath); + addon.initialize(workerData.libPath); + for (let i = 0; i < workerData.iterations; i++) addon.createEngine(); + parentPort.postMessage("created"); + `, { eval: true, workerData: { addonPath, libPath, iterations } }); + const [message] = await withTimeout(once(worker, "message"), "resolver-less worker creation"); + assert(message === "created", "resolver-less worker did not create engines"); + const [exitCode] = await withTimeout(once(worker, "exit"), "resolver-less worker finalization"); + assert(exitCode === 0, `resolver-less worker exited with ${exitCode}`); + + await withTimeout(addon.cleanup(), "resolver-less finalization cleanup"); + return { + expectedFrees: iterations * 2, + actualFrees: Number(count("__test_bridgeFreeCount") - freesBefore), + }; +} + async function oneShotSite() { addon.initialize(libPath); const first = addon.createEngine(); @@ -665,6 +695,7 @@ async function main() { else if (mode === "handle-exhaustion-rollback-strand") result = await handleExhaustionRollbackStrand(); else if (mode === "detach-publication-race") result = await detachPublicationRace(process.argv[5]); else if (mode === "handle-exhaustion-owner-cleanup") result = await handleExhaustionOwnerCleanup(); + else if (mode === "resolverless-finalization") result = await resolverlessFinalization(); else if (mode === "hooks-absent") result = hooksAbsent(); else throw new Error(`unknown fixture mode: ${mode}`); process.stdout.write(`${JSON.stringify(result)}\n`); From 213c68febacc190d449daf30d76f6de50e02a17b Mon Sep 17 00:00:00 2001 From: mlischetti Date: Thu, 10 Sep 2026 15:10:43 -0300 Subject: [PATCH 44/48] test(node): detect post-reclamation finalization --- native-lib/node/src/addon.c | 70 ++++++++++++++----- .../integration/detach-poison-hook.test.ts | 1 + .../fixtures/detach-poison-sync.cjs | 3 + native-lib/node/vitest.config.ts | 5 +- 4 files changed, 61 insertions(+), 18 deletions(-) diff --git a/native-lib/node/src/addon.c b/native-lib/node/src/addon.c index b6f171fc..5432c635 100644 --- a/native-lib/node/src/addon.c +++ b/native-lib/node/src/addon.c @@ -229,6 +229,7 @@ static bool g_test_release_detach_publication = false; static unsigned g_detach_publication_waiters = 0; static uint64_t g_test_live_resolver_refs = 0; static uint64_t g_test_bridge_frees = 0; +static uint64_t g_test_post_reclamation_actions = 0; static void bridge_env_cleanup(void* arg); static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive); @@ -554,16 +555,50 @@ static void resolver_results_free_all(engine_bridge_t* b) { b->results = NULL; } +typedef struct bridge_finalize_action { + napi_threadsafe_function owner_cleanup_tsfn; + bool bridge_reclaimed; +} bridge_finalize_action_t; + // Caller holds g_mutex. A call that can satisfy all three free predicates must // be the caller's final operation through b. -static void bridge_release_if_unowned_locked(engine_bridge_t* b) { +static void bridge_release_if_unowned_locked( + engine_bridge_t* b, bridge_finalize_action_t* action) { if (b == NULL || b->native_alive || b->owner_alive || b->owner_cleanup_tsfn != NULL) return; resolver_results_free_all(b); - if (g_test_hooks) g_test_bridge_frees++; + if (g_test_hooks) { + g_test_bridge_frees++; + if (action != NULL) action->bridge_reclaimed = true; + } free(b); } +// Executes the final owner-side action from stack state captured before any +// bridge reclamation. The probe is independent of b, so a reordered caller can +// report an action attempted after logical reclamation without touching freed +// memory. When bridge is non-NULL and no TSFN release is pending, reclamation is +// performed here as this function's final bridge operation. +static void bridge_finalize_execute_action( + bridge_finalize_action_t* action, engine_bridge_t* bridge) { + if (action == NULL) return; + if (action->bridge_reclaimed) { + uv_mutex_lock(&g_mutex); + g_test_post_reclamation_actions++; + uv_mutex_unlock(&g_mutex); + } + if (action->owner_cleanup_tsfn != NULL) { + napi_release_threadsafe_function( + action->owner_cleanup_tsfn, napi_tsfn_release); + return; + } + if (bridge != NULL) { + uv_mutex_lock(&g_mutex); + bridge_release_if_unowned_locked(bridge, action); + uv_mutex_unlock(&g_mutex); + } +} + static void bridge_owner_cleanup_finalize( napi_env env, void* finalize_data, void* finalize_hint) { (void)env; @@ -578,7 +613,7 @@ static void bridge_owner_cleanup_finalize( } b->owner_alive = false; b->resolver_js = NULL; - bridge_release_if_unowned_locked(b); + bridge_release_if_unowned_locked(b, NULL); uv_mutex_unlock(&g_mutex); } @@ -632,7 +667,7 @@ static void bridge_release_native_and_handoff(engine_bridge_t* b) { if (status != napi_ok) b->owner_cleanup_queued = false; } b->native_alive = false; - bridge_release_if_unowned_locked(b); + bridge_release_if_unowned_locked(b, NULL); uv_mutex_unlock(&g_mutex); } @@ -859,7 +894,7 @@ static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive) { uv_mutex_unlock(&g_mutex); } } - napi_threadsafe_function owner_cleanup_tsfn = NULL; + bridge_finalize_action_t action = {0}; uv_mutex_lock(&g_mutex); if (env_still_alive || b->env == NULL || !b->owner_alive) { b->resolver_js = NULL; @@ -868,7 +903,7 @@ static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive) { if (env_still_alive && b->owner_cleanup_tsfn != NULL && !b->owner_cleanup_released) { b->owner_cleanup_released = true; - owner_cleanup_tsfn = b->owner_cleanup_tsfn; + action.owner_cleanup_tsfn = b->owner_cleanup_tsfn; } b->native_alive = false; uv_mutex_unlock(&g_mutex); @@ -877,14 +912,7 @@ static void bridge_finalize_free(engine_bridge_t* b, bool env_still_alive) { // the final bridge free. If no TSFN exists (resolver-less), the ownership // drop below frees directly. In both cases the finalization action is the // final bridge operation: never dereference b afterward. - if (owner_cleanup_tsfn != NULL) { - napi_release_threadsafe_function( - owner_cleanup_tsfn, napi_tsfn_release); - } else { - uv_mutex_lock(&g_mutex); - bridge_release_if_unowned_locked(b); - uv_mutex_unlock(&g_mutex); - } + bridge_finalize_execute_action(&action, b); } // Thin wrapper preserving the original signature and every call site. Registry @@ -1060,14 +1088,14 @@ static void bridge_env_cleanup(void* arg) { if (bridge_finalize_registry(b)) { uv_mutex_lock(&g_mutex); b->native_alive = false; - bridge_release_if_unowned_locked(b); + bridge_release_if_unowned_locked(b, NULL); uv_mutex_unlock(&g_mutex); } else { bridge_retain_stranded(b); } } else { uv_mutex_lock(&g_mutex); - bridge_release_if_unowned_locked(b); + bridge_release_if_unowned_locked(b, NULL); uv_mutex_unlock(&g_mutex); } } @@ -5273,6 +5301,15 @@ static napi_value napi_test_bridge_free_count( return test_uint64_counter(env, count); } +static napi_value napi_test_post_reclamation_action_count( + napi_env env, napi_callback_info info) { + (void)info; + uv_mutex_lock(&g_mutex); + uint64_t count = g_test_post_reclamation_actions; + uv_mutex_unlock(&g_mutex); + return test_uint64_counter(env, count); +} + static napi_value napi_test_fail_next_engine_record_allocation( napi_env env, napi_callback_info info) { (void)info; @@ -5656,6 +5693,7 @@ static napi_value Init(napi_env env, napi_value exports) { !export_function(env, exports, "__test_releaseDetachPublication", napi_test_release_detach_publication) || !export_function(env, exports, "__test_liveStrandedResolverRefCount", napi_test_live_stranded_resolver_ref_count) || !export_function(env, exports, "__test_bridgeFreeCount", napi_test_bridge_free_count) || + !export_function(env, exports, "__test_postReclamationActionCount", napi_test_post_reclamation_action_count) || !export_function(env, exports, "__test_outputStats", napi_test_output_stats) || !export_function(env, exports, "__test_outputOperationId", napi_test_output_operation_id) || !export_function(env, exports, "__test_createForeignWrappedObject", napi_test_create_foreign_wrapped_object) || diff --git a/native-lib/node/tests/integration/detach-poison-hook.test.ts b/native-lib/node/tests/integration/detach-poison-hook.test.ts index 0c115d14..fe752ee1 100644 --- a/native-lib/node/tests/integration/detach-poison-hook.test.ts +++ b/native-lib/node/tests/integration/detach-poison-hook.test.ts @@ -210,6 +210,7 @@ describe("detach failure poisoning and recovery", () => { expect(runFixture(SYNC_FIXTURE, ["resolverless-finalization"])).toEqual({ expectedFrees: 200, actualFrees: 200, + postReclamationActions: 0, }); }); diff --git a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs index 1eff7ca3..f2c03fc6 100644 --- a/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs +++ b/native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs @@ -31,6 +31,7 @@ const TEST_HOOKS = [ "__test_releaseDetachPublication", "__test_liveStrandedResolverRefCount", "__test_bridgeFreeCount", + "__test_postReclamationActionCount", ]; const MAX_SAFE_HANDLE = Number.MAX_SAFE_INTEGER; const UINT64_MAX = 18_446_744_073_709_551_615n; @@ -572,6 +573,7 @@ async function handleExhaustionOwnerCleanup() { async function resolverlessFinalization() { const iterations = 100; const freesBefore = count("__test_bridgeFreeCount"); + const actionsBefore = count("__test_postReclamationActionCount"); addon.initialize(libPath); for (let i = 0; i < iterations; i++) { const handle = addon.createEngine(); @@ -595,6 +597,7 @@ async function resolverlessFinalization() { return { expectedFrees: iterations * 2, actualFrees: Number(count("__test_bridgeFreeCount") - freesBefore), + postReclamationActions: Number(count("__test_postReclamationActionCount") - actionsBefore), }; } diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index e2d8b5d1..34340b59 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -27,8 +27,9 @@ export default defineConfig({ // Opt the integration lane into the addon's test-only entrypoints // (__test_forceStrandOnce / __test_strandedCount / // __test_resolverRefDeleteCount and detach-poison fault/counter hooks, - // including detach-publication barriers, identity-boundary hooks, and - // the engine-record allocation failure hook). Set before any + // including detach-publication barriers, bridge finalization counters, + // identity-boundary hooks, and the engine-record allocation failure + // hook). Set before any // integration worker loads the addon, so its Init() getenv() sees it; // inert in every other lane and in production. Workers spawned by a // test inherit this env, so the addon Init() in a worker sees it too. From 30810ba37c1c06544b80bfb2a3065add746fa376 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 11 Sep 2026 10:03:25 -0300 Subject: [PATCH 45/48] docs(native-lib): document hardened multi-engine contracts --- ...26-08-04-nodejs-external-modules-design.md | 14 ++-- ...26-08-07-native-lib-multi-engine-design.md | 84 ++++++++++++------- native-lib/README.md | 18 ++++ native-lib/node/README.md | 29 ++++++- native-lib/python/README.md | 23 ++++- native-lib/python/src/dataweave/native.py | 1 - 6 files changed, 127 insertions(+), 42 deletions(-) diff --git a/docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md b/docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md index 445e7202..77289a6d 100644 --- a/docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md +++ b/docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md @@ -1,7 +1,7 @@ # Design: External DataWeave Module Support in Node.js Binding -**Date:** 2026-08-04 -**Status:** Superseded +**Date:** 2026-08-04 +**Status:** Superseded **Related Proposal:** [docs/proposals/nodejs-external-modules.md](../../proposals/nodejs-external-modules.md) > **⚠️ Superseded (2026-08-31).** This document describes the original @@ -366,20 +366,20 @@ public interface ResolveModuleCallback extends CFunctionPointer { ```java public class CallbackWeaveResourceResolver implements WeaveResourceResolver { private final ResolveModuleCallback callback; - + public CallbackWeaveResourceResolver(ResolveModuleCallback callback) { this.callback = callback; } - + @Override public Option resolve(ResourceDescriptor descriptor) { CCharPointer pathPtr = CTypeConversion.toCString(descriptor.path()).get(); CCharPointer resultPtr = callback.invoke(CurrentIsolate.getCurrentThread(), pathPtr); - + if (resultPtr.isNull()) { return Option.empty(); // Resolver returned null } - + String source = CTypeConversion.toJavaString(resultPtr); // Note: host must free resultPtr after this returns return Option.apply(new StringWeaveResource(descriptor.path(), source)); @@ -462,7 +462,7 @@ static char* resolve_module_callback(void* thread, const char* module_path) { // Return result_source (or NULL) } -static void resolver_js_callback(napi_env env, napi_value js_callback, +static void resolver_js_callback(napi_env env, napi_value js_callback, void* context, void* data) { // Call JS: result = resolveModule(modulePath) // Extract result string or null diff --git a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md index 00772917..f692f2af 100644 --- a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md +++ b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md @@ -493,17 +493,17 @@ regardless of which OS thread performs the last release. - **`run` / `run_streaming` / `run_callback` / `run_transform`** — route through the `*_engine` entrypoints with the instance's `handle`, per §7.2's attachment rules. Per-instance execution is serialized (`_serialized_native_operation`); different instances run concurrently. -- **`cleanup()`** — drain *this instance's* stream workers (signal cancel + **join** the threads; - synchronous, Python owns them, so no event loop and no deadlock); `destroy_engine(handle)`; remove - the resolver-map entry; clear the instance handle; then release the isolate ref (`-= 1`), tearing - the isolate down on the last release. `cleanup()` on an uninitialized/already-cleaned instance is a - no-op; double-`cleanup()` releases the ref only once (guarded by the instance handle being - cleared). If `destroy_engine` throws, the isolate ref is still released so a throwing destroy - cannot strand the isolate; the error is re-raised after the release. - -**Why this stays simple:** teardown happens only on the *last* release, by which point every -instance has already joined its own workers, so the isolate has no attached worker threads when -`graal_tear_down_isolate` runs. +- **`cleanup()`** — refuses while this instance has an active registered stream worker; it does not + use Node's cancel-and-wait cleanup protocol. Once no worker is active it destroys the engine, + removes the resolver-map entry, clears the operation token, and releases the isolate ref. Worker + registration validates its captured `{handle, generation}` token while holding the registry lock, + so cleanup/reinitialize cannot admit stale work. Cleanup on an uninitialized/already-cleaned + instance is a no-op; double cleanup releases the ref only once. If `destroy_engine` throws, the + isolate ref is still released and the error is re-raised. + +**Why this stays simple:** teardown happens only on the *last* release after each instance has no +registered active worker. A still-running daemon worker prevents its instance's cleanup, rather than +allowing teardown to race an attached worker thread. ### 7.4 Resolver dispatch and the streaming/resolver hazard @@ -532,12 +532,12 @@ instance has already joined its own workers, so the isolate has no attached work ### Layer 1 — Java (`native-lib/src/main/java/org/mule/weave/lib/`) — shared by both bindings -- **`ScriptRuntime.java`** — from static singleton to per-instance + a - `ConcurrentHashMap` registry with `register`/`get`/`destroy` and an - `AtomicLong` handle allocator. The resolver is bound once at construction (immutable for the - instance's lifetime); the `static setResolver` write-once mutation is removed. - `compositeResolver()` / `createModuleComponentsFactory()` become instance methods. - `getInstance()` / `defaultInstance` are **removed** — `ScriptRuntime` is purely handle-addressed. +- **`ScriptRuntime.java`** — a handle-keyed registry of lifecycle records, each retaining its + `ScriptRuntime` plus `LIVE`, `CLOSING`, or `DESTROYED` state and a count of admitted operation + leases. `acquire(handle)` admits only `LIVE` records and returns a lease; `destroy(handle)` closes + admission, waits for admitted leases to drain, then removes the record. The resolver is bound at + construction for the runtime lifetime; the former write-once static resolver and `getInstance()` + / `defaultInstance` are removed. - **`CallbackWeaveResourceResolver.java`** — stores a `PointerBase ctx` alongside the callback, forwarded on every `callback.invoke(...)`; constructor `(ResolveModuleCallback, PointerBase ctx)`. - **`NativeCallbacks.java`** — `ResolveModuleCallback` is the 3-arg ctx form @@ -546,8 +546,10 @@ instance has already joined its own workers, so the isolate has no attached work to the correct per-handle resolver on the C/Python side. The old 2-arg form is gone. - **`NativeLib.java`** — exposes only the handle-based lifecycle + execution entrypoints (`create_engine`, `create_engine_with_resolver`, `destroy_engine`, `run_script_engine`, - `run_script_callback_engine`, `run_script_input_output_callback_engine`) resolving via - `ScriptRuntime.get(handle)`. The three legacy singleton entrypoints (`run_script`, + `run_script_callback_engine`, `run_script_input_output_callback_engine`). Java entrypoint + exceptions return ABI sentinels: create returns `0`, and run entrypoints return `NULL`. + Normal script failures remain allocated non-`NULL` JSON envelopes with `success:false`. + The three legacy singleton entrypoints (`run_script`, `run_script_callback`, `run_script_input_output_callback`) and the old `*_with_resolver` entrypoints are **removed** (see §10). @@ -583,12 +585,14 @@ instance has already joined its own workers, so the isolate has no attached work ### Layer 4 — Python (`native-lib/python/src/dataweave/`) - **`native.py` (`NativeRuntime`)** — the shared-model glue (§7): module-level refcounted isolate, - attach-on-demand thread handling, the 3-arg ctx resolver trampoline + `_resolver_registry`, and - the `*_engine` + `create_engine[_with_resolver]` + `destroy_engine` symbol bindings. + attach-on-demand thread handling, the 3-arg ctx resolver trampoline + `_resolver_registry`, + generation-bound `{handle, generation}` operation tokens, same-thread callback reentrancy guard, + and the `*_engine` + `create_engine[_with_resolver]` + `destroy_engine` symbol bindings. - **`runtime.py` (`DataWeave`)** — `initialize()` acquires an isolate ref + creates one engine and - stores its `handle`; run methods route through the `*_engine` entrypoints with that handle; - `cleanup()` drains this instance's stream workers, `destroy_engine(handle)`, releases the ref. - The public API surface is unchanged. + stores its generation-bound operation token; run methods route through the `*_engine` entrypoints. + Cleanup refuses while an active stream worker is registered; registration validates the token so + stale work cannot be admitted. It then destroys the engine and releases the ref. The public API + surface is unchanged. - **`models.py`** — `RESOLVE_MODULE_CALLBACK` ctypes signature carries the `ctx` argument. ## 9. Data Flow @@ -606,7 +610,7 @@ new DataWeave({ resolveModule: A }).initialize() dwA.run(script importing "custom/lib.dwl") → ffi.runScriptEngine(handle_A, script, inputs) → addon.c: admission (g_active_ops++, in_flight++ on bridge_A) → attach → fn_run_script_engine - → Java: ScriptRuntime.get(handle_A).run(...) + → Java: ScriptRuntime.acquire(handle_A) → admitted lease → runtime.run(...) composite resolver: ClassLoader miss → callback.invoke(thread, ctx=&bridge_A, "custom/lib.dwl") → C: resolve_module_callback casts ctx→bridge_A; thread==owner? yes → call resolver A synchronously → result flows back, script compiles; on completion: in_flight--, g_active_ops-- @@ -641,9 +645,9 @@ dwB.cleanup() → join workers; destroy_engine(handleB); ref 1→0 → attach f per-handle). - **Wrong-thread resolver invocation:** per-handle/per-token `owner` check fails closed to "not found" rather than touching the host callback cross-thread — identical in both bindings. -- **Invalid/unknown/destroyed handle:** `ScriptRuntime.get(handle)` returns null → the entrypoint - returns `{"success":false,"error":"Unknown engine handle"}` (resolved for async ops, returned as - the JSON string for sync `run()`), never an NPE/crash. +- **Invalid/unknown/destroyed handle:** `ScriptRuntime.acquire(handle)` returns no lease → the + entrypoint returns `{"success":false,"error":"Unknown engine handle"}` (resolved for async ops, + returned as the JSON string for sync `run()`), never an NPE/crash. - **Node admission / argument / allocation failures:** synchronous `napi_throw_error` (generic Error); worker-thread OOM → terminal error JSON. Never `napi_reject_deferred` (absent from `addon.c`). @@ -736,6 +740,30 @@ uphold all six: 6. A failed engine-create rolls back the isolate ref; a throwing `destroy_engine` still releases the ref. +## 12.1 Final hardening contract + +This section records final behavior that is intentionally narrower than a public API guarantee. +Both bindings use callback thread-local state to reject same-thread resolver/read/write callback +reentry into lifecycle or execution APIs with their public `DataWeaveError`. Both generation-bind +lazy stream and transform work; cleanup or reinitialization before consumption or admission rejects +the stale work rather than allowing it to execute on a replacement engine. + +Node's native output bridge bounds its own outstanding bytes and chunks and uses a finite TSFN +queue. Buffers retained by user code after yield are outside that bound. Controller and sequence +credit values are internal correctness machinery, not a public BigInt sequence contract. Node +cleanup cancels and waits for abandoned active streams/transforms before engine destruction. + +Python deliberately differs: it cannot force-cancel an active native call, so cleanup refuses while +an active streaming worker is attached. Its registration validation is generation-safe; it does not +adopt Node's cancellation behavior. + +**Test-only detach hooks.** The addon enables fault injection only when `DATAWEAVE_TEST_HOOKS` is +non-empty. A detach hook always performs a real detach first and can synthesize a failure only after +that detach succeeds; it does not model a physically stuck Graal thread. A real detach failure +poisons admission fail-closed. Cleanup abandons the old published generation, and a later fresh +initialization can recover with a new isolate. These hooks and their names are implementation/test +details, not stable APIs. + ## 13. Follow-Up Work - **Streaming/transform + custom-module resolution** across the background-thread boundary remains diff --git a/native-lib/README.md b/native-lib/README.md index 87cfdc98..93a7699c 100644 --- a/native-lib/README.md +++ b/native-lib/README.md @@ -68,6 +68,24 @@ isolate for the process lifetime, and lets a future initialization build a fresh isolate. This ref-counting and teardown policy lives in the binding code, not in the dwlib engine ABI. +## Raw engine ABI contract + +The exported engine entrypoints are `create_engine`, +`create_engine_with_resolver`, `destroy_engine`, `run_script_engine`, +`run_script_callback_engine`, and `run_script_input_output_callback_engine`. +`create_engine` and `create_engine_with_resolver` return `0` when their Java C +entrypoint fails. The three `run_*_engine` entrypoints return `NULL` when their +Java C entrypoint fails; callers must not pass `NULL` to `free_cstring`. Those +sentinels are distinct from normal DataWeave failures: a script error is a +non-`NULL` JSON envelope with `success:false`, and that allocated result must be +freed normally. + +`destroy_engine` closes admission for the handle and blocks until operations +already admitted to that engine drain. Resolver, read, and write callback +contexts must remain valid until `destroy_engine` returns. A callback must not +call `destroy_engine` synchronously for the engine invoking it: that operation +holds an admitted lease and would wait for itself to finish. + ## Building with Gradle ### Prerequisites diff --git a/native-lib/node/README.md b/native-lib/node/README.md index 75d0090f..25ea51bf 100644 --- a/native-lib/node/README.md +++ b/native-lib/node/README.md @@ -232,6 +232,26 @@ import { cleanup } from 'dataweave-native'; await cleanup(); ``` +### Callback and stream lifecycle + +Resolver, read, and write callbacks must not call DataWeave lifecycle or +execution APIs on the same thread. The binding rejects that reentry with a +public `DataWeaveError` instead of recursively entering the native runtime. +Native addon callers receive the message `DataWeave native methods cannot be +called from a native callback` and code `ERR_DATAWEAVE_CALLBACK_REENTRANCY`. + +Streaming and transform work captures the initialized engine generation. If +cleanup or reinitialization happens before it is consumed or admitted, it fails +with `DataWeaveError: DataWeave operation belongs to a stale engine generation.` +and never runs against the replacement engine. Cleanup cancels and waits for +abandoned active streams and transforms before destroying their engine. + +The addon uses bounded native output buffering. Its byte and chunk watermarks, +finite thread-safe-function queue, and controller/sequence credit bookkeeping +are implementation details, not public configuration or a BigInt sequence API. +The bound excludes `Buffer` objects retained by application code after a chunk +is yielded. + ### Class-Based API For more control, use the `DataWeave` class directly: @@ -411,8 +431,9 @@ console.log(result.getString()); // "300" ### Streaming Large Files ```javascript -import { runTransform } from 'dataweave-native'; -import { readFileSync, createWriteStream } from 'fs'; +import { once } from "node:events"; +import { readFileSync, createWriteStream } from "node:fs"; +import { runTransform } from "dataweave-native"; const script = ` %dw 2.0 @@ -439,7 +460,9 @@ const generator = runTransform( const output = createWriteStream('filtered.json'); for await (const chunk of generator) { - output.write(chunk); + if (!output.write(chunk)) { + await once(output, "drain"); + } } output.end(); diff --git a/native-lib/python/README.md b/native-lib/python/README.md index 3e776559..762e825b 100644 --- a/native-lib/python/README.md +++ b/native-lib/python/README.md @@ -222,6 +222,23 @@ callback until its engine is destroyed, then releases callback references during `cleanup()`. Different live instances can therefore use different resolvers. +### Callback and stream lifecycle + +Resolver, read, and write callbacks must not call DataWeave lifecycle or +execution APIs on the same thread. The binding rejects that reentry with the +public `DataWeaveError` message `DataWeave lifecycle and execution are not +allowed from a native callback on the same thread.` rather than recursively +entering the native runtime. + +Streaming and transform work captures its initialized `{handle, generation}` +operation identity. Cleanup or reinitialization before a captured operation is +consumed or registered rejects it with `DataWeaveError: DataWeave operation +belongs to a stale engine generation.`; it never runs on the replacement +engine. Cleanup differs from Node: it refuses with `DataWeaveError` while an +active streaming worker is attached. Safe worker registration validates the +captured operation while holding the worker registry lock, so cleanup cannot +admit work for an old generation. + ### Custom module resolution scope - A `resolve_module` you configure applies to `run()`. @@ -274,9 +291,9 @@ print(f"\nDone: {metadata.mime_type}, {metadata.charset}") Call `stream.close()` when stopping consumption early. `Stream` also supports a context manager, as above. Closing requests cancellation and waits only briefly -for the native worker. A native call cannot be forcibly cancelled by Python, so -an unresponsive call is left to finish in a daemon worker rather than delaying -application shutdown or raising during finalization. +for the daemon worker. A native call cannot be forcibly cancelled by Python; +`DataWeave.cleanup()` instead refuses while an active streaming worker remains +attached, preventing destruction of its engine until the worker unregisters. Or with explicit context: diff --git a/native-lib/python/src/dataweave/native.py b/native-lib/python/src/dataweave/native.py index 380fb1ae..3c89bb63 100644 --- a/native-lib/python/src/dataweave/native.py +++ b/native-lib/python/src/dataweave/native.py @@ -493,7 +493,6 @@ def decode_and_free(self, ptr, thread=None) -> str: except Exception: if primary_error is None: raise - def run_engine_and_decode(self, script: bytes, inputs: bytes, operation: _EngineOperation) -> str: with self._serialized_native_operation(operation) as operation_lock: with self._current_thread_attachment(self.thread) as thread: From f8d1d350fa7d149a38833126d5356e9c5f0626cc Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 11 Sep 2026 10:08:55 -0300 Subject: [PATCH 46/48] docs(native-lib): align lifecycle design details --- ...26-08-07-native-lib-multi-engine-design.md | 34 +++++++++++-------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md index f692f2af..6fe35355 100644 --- a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md +++ b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md @@ -351,13 +351,12 @@ because a boolean cannot represent the window during which `cleanup()` has start concurrent `initialize()` is deterministically rejected instead of racing a fresh isolate against the in-flight release. `initialize()` and `run()` stay **synchronous** (an async signature would be an API break). -- **`run()` / `runStreaming()` / `runTransform()`** — gated by `ensureReady()`: throw - `DataWeaveError` unless `state === "ready"`, so the internal `engineHandle === null` cleanup - window is unreachable by any public method (defense-in-depth behind the C admission check). - `runTransform` additionally re-checks `ensureReady()` **after** `await createChunkReader(input)` - (async input pre-buffering can span arbitrary time; the instance may be cleaned up during it) so - a misused instance gets a synchronous `DataWeaveError` rather than a resolved `Unknown engine - handle` envelope. +- **`run()` / `runStreaming()` / `runTransform()`** — `captureOperationToken()` initially checks + readiness and captures `{handle, generation}`. `runTransform` calls + `assertCurrentOperation(token)` both after `await createChunkReader(input)` and immediately at + native admission. Async input pre-buffering can span arbitrary time; cleanup or reinitialization + during it therefore raises the public stale-generation `DataWeaveError` instead of admitting work + to a replacement engine or returning an `Unknown engine handle` envelope. - **`cleanup()` / `doCleanup()`** — set `state = "cleaning-up"` synchronously *before* `ffi.destroyEngine` / `ffi.cleanup` (the key ordering). It always runs `await ffi.cleanup()` even if `destroyEngine()` throws (a real path — wrong-thread destruction throws synchronously), @@ -634,8 +633,12 @@ dwA.run("... import custom/lib ...") → Java engine A: ClassLoader miss → callback(thread, ctx=tokenA, "custom/lib") → trampoline: registry[tokenA] → resolver A → source; A's cache used, B untouched -dwA.cleanup() → join dwA workers; destroy_engine(handleA); ref 2→1 (isolate stays) -dwB.cleanup() → join workers; destroy_engine(handleB); ref 1→0 → attach fresh thread + graal_tear_down_isolate(); _isolate=None +dwA.cleanup() → verify no active registered worker; destroy_engine(handleA); ref 2→1 (isolate stays) +dwB.cleanup() → verify no active registered worker; destroy_engine(handleB); ref 1→0 + → attach fresh thread + graal_tear_down_isolate(); _isolate=None + +active worker at cleanup → reject without joining or cancelling the worker; its generation-safe + registration prevents stale work from being admitted after lifecycle changes ``` ## 10. Error Handling & Backward Compatibility @@ -711,12 +714,13 @@ dwB.cleanup() → join workers; destroy_engine(handleB); ref 1→0 → attach f modules; streaming/transform still stream; streaming custom-module resolution fails closed (parity); a **foreign-thread last-release no-hang** regression (init on a worker thread, last release/cleanup on a different thread, bounded timeout); TCK conformance stays green. -- **Documented posture on non-forceable paths (Node).** Allocator/N-API fault injection and exact - cross-thread teardown interleavings are **not deterministically forceable** from JS/vitest (no - addon-boundary fault-injection hook — deliberately not added, YAGNI/test-only surface). Their - correctness rests on the C-level invariants in §6, verified by code reasoning and adversarial - review; the Worker tests are best-effort probabilistic guards. This is a standing, documented - decision. +- **Test-only addon hook posture (Node).** With non-empty `DATAWEAVE_TEST_HOOKS`, internal, + non-public `__test_*` exports deterministically cover selected detach failures, engine-record + allocation failure, engine generation and handle boundaries, output settlement/status failures, + output-credit accounting, and scheduling/output-flow delivery paths. The hooks are intentionally + limited: unhooked native and N-API failure branches and arbitrary cross-thread interleavings retain + static or probabilistic Worker-test coverage. Detach injection runs only after a real detach has + succeeded, so it cannot safely reproduce a physically stuck attached Graal thread. - **Native image build** (`native-lib:nativeCompile`) stays green with the legacy entrypoints removed (confirms no SPI/reflection config referenced them). From e3416e513e18d01626846220259e1aee5283c776 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 11 Sep 2026 10:12:24 -0300 Subject: [PATCH 47/48] docs(native-lib): correct Python cleanup contract --- .../2026-08-07-native-lib-multi-engine-design.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md index 6fe35355..c82ff66d 100644 --- a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md +++ b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md @@ -116,8 +116,9 @@ no change to isolate lifecycle management for the *feature*. Node requires the c reference-and-teardown coordination in §6 because the isolate is shared by independently created and destroyed engines across threads. **Python can adopt the same model trivially**: its ctypes calls are synchronous and it owns its stream-worker threads directly, so it needs none of Node's -*asynchronous* `PENDING_WAIT`/waiter-thread/adoption machinery — a reference count, a -synchronous drain-before-teardown, and a simpler *synchronous* teardown retry suffice (§7). +*asynchronous* `PENDING_WAIT`/waiter-thread/adoption machinery — a reference count, synchronous +refusal while an active registered worker exists, normal teardown once no worker is registered, and +a simpler *synchronous* teardown retry (§7). **Accepted trade-off (Python).** Python instances in one process now share one isolate's heap instead of having separate heaps. This is weaker memory isolation, relevant only if @@ -423,9 +424,10 @@ Python drives the **same** shared Java engine layer and the **same** `*_engine` its isolate/thread glue (`native-lib/python/src/dataweave/native.py`) is much simpler than §6: ctypes calls are synchronous and Python owns its stream-worker threads directly, so it needs none of Node's *asynchronous* `PENDING_WAIT`/waiter-thread/adoption machinery. It still needs a -reference count, a synchronous drain-before-teardown, and a simpler *synchronous* teardown retry -(`_teardown_needed`, retried on the next `initialize()` — see §7.2). The **public Python API is -unchanged** by the unification. +reference count, synchronous refusal while an active registered worker exists, normal teardown +after that worker unregisters, and a simpler *synchronous* teardown retry (`_teardown_needed`, +retried on the next `initialize()` — see §7.2). Unlike Node (§6), Python cleanup does not cancel or +wait for an active worker. The **public Python API is unchanged** by the unification. ### 7.1 Shared state and the reference-count invariant From 802cef25f5ed7b91ed8aa2406e227aefe40a6fb3 Mon Sep 17 00:00:00 2001 From: mlischetti Date: Fri, 11 Sep 2026 10:33:57 -0300 Subject: [PATCH 48/48] docs: consolidate multi-engine hardening design --- .../2026-09-02-pr157-review-22-remediation.md | 1208 ----------------- ...26-08-07-native-lib-multi-engine-design.md | 57 +- ...9-02-pr157-review-22-remediation-design.md | 365 ----- 3 files changed, 37 insertions(+), 1593 deletions(-) delete mode 100644 docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md delete mode 100644 docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md diff --git a/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md b/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md deleted file mode 100644 index bdfee097..00000000 --- a/docs/superpowers/plans/2026-09-02-pr157-review-22-remediation.md +++ /dev/null @@ -1,1208 +0,0 @@ -# PR #157 Review 22 Remediation Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Resolve all eight PR #157 review-22 findings and deliver a verified PR from `w-23692110-review-22-fixes` to `w-23692110-multi-engine-design`. - -**Architecture:** The Java engine registry owns `LIVE -> CLOSING -> DESTROYED` records and leases so the raw C ABI is lifetime-safe. Node and Python add same-OS-thread native-callback guards and immutable `{handle, generation}` operation tokens; Node additionally uses one credit/ack flow-control object per asynchronous stream to bound native and JavaScript buffering and to cancel abandoned consumers safely. - -**Tech Stack:** Java 17, GraalVM Community Java 24 Native Image, C11/N-API 8/libuv, TypeScript 5.5/Node.js 18+, Python 3.9+/ctypes, Gradle, JUnit 5, Vitest 3, pytest. - -**Spec:** `docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md` - -## Global Constraints - -- Use the checked-in `./gradlew` wrapper and GraalVM Community Java 24 for native verification. -- Java remains source/target 17; Scala remains 2.12; Python remains 3.9+; Node remains 18+. -- Preserve exported C names, argument order, callback semantics, and existing JSON wire fields. -- Normal script failures continue to return non-null `{"success":false,...}` envelopes. -- Never let Java, JavaScript, or Python exceptions unwind across C callbacks. -- Every OS thread calling Graal attaches its own isolate thread and detaches afterward unless a successful teardown has invalidated the attachment. -- Node shared C lifecycle state remains guarded by `g_mutex`; per-stream flow state uses its own mutex with explicit lock ordering. -- Python module isolate state remains guarded by `_isolate_lock`; user callbacks run without module locks held. -- Use test-first red-green cycles for every behavior change. Run the named failing test before editing production code and record the expected failure. -- Work only in `/private/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review-22-fixes` on `w-23692110-review-22-fixes`. -- Do not modify unrelated untracked or generated artifacts. Do not commit `node_modules`, staged `native/`, `dist`, native build outputs, coverage, wheels, or downloaded TCK suites. - -## File Map - -- `native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java`: owns the Java engine registry, lifecycle records, and operation leases. -- `native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java`: contains allocation-free GraalVM exception handlers by ABI return category. -- `native-lib/src/main/java/org/mule/weave/lib/NativeLib.java`: validates exported entrypoint inputs, acquires core leases, and applies exception sentinels. -- `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java`: exercises hosted Java lease and destroy concurrency. -- `native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java`: verifies every exported entrypoint declares the intended exception handler. -- `native-lib/python/src/dataweave/native.py`: owns Python immutable operation tokens, serialized admission, and callback-thread-local state. -- `native-lib/python/src/dataweave/runtime.py`: captures operation generations at public API entry and binds stream workers to them. -- `native-lib/python/tests/unit/test_native.py`: tests immutable native admission and callback TLS. -- `native-lib/python/tests/unit/test_streaming.py`: tests stale stream rejection and callback wrappers. -- `native-lib/python/tests/integration/test_module_resolver.py`: isolates resolver reentrancy so a regression cannot kill pytest. -- `native-lib/python/tests/integration/test_lifecycle.py`: holds direct ctypes ABI sentinel and lease/drain subprocess tests. -- `native-lib/node/src/addon.c`: owns authoritative callback TLS, asynchronous stream flow control, cancellation, and detach fault injection. -- `native-lib/node/src/ffi.ts`: types and wraps the internal native streaming operation controller. -- `native-lib/node/src/stream.ts`: acknowledges chunks at dequeue and cancels abandoned generators. -- `native-lib/node/src/dataweave.ts`: owns Node generations, token validation, public error mapping, and active-stream cleanup. -- `native-lib/node/tests/unit/dataweave-initialize.test.ts`: tests Node generation capture, handle reuse, and active-stream cleanup. -- `native-lib/node/tests/unit/stream.test.ts`: tests credit acknowledgment and cancellation semantics without native code. -- `native-lib/node/tests/integration/resolver-reentrancy.test.ts`: child-process proof that nested resolver execution no longer exits 99. -- `native-lib/node/tests/integration/stream-backpressure.test.ts`: real-addon bounded-buffer and cancellation tests. -- `native-lib/node/tests/integration/detach-poison-hook.test.ts`: child-process detach-poison and fresh-isolate recovery tests. -- `native-lib/README.md`, `native-lib/node/README.md`, `native-lib/python/README.md`, and the consolidated design: public and maintainer-facing contract updates. - ---- - -### Task 1: Core Java Engine Leases - -**Files:** -- Create: `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java` -- Modify: `native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java:629-691` -- Modify: `native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java:40-59` - -**Interfaces:** -- Consumes: existing `ScriptRuntime.register(ScriptRuntime)` and `ScriptRuntime.destroy(long)` semantics. -- Produces: `ScriptRuntime.EngineLease acquire(long handle)`, `EngineLease.runtime()`, `EngineLease.close()`, and blocking/idempotent `destroy(long handle)`. - -- [ ] **Step 1: Write lifecycle tests that expose the missing lease** - -Create `ScriptRuntimeLifecycleTest` with deterministic latches. The central test shape is: - -```java -@Test -void destroyWaitsForAnAdmittedLeaseAndRejectsNewAdmission() throws Exception { - long handle = ScriptRuntime.register(new ScriptRuntime()); - ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle); - assertNotNull(lease); - - CountDownLatch destroyStarted = new CountDownLatch(1); - AtomicBoolean destroyReturned = new AtomicBoolean(false); - Thread destroyer = new Thread(() -> { - destroyStarted.countDown(); - ScriptRuntime.destroy(handle); - destroyReturned.set(true); - }); - destroyer.start(); - - assertTrue(destroyStarted.await(1, TimeUnit.SECONDS)); - awaitCondition(() -> ScriptRuntime.acquire(handle) == null); - assertFalse(destroyReturned.get()); - - lease.close(); - destroyer.join(1_000); - assertFalse(destroyer.isAlive()); - assertTrue(destroyReturned.get()); - assertNull(ScriptRuntime.acquire(handle)); -} -``` - -Add focused cases for: - -```java -multipleLeasesMustAllDrainBeforeDestroyReturns(); -concurrentDestroyCallsCoordinateAndComplete(); -closingAnEngineLeaseTwiceIsHarmless(); -interruptedDestroyRestoresInterruptAfterTheLeaseDrains(); -unknownHandleCannotAcquireALease(); -``` - -Use a local polling helper with a one-second deadline instead of sleeps. Every test must close admitted leases in `finally`. - -- [ ] **Step 2: Run the lifecycle test and verify RED** - -Run: - -```bash -./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true -``` - -Expected: compilation fails because `ScriptRuntime.EngineLease` and `ScriptRuntime.acquire(long)` do not exist. - -- [ ] **Step 3: Implement the lifecycle record and lease** - -Replace `ConcurrentHashMap` with `ConcurrentHashMap`. Keep all lifecycle types in `ScriptRuntime.java`: - -```java -public static final class EngineLease implements AutoCloseable { - private final EngineRecord record; - private final ScriptRuntime runtime; - private boolean closed; - - private EngineLease(EngineRecord record, ScriptRuntime runtime) { - this.record = record; - this.runtime = runtime; - } - - public ScriptRuntime runtime() { - return runtime; - } - - @Override - public void close() { - synchronized (this) { - if (closed) { - return; - } - closed = true; - } - record.release(); - } -} -``` - -The record owns `State { LIVE, CLOSING, DESTROYED }`, `activeLeases`, `tryAcquire()`, `closeAndAwait()`, and `release()`. The state check and increment are one synchronized action. `closeAndAwait()` keeps waiting after `InterruptedException`, sets `DESTROYED` only after `activeLeases == 0`, and restores the interrupt bit after leaving the monitor. - -`destroy(handle)` obtains the record, invokes `closeAndAwait()`, and removes exactly that record with `REGISTRY.remove(handle, record)`. Do not remove the record before draining because concurrent destroy callers need the same coordination object. - -Make `register` reject null runtimes and ensure generated handles remain positive. If `NEXT_HANDLE.getAndIncrement()` returns a non-positive value after overflow, fail registration rather than publishing an invalid ABI handle. - -- [ ] **Step 4: Update existing registry tests to use leases** - -Replace every `ScriptRuntime.get(handle)` in `ScriptRuntimeTest` with scoped acquisition: - -```java -try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(hA)) { - assertNotNull(lease); - assertEquals("\"A:X\"", Result.parse(lease.runtime().run(IMPORT_A)).result); -} -``` - -Use `assertNull(ScriptRuntime.acquire(handle))` for absent or destroyed handles. Delete the Javadoc claim that checking `UNKNOWN_ENGINE_HANDLE_JSON` and `get()` verifies the complete entrypoint contract; retain only the exact-string assertion. - -- [ ] **Step 5: Run focused and module Java tests and verify GREEN** - -Run: - -```bash -./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true -./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeTest" -PskipNodeTests=true -PskipPythonTests=true -``` - -Expected: both commands exit `0`; the new lifecycle class reports all tests passed. - -- [ ] **Step 6: Commit the core lease** - -```bash -git add native-lib/src/main/java/org/mule/weave/lib/ScriptRuntime.java \ - native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeLifecycleTest.java \ - native-lib/src/test/java/org/mule/weave/lib/ScriptRuntimeTest.java -git commit -m "fix(native-lib): lease engines across admitted operations" -``` - -### Task 2: C Entrypoint Exception Sentinels and Lease Adoption - -**Files:** -- Create: `native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java` -- Create: `native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java` -- Modify: `native-lib/src/main/java/org/mule/weave/lib/NativeLib.java:39-45,540-665` -- Modify: `native-lib/node/src/addon.c:2269-2273,2409-2412` - -**Interfaces:** -- Consumes: `ScriptRuntime.acquire(long)` and `EngineLease` from Task 1. -- Produces: explicit `ReturnZero`, `ReturnNullPointer`, and `ReturnVoid` C-entrypoint handlers; every run entrypoint is leased. - -- [ ] **Step 1: Write reflection tests for all exported handlers** - -Create `NativeLibEntryPointContractTest`. Use `NativeLib.class.getDeclaredMethod(...)` and `getAnnotation(CEntryPoint.class)` to assert exact handlers: - -```java -assertEquals( - CEntryPointExceptionHandlers.ReturnZero.class, - annotation("createEngine", IsolateThread.class).exceptionHandler()); -assertEquals( - CEntryPointExceptionHandlers.ReturnNullPointer.class, - annotation("runScriptEngine", IsolateThread.class, long.class, - CCharPointer.class, CCharPointer.class).exceptionHandler()); -assertEquals( - CEntryPointExceptionHandlers.ReturnVoid.class, - annotation("destroyEngine", IsolateThread.class, long.class).exceptionHandler()); -``` - -Cover all seven exports, including `freeCString` and both callback run entrypoints. - -- [ ] **Step 2: Run the handler contract test and verify RED** - -Run: - -```bash -./gradlew native-lib:test --tests "org.mule.weave.lib.NativeLibEntryPointContractTest" -PskipNodeTests=true -PskipPythonTests=true -``` - -Expected: compilation fails because `CEntryPointExceptionHandlers` does not exist. - -- [ ] **Step 3: Add allocation-free GraalVM handlers** - -Create the support class using `com.oracle.svm.core.Uninterruptible`: - -```java -final class CEntryPointExceptionHandlers { - private CEntryPointExceptionHandlers() { - } - - static final class ReturnZero implements CEntryPoint.ExceptionHandler { - @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") - static long handle(Throwable ignored) { - return 0L; - } - } - - static final class ReturnNullPointer implements CEntryPoint.ExceptionHandler { - @Uninterruptible(reason = "Return an ABI sentinel after an entrypoint exception") - static CCharPointer handle(Throwable ignored) { - return WordFactory.nullPointer(); - } - } - - static final class ReturnVoid implements CEntryPoint.ExceptionHandler { - @Uninterruptible(reason = "Contain an exception at the C ABI boundary") - static void handle(Throwable ignored) { - } - } -} -``` - -Each nested handler class must declare exactly one method. Do not allocate, log, or throw in these methods. - -- [ ] **Step 4: Apply handlers and leases to all entrypoints** - -Annotate every export with the intended `exceptionHandler`. Change each run entrypoint from `ScriptRuntime.get(handle)` to: - -```java -try (ScriptRuntime.EngineLease lease = ScriptRuntime.acquire(handle)) { - if (lease == null) { - return toUnmanagedCString(UNKNOWN_ENGINE_HANDLE_JSON); - } - // Convert required pointers and execute through lease.runtime(). -} -``` - -Validate required pointer and callback arguments before dereference. Return a non-null `success:false` envelope for expected invalid arguments when possible. Keep null-pointer result only as the exception-handler fallback. - -The lease must include the callback loop, feeder join, and final result allocation. Keep `destroyEngine` idempotent and blocking through `ScriptRuntime.destroy(handle)`. - -Update the two addon comments so they state that `0` is the explicit Java ABI exception sentinel, not GraalVM default-value behavior. - -- [ ] **Step 5: Run Java tests and verify GREEN** - -Run: - -```bash -./gradlew native-lib:test --tests "org.mule.weave.lib.NativeLibEntryPointContractTest" -PskipNodeTests=true -PskipPythonTests=true -./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeLifecycleTest" -PskipNodeTests=true -PskipPythonTests=true -``` - -Expected: both commands exit `0`. - -- [ ] **Step 6: Build the native library and inspect generated exports** - -Run with the checked-in GraalVM: - -```bash -export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" -export JAVA_HOME="$GRAALVM_HOME" -./gradlew native-lib:nativeCompile -PskipStripDebug=true -``` - -Expected: exit `0`; Native Image accepts all custom handlers. Verify `native-lib/build/native/nativeCompile/dwlib.h` still exports the same seven C names with unchanged signatures. - -- [ ] **Step 7: Commit the ABI containment** - -```bash -git add native-lib/src/main/java/org/mule/weave/lib/CEntryPointExceptionHandlers.java \ - native-lib/src/main/java/org/mule/weave/lib/NativeLib.java \ - native-lib/src/test/java/org/mule/weave/lib/NativeLibEntryPointContractTest.java \ - native-lib/node/src/addon.c -git commit -m "fix(native-lib): contain C entrypoint exceptions" -``` - -### Task 3: Raw ABI Native Regression Tests - -**Files:** -- Modify: `native-lib/python/tests/integration/test_lifecycle.py` - -**Interfaces:** -- Consumes: native sentinels from Task 2 and blocking `destroy_engine` lease semantics from Task 1. -- Produces: isolated subprocess tests for process survival and resolver-context drain. - -- [ ] **Step 1: Add null-resolver subprocess coverage** - -Add a helper that runs Python code with `DATAWEAVE_NATIVE_LIB` and the package source on `PYTHONPATH`. In the child, bind the raw ABI with `ctypes`, create an isolate, detach bootstrap, attach the current thread, and call: - -```python -null_resolver = ctypes.cast(None, RESOLVE_MODULE_CALLBACK) -handle = lib.create_engine_with_resolver(thread, null_resolver, None) -assert handle == 0 -healthy = lib.create_engine(thread) -assert healthy > 0 -null_result = lib.run_script_engine(thread, healthy, None, None) -assert not null_result -lib.destroy_engine(thread, healthy) -``` - -The child must not call `free_cstring` for the null result pointer. It then cleans up/detaches correctly and prints one JSON object. Parent assertions require exit `0`, `handle == 0`, a valid later handle, a null run result, and no `Fatal error` in stderr. - -- [ ] **Step 2: Add a resolver-context lease/drain subprocess test** - -Use two OS threads and direct ctypes, bypassing `NativeRuntime` serialization: - -```python -resolver_entered = Event() -release_resolver = Event() -destroy_returned = Event() - -@RESOLVE_MODULE_CALLBACK -def resolver(_thread, _ctx, _path): - resolver_entered.set() - assert release_resolver.wait(5) - return module_source_address -``` - -Thread A attaches and calls `run_script_engine` on a resolver-backed engine. Thread B attaches only after `resolver_entered`, calls `destroy_engine`, and sets `destroy_returned` afterward. Assert in the parent process logic that `destroy_returned.wait(0.1)` is false before releasing the callback, then true after run completion. Retain the resolver source buffer until destroy returns. - -- [ ] **Step 3: Prove the new tests fail against an unmodified base-branch native library** - -Create a disposable verification worktree at the base branch and build its native library: - -```bash -git worktree add --detach \ - "/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red" \ - w-23692110-multi-engine-design -GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" \ -JAVA_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" \ - ./gradlew native-lib:nativeCompile -PskipStripDebug=true -``` - -Run the fix branch's new tests against that base library: - -```bash -cd native-lib/python -DATAWEAVE_NATIVE_LIB="/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red/native-lib/build/native/nativeCompile/dwlib.dylib" \ - python3 -m pytest tests/integration/test_lifecycle.py -k "raw_abi" -q -``` - -Expected on the old implementation: the null-resolver child exits `99`, and the destroy-drain assertion reports that destroy returned while the resolver remained blocked. Remove the disposable worktree afterward with `git worktree remove "/var/folders/n2/069kfxz14k3dg0dctt0gblm80000gn/T/opencode/data-weave-cli-review22-red"`; do not edit or commit from it. - -- [ ] **Step 4: Run the native tests against the fixed library and verify GREEN** - -Run: - -```bash -export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" -export JAVA_HOME="$GRAALVM_HOME" -./gradlew native-lib:nativeCompile -PskipStripDebug=true -cd native-lib/python -DATAWEAVE_NATIVE_LIB="../build/native/nativeCompile/dwlib.dylib" \ - python3 -m pytest tests/integration/test_lifecycle.py -k "raw_abi" -q -``` - -Expected: child processes exit `0`; destroy remains blocked until the callback is released. - -- [ ] **Step 5: Commit the raw ABI regressions** - -```bash -git add native-lib/python/tests/integration/test_lifecycle.py -git commit -m "test(native-lib): cover C ABI failure and lease contracts" -``` - -### Task 4: Python Callback Reentrancy Guard - -**Files:** -- Modify: `native-lib/python/src/dataweave/native.py:1-12,344-623` -- Modify: `native-lib/python/src/dataweave/runtime.py:44-75,122-134,225-249,263-287` -- Modify: `native-lib/python/tests/unit/test_native.py` -- Modify: `native-lib/python/tests/unit/test_streaming.py:98-175` -- Modify: `native-lib/python/tests/integration/test_module_resolver.py:240-323` - -**Interfaces:** -- Consumes: existing `DataWeaveError` and callback wrappers. -- Produces: `_native_callback_scope()`, `_raise_if_native_callback_active()`, and process-wide thread-local callback depth. - -- [ ] **Step 1: Add isolated cross-engine resolver reentry test** - -Add a child-process test next to `test_overlapping_resolver_aware_runs_are_serialized`. The child initializes `inner` and resolver-backed `outer`; the outer resolver calls `inner.run("40 + 2")`, catches `DataWeaveError`, and returns `%dw 2.0\nfun answer() = 42`. Print JSON containing nested error type/message and outer result. - -Parent assertions: - -```python -assert completed.returncode == 0 -assert response["nested_type"] == "DataWeaveError" -assert "native callback" in response["nested_error"].lower() -assert response["outer"] == {"success": True, "value": "42"} -assert "Fatal error" not in completed.stderr -``` - -- [ ] **Step 2: Run the integration test and verify RED** - -Run against the staged native library: - -```bash -cd native-lib/python -python3 -m pytest tests/integration/test_module_resolver.py -k "cross_engine_resolver_reentry" -q -``` - -Expected: child exits `99` with GraalVM thread-state fatal stderr. - -- [ ] **Step 3: Add unit tests for shared thread-local guard behavior** - -Test that callback scope on one thread rejects `capture_operation`, `initialize`, and `cleanup` on any instance on that same thread, but does not reject another Python thread. Extend write/read callback tests so reentry through a second `DataWeave` instance returns callback status `-1` and never calls its native attach function. - -- [ ] **Step 4: Implement callback TLS and guard all isolate-touching public paths** - -Import `local` and define: - -```python -_native_callback_state = local() - -def _raise_if_native_callback_active() -> None: - if getattr(_native_callback_state, "depth", 0) > 0: - raise DataWeaveError( - "DataWeave lifecycle and execution are not allowed from a native callback on the same thread." - ) - -@contextmanager -def _native_callback_scope(): - previous = getattr(_native_callback_state, "depth", 0) - _native_callback_state.depth = previous + 1 - try: - yield - finally: - if previous == 0: - del _native_callback_state.depth - else: - _native_callback_state.depth = previous -``` - -Wrap only the direct user callback invocation, not parsing or diagnostics. Apply it to resolver, public write callback, and public read callback wrappers. Check `_raise_if_native_callback_active()` before lifecycle locks and before serialized native admission. Do not hold `_isolate_lock`, `_resolver_lock_global`, or an instance operation lock while user code runs. - -- [ ] **Step 5: Run focused Python tests and verify GREEN** - -```bash -cd native-lib/python -python3 -m pytest tests/unit/test_native.py -k "native_callback" -q -python3 -m pytest tests/unit/test_streaming.py -k "reentry" -q -python3 -m pytest tests/integration/test_module_resolver.py -k "cross_engine_resolver_reentry" -q -``` - -Expected: all selected tests pass; the child process remains alive. - -- [ ] **Step 6: Commit the Python callback guard** - -```bash -git add native-lib/python/src/dataweave/native.py \ - native-lib/python/src/dataweave/runtime.py \ - native-lib/python/tests/unit/test_native.py \ - native-lib/python/tests/unit/test_streaming.py \ - native-lib/python/tests/integration/test_module_resolver.py -git commit -m "fix(python): reject native callback reentrancy" -``` - -### Task 5: Python Atomic Admission and Generation-Bound Streams - -**Files:** -- Modify: `native-lib/python/src/dataweave/native.py:344-492,552-600` -- Modify: `native-lib/python/src/dataweave/runtime.py:87-103,109-223,251-287` -- Modify: `native-lib/python/tests/unit/test_native.py` -- Modify: `native-lib/python/tests/unit/test_streaming.py` -- Modify: `native-lib/python/tests/integration/test_streaming.py` - -**Interfaces:** -- Consumes: `_raise_if_native_callback_active()` from Task 4. -- Produces: frozen `_EngineOperation(handle, generation)`, `capture_operation()`, `validate_operation()`, and token-taking native run methods. - -- [ ] **Step 1: Write deterministic paused-admission test** - -Initialize a wrapper with `FakeLibrary`. Patch `_require_initialized` so it captures and returns generation A, signals an event, and waits. While the run thread is paused, cleanup and reinitialize to generation B, then resume. Assert `DataWeaveError` contains `stale engine generation` and the fake `run_script_engine` was never called with B. - -Add a complementary test that blocks after serialized admission, starts cleanup, and proves the admitted operation uses A before cleanup destroys A. - -- [ ] **Step 2: Write stale stream tests before implementation** - -Parameterize `run_streaming` and `run_transform`: - -```python -stream = create_stream(runtime) -old_handle = runtime._native.handle -runtime.cleanup() -runtime.initialize() -assert runtime._native.handle != old_handle -with pytest.raises(dataweave.DataWeaveError, match="stale engine generation"): - next(stream) -``` - -Assert no worker registered, no attach occurred for the stale stream, and no callback entrypoint received the replacement handle. Add a handle-reuse variant where the fake returns the same numeric handle for both generations. - -- [ ] **Step 3: Run focused tests and verify RED** - -```bash -cd native-lib/python -python3 -m pytest tests/unit/test_native.py -k "admission_generation" -q -python3 -m pytest tests/unit/test_streaming.py -k "stale_generation" -q -``` - -Expected: the paused run executes replacement handle B, and old streams execute instead of raising. - -- [ ] **Step 4: Implement immutable operation tokens** - -Add: - -```python -@dataclass(frozen=True) -class _EngineOperation: - handle: int - generation: int -``` - -`NativeRuntime` keeps monotonic `_generation` and canonical `_engine_operation`. Publish a new token only after successful engine creation. Clear `_engine_operation` during cleanup but never reset `_generation`. - -`capture_operation()` fails when uninitialized and returns the immutable token. `_serialized_native_operation(expected)` checks callback TLS before waiting, acquires the existing per-instance lock, validates exact token equality, and yields the token. All native run methods accept `operation` explicitly and call `operation.handle`; none reads `self.handle` after admission. - -- [ ] **Step 5: Bind public calls and worker registration to captured tokens** - -Change `_require_initialized` to return `_EngineOperation`. Capture it in `run`, `run_callback`, `run_input_output_callback`, `run_streaming`, and `run_transform` before constructing callbacks or generators. - -Change `_stream_worker(operation, invoke, cancelled)` and `_register_stream_worker(worker, operation)`. Under `_stream_workers_lock`, reject `_cleaning_up`, validate the operation, then register. Native worker closures carry `operation` into the token-taking native method. - -Keep lock order `_stream_workers_lock -> brief token validation`; native execution must release its operation lock before `_unregister_stream_worker` takes the worker lock. - -- [ ] **Step 6: Run unit and real-native stale-stream tests and verify GREEN** - -```bash -cd native-lib/python -python3 -m pytest tests/unit/test_native.py -k "admission_generation" -q -python3 -m pytest tests/unit/test_streaming.py -k "stale_generation" -q -python3 -m pytest tests/integration/test_streaming.py -k "stale_generation" -q -``` - -Expected: stale operations fail before replacement-handle invocation; admitted work stays on its captured engine. - -- [ ] **Step 7: Run the complete Python unit lane** - -```bash -cd native-lib/python -python3 -m pytest -m unit -q -``` - -Expected: all unit tests pass. Update `configured_runtime` test helpers to initialize `_generation` and `_engine_operation` explicitly rather than weakening production fallbacks. - -- [ ] **Step 8: Commit Python generations** - -```bash -git add native-lib/python/src/dataweave/native.py \ - native-lib/python/src/dataweave/runtime.py \ - native-lib/python/tests/unit/test_native.py \ - native-lib/python/tests/unit/test_streaming.py \ - native-lib/python/tests/integration/test_streaming.py -git commit -m "fix(python): bind operations to engine generations" -``` - -### Task 6: Node Callback Reentrancy Guard - -**Files:** -- Modify: `native-lib/node/src/addon.c:31-59,2094-2231,2235-2719,3373-3453` -- Modify: `native-lib/node/src/ffi.ts` -- Modify: `native-lib/node/src/dataweave.ts:73-139,229-323` -- Create: `native-lib/node/tests/integration/resolver-reentrancy.test.ts` -- Create: `native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs` - -**Interfaces:** -- Consumes: existing N-API methods and `DataWeaveError`. -- Produces: addon error code `ERR_DATAWEAVE_CALLBACK_REENTRANCY` and TypeScript `callNative()` error mapping. - -- [ ] **Step 1: Write the child-process resolver reentry regression** - -The fixture creates inner and outer `DataWeave` instances. The resolver attempts `inner.run`, catches the error, returns a valid module, prints JSON, and cleans both instances in `finally`. Parent asserts exit `0`, nested error name `DataWeaveError`, outer result `42`, and no Graal fatal stderr. - -Add a raw-addon case in the same fixture that recursively calls `runScriptEngine` from `createEngineWithResolver` and records the addon's stable error code. - -- [ ] **Step 2: Run the resolver test and verify RED** - -```bash -cd native-lib/node -npm run test:integration -- tests/integration/resolver-reentrancy.test.ts -``` - -Expected: child exits `99` with `Must either be at a safepoint or in native mode`. - -- [ ] **Step 3: Implement OS-thread-local callback depth in the addon** - -Initialize a `uv_key_t` in the existing `uv_once` initializer. Add helpers: - -```c -static unsigned native_callback_depth(void); -static void native_callback_enter(void); -static void native_callback_exit(void); -static bool native_callback_active(void); -static napi_value throw_callback_reentrancy(napi_env env); -``` - -Wrap `napi_call_function` in `resolve_module_callback`, `call_js_read`, and output callback bridges with enter/exit on all statuses. Reject callback reentry at the start of create, create-with-resolver, destroy, synchronous run, stream start, transform start, and cleanup before any state mutation or attachment. Set JavaScript error property `code` to `ERR_DATAWEAVE_CALLBACK_REENTRANCY` before throwing. - -- [ ] **Step 4: Map the native error to DataWeaveError** - -In `ffi.ts`, normalize calls through: - -```ts -function callNative(invoke: () => T): T { - try { - return invoke(); - } catch (error) { - if (error && typeof error === "object" && - "code" in error && error.code === "ERR_DATAWEAVE_CALLBACK_REENTRANCY") { - throw new DataWeaveError(String((error as Error).message)); - } - throw error; - } -} -``` - -Apply it to every isolate-touching wrapper. Avoid wrapping promise rejections twice; only normalize synchronous native admission errors. - -- [ ] **Step 5: Build and run resolver tests and verify GREEN** - -```bash -cd native-lib/node -npm run build:addon -npm run build:ts -npm run test:integration -- tests/integration/resolver-reentrancy.test.ts -``` - -Expected: build exits `0`; child processes survive and return the intended typed errors. - -- [ ] **Step 6: Commit the Node callback guard** - -```bash -git add native-lib/node/src/addon.c native-lib/node/src/ffi.ts \ - native-lib/node/src/dataweave.ts \ - native-lib/node/tests/integration/resolver-reentrancy.test.ts \ - native-lib/node/tests/integration/fixtures/resolver-reentrancy.cjs -git commit -m "fix(node): reject native callback reentrancy" -``` - -### Task 7: Node Generation-Bound Lazy Streams - -**Files:** -- Modify: `native-lib/node/src/dataweave.ts:49-55,81-139,186-216,229-323` -- Modify: `native-lib/node/tests/unit/dataweave-initialize.test.ts` -- Modify: `native-lib/node/tests/integration/instance-lifecycle.test.ts` - -**Interfaces:** -- Consumes: existing handle-based FFI and `DataWeaveError`. -- Produces: internal `EngineOperationToken`, `captureOperationToken()`, `assertCurrentOperation(token)`, and private token-taking async generators. - -- [ ] **Step 1: Add unit tests for stale streams and handle reuse** - -Mock FFI to return handle `2`, create `runStreaming()` without iterating, cleanup, initialize with handle `3`, and assert first pull rejects before `runScriptStreamingEngine` is called. Repeat with handle `2` reused; generation must still reject. Mirror both cases for `runTransform`. - -Add an async transform-input test that pauses `createChunkReader` consumption, performs cleanup/reinitialize, resumes input, and asserts no transform native call. - -- [ ] **Step 2: Run the Node generation tests and verify RED** - -```bash -cd native-lib/node -npm run test:unit -- tests/unit/dataweave-initialize.test.ts -t "stale engine generation" -``` - -Expected: old streams call FFI with the replacement handle or otherwise fail the no-call assertion. - -- [ ] **Step 3: Implement immutable Node tokens** - -Add: - -```ts -interface EngineOperationToken { - readonly handle: number; - readonly generation: number; -} -``` - -Increment `engineGeneration` only after successful handle publication. Never reset it in cleanup. `captureOperationToken()` checks readiness and returns current identity. `assertCurrentOperation()` requires state ready, exact handle, and exact generation; stale work throws `DataWeaveError("DataWeave operation belongs to a stale engine generation.")`. - -Convert public stream APIs to ordinary methods: - -```ts -runStreaming(...): AsyncGenerator { - const token = this.captureOperationToken(); - return this.runStreamingInternal(token, script, inputs); -} -``` - -Private async generators validate immediately before native admission and use `token.handle`. `runTransformInternal` validates before and after async input pre-buffering. - -- [ ] **Step 4: Add real-native stale stream coverage** - -In `instance-lifecycle.test.ts`, hold an anchor instance initialized so Java handles do not reset with isolate teardown. Create target stream under old handle, cleanup/reinitialize target, then consume old stream and assert `DataWeaveError`; a new stream must still succeed. Cover stream and transform. - -- [ ] **Step 5: Run focused tests and typecheck and verify GREEN** - -```bash -cd native-lib/node -npm run test:unit -- tests/unit/dataweave-initialize.test.ts -npm run test:integration -- tests/integration/instance-lifecycle.test.ts -npm run build:ts -``` - -Expected: all commands exit `0` and FFI receives captured handles only. - -- [ ] **Step 6: Commit Node generations** - -```bash -git add native-lib/node/src/dataweave.ts \ - native-lib/node/tests/unit/dataweave-initialize.test.ts \ - native-lib/node/tests/integration/instance-lifecycle.test.ts -git commit -m "fix(node): bind lazy streams to engine generations" -``` - -### Task 8: TypeScript Streaming Operation and Consumer Credits - -**Files:** -- Modify: `native-lib/node/src/ffi.ts:4-27,58-87` -- Modify: `native-lib/node/src/stream.ts` -- Modify: `native-lib/node/src/dataweave.ts:49-56,160-216,254-313` -- Modify: `native-lib/node/tests/unit/stream.test.ts` -- Modify: `native-lib/node/tests/unit/dataweave-initialize.test.ts` - -**Interfaces:** -- Consumes: token-taking stream methods from Task 7. -- Produces: internal `NativeStreamingOperation` with `completion`, `acknowledge`, `cancel`, and `close`; active-operation cleanup tracking. - -- [ ] **Step 1: Rewrite unit fakes around a streaming controller and add credit assertions** - -Define a test helper: - -```ts -function operation(completion: Promise) { - return { - completion, - acknowledge: vi.fn(), - cancel: vi.fn(), - close: vi.fn(), - }; -} -``` - -Add tests proving: - -- pushing a chunk into the JS queue does not acknowledge it; -- the first `.next()` acknowledges exactly that chunk's byte length before yielding; -- draining buffered chunks acknowledges each once; -- `generator.return(undefined)` cancels and closes once; -- native rejection still acknowledges already-buffered chunks before throwing; -- zero-chunk completion closes without cancellation; -- cancel/close paths are idempotent. - -Rename the current test containing `(backpressure)` to describe parking/wakeup only. - -- [ ] **Step 2: Run stream unit tests and verify RED** - -```bash -cd native-lib/node -npm run test:unit -- tests/unit/stream.test.ts -``` - -Expected: TypeScript compilation fails because the start callback still returns `Promise` and has no credit methods. - -- [ ] **Step 3: Implement the internal controller contract** - -Export from `ffi.ts`: - -```ts -export interface NativeStreamingOperation { - readonly completion: Promise; - acknowledge(bytes: number): void; - cancel(): void; - close(): void; -} -``` - -Change addon method typings and wrappers to return this object for stream and transform. Change `StartStreaming` accordingly. - -In `streamFromNative`, call `start` once, attach completion handlers, acknowledge immediately when dequeuing, and use `try/finally` to cancel only when iteration ended before native completion. Always `close()` exactly once after settlement/finalization. When abandoning buffered chunks, acknowledge their lengths before clearing them. - -- [ ] **Step 4: Track and cancel active operations during DataWeave cleanup** - -Add `activeStreams: Set`. Register an operation when native start returns and unregister it from `streamFromNative`'s close callback. In `doCleanup()`, synchronously cancel a snapshot before `ffi.destroyEngine`; await their completion settlements without masking the primary destroy/cleanup error. - -Do not start native work at stream method call. Registration still occurs on first iteration after generation validation. - -- [ ] **Step 5: Run TypeScript tests and verify GREEN** - -```bash -cd native-lib/node -npm run test:unit -- tests/unit/stream.test.ts -npm run test:unit -- tests/unit/dataweave-initialize.test.ts -npm run build:ts -``` - -Expected: all commands exit `0`. - -- [ ] **Step 6: Commit the TypeScript flow contract** - -```bash -git add native-lib/node/src/ffi.ts native-lib/node/src/stream.ts \ - native-lib/node/src/dataweave.ts \ - native-lib/node/tests/unit/stream.test.ts \ - native-lib/node/tests/unit/dataweave-initialize.test.ts -git commit -m "fix(node): propagate streaming consumer credits" -``` - -### Task 9: Native Node Output Flow Control - -**Files:** -- Modify: `native-lib/node/src/addon.c:1124-2092,3378-3453` -- Create: `native-lib/node/tests/integration/stream-backpressure.test.ts` - -**Interfaces:** -- Consumes: TypeScript controller contract from Task 8. -- Produces: `output_flow_t`, finite TSFN queues, native acknowledge/cancel/close methods, and test-only flow statistics. - -- [ ] **Step 1: Add integration tests that inspect a paused producer** - -Extend the test-only addon interface with flow stats and fixed watermarks. Start a large `deferred=true` output, consume one chunk, pause, and poll until `paused` is true. Assert: - -```ts -expect(stats.peakBufferedChunks).toBeLessThanOrEqual(stats.highChunks + 1); -expect(stats.peakBufferedBytes).toBeLessThanOrEqual( - stats.highBytes + stats.largestChunkBytes -); -expect(completionSettled).toBe(false); -``` - -Resume slowly and verify complete ordered output and successful metadata. Mirror the pause/drain assertion for `runTransform`. Add early-return and cleanup-while-paused tests with bounded timeouts. - -- [ ] **Step 2: Run the backpressure test and verify RED** - -```bash -cd native-lib/node -npm run test:integration -- tests/integration/stream-backpressure.test.ts -``` - -Expected: native controller/stat hooks are missing, or peak buffering exceeds the intended bound. - -- [ ] **Step 3: Implement `output_flow_t` and lifetime rules** - -Add fixed constants: - -```c -#define OUTPUT_HIGH_BYTES (1024 * 1024) -#define OUTPUT_LOW_BYTES (512 * 1024) -#define OUTPUT_HIGH_CHUNKS 128 -#define OUTPUT_LOW_CHUNKS 64 -#define OUTPUT_TSFN_QUEUE_SIZE 129 -``` - -`output_flow_t` contains mutex, condition, outstanding/peak counters, largest chunk, paused/cancelled/done flags, and refcount. Implement create/retain/release/reserve/acknowledge/cancel/mark_done. `reserve` waits only on the native producer thread and admits one oversized chunk when the window is empty. - -Each `chunk_data` records its flow pointer and accounted length. Reserve before allocation/enqueue; on OOM or TSFN enqueue failure, release the reserved credit. Non-sentinel JS callbacks keep credit outstanding after copying into a Buffer. Sentinel and env-dead paths cancel/settle and release ownership exactly once. - -- [ ] **Step 4: Return native controller objects** - -Instead of returning the bare promise, create an object with named properties/methods: - -```text -completion: Promise -acknowledge(bytes): void -cancel(): void -close(): void -``` - -Each method resolves the operation flow through N-API external data or a finalizer-safe holder. Validate bytes as a non-negative integer and make cancel/close idempotent. The holder keeps the flow alive until both worker and JS controller release it. - -Use finite output TSFN queues for streaming and transform writes. Keep transform read TSFN behavior unchanged. - -- [ ] **Step 5: Make cancellation unblock every producer path** - -Cancellation broadcasts the flow condition. Write callbacks return `-1` after cancellation. Generator abandonment, cleanup, env teardown, JS callback allocation/call failure, and controller finalization all route through the same idempotent cancel. Never wait on a flow condition while holding `g_mutex`; release the flow mutex before bridge/global completion accounting. - -- [ ] **Step 6: Run addon, focused unit, and integration tests and verify GREEN** - -```bash -cd native-lib/node -npm run build:addon -npm run build:ts -npm run test:unit -- tests/unit/stream.test.ts -npm run test:integration -- tests/integration/stream-backpressure.test.ts -npm run test:integration -- tests/integration/teardown-deadlock.test.ts -``` - -Expected: all commands exit `0`; paused producer counters remain bounded; early return and cleanup do not hang. - -- [ ] **Step 7: Commit native backpressure** - -```bash -git add native-lib/node/src/addon.c \ - native-lib/node/tests/integration/stream-backpressure.test.ts -git commit -m "fix(node): bound asynchronous output buffering" -``` - -### Task 10: Detach-Poison Failure Injection and Fail-Closed Admission - -**Files:** -- Modify: `native-lib/node/src/addon.c:135-154,237-297,455-505,1241-1341,1767-1866,2235-2719,2761-2876,3378-3453` -- Create: `native-lib/node/tests/integration/detach-poison-hook.test.ts` -- Create: `native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs` -- Create: `native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs` -- Modify: `native-lib/node/vitest.config.ts:27-33` - -**Interfaces:** -- Consumes: existing `g_isolate_poisoned` cleanup behavior. -- Produces: `detach_thread_checked(detach_site_t, void*)`, one-shot fault hooks, counters, and rejection of new work on a poisoned isolate. - -- [ ] **Step 1: Add child-process poison tests** - -Synchronous fixture sequence: - -```text -initialize -> create engine -> arm sync-run detach failure -> run succeeds --> poison is true -> new run/create rejects -> destroy + cleanup settle --> teardown count unchanged, abandon count +1 -> initialize fresh -> run succeeds -``` - -Transform fixture arms the transform-worker site, starts final cleanup while the operation is active, completes the worker, and asserts cleanup skips teardown and resolves. Parent tests enforce timeouts and reject any exit `99`, signal, or fatal stderr. - -- [ ] **Step 2: Run the poison tests and verify RED** - -```bash -cd native-lib/node -npm run test:integration -- tests/integration/detach-poison-hook.test.ts -``` - -Expected: required test hooks are undefined. - -- [ ] **Step 3: Centralize ordinary detach calls** - -Add `detach_site_t` entries for bridge finalize, stream worker, transform worker, create engine, create rollback, resolver create, unknown destroy, and sync run. Replace the eight ordinary detach sites with `detach_thread_checked(site, thread)`. Leave teardown-helper follow-up detach calls direct because they classify teardown-plus-detach double failure rather than ordinary operation poison. - -Under test hooks, call real detach first and substitute nonzero only when the selected one-shot site is armed and real detach succeeded. Count forced failures, isolate creates, teardown calls, and abandon operations under `g_mutex`. - -- [ ] **Step 4: Reject new admission after poison** - -In create/run/stream/transform admission critical sections, add `g_isolate_poisoned` to the rejection condition. Use a stable error `DataWeave isolate is unavailable after a thread detach failure; clean up and initialize again.` Existing admitted work continues draining; final cleanup follows `CLEANUP_UNRECOVERABLE` and abandons published state. - -- [ ] **Step 5: Export test-only hooks and stats** - -When `DATAWEAVE_TEST_HOOKS` is enabled, export: - -```text -__test_forceDetachFailureOnce(site) -__test_isolatePoisoned() -__test_isolateCreationCount() -__test_teardownCallCount() -__test_abandonedIsolateCount() -``` - -Reject unknown site strings synchronously. Update the Vitest comment listing enabled hooks. - -- [ ] **Step 6: Build and run poison/lifecycle tests and verify GREEN** - -```bash -cd native-lib/node -npm run build:addon -npm run test:integration -- tests/integration/detach-poison-hook.test.ts -npm run test:integration -- tests/integration/engine-strand-hook.test.ts -npm run test:integration -- tests/integration/instance-lifecycle.test.ts -``` - -Expected: all commands exit `0`; cleanup never hangs; fresh isolate recovery succeeds. - -- [ ] **Step 7: Commit detach-poison coverage** - -```bash -git add native-lib/node/src/addon.c native-lib/node/vitest.config.ts \ - native-lib/node/tests/integration/detach-poison-hook.test.ts \ - native-lib/node/tests/integration/fixtures/detach-poison-sync.cjs \ - native-lib/node/tests/integration/fixtures/detach-poison-transform.cjs -git commit -m "test(node): inject detach failures across isolate recovery" -``` - -### Task 11: Documentation and Whitespace Contract - -**Files:** -- Modify: `docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md:3-4,369-382,465` -- Modify: `native-lib/python/src/dataweave/native.py:624` -- Modify: `native-lib/README.md` -- Modify: `native-lib/node/README.md` -- Modify: `native-lib/python/README.md` -- Modify: `docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md` - -**Interfaces:** -- Consumes: final behavior from Tasks 1-10. -- Produces: accurate public ABI, lifecycle, reentrancy, streaming, cancellation, and failure-injection documentation. - -- [ ] **Step 1: Remove only the reported whitespace errors** - -Remove trailing spaces from the superseded Node design and remove the extra final blank line in `native.py`. Do not run a repository-wide formatter. - -- [ ] **Step 2: Update raw ABI documentation** - -In `native-lib/README.md`, document: - -```text -create_engine* returns 0 on entrypoint failure. -run_*_engine returns NULL on entrypoint-level failure; do not free NULL. -Normal script failures remain non-NULL JSON envelopes. -destroy_engine closes admission and blocks until admitted operations drain. -Resolver/read/write ctx storage must remain valid through destroy_engine return. -destroy_engine must not be called synchronously from that engine's callback. -``` - -- [ ] **Step 3: Update binding documentation** - -In both binding READMEs, document same-thread callback reentrancy rejection and stale-generation stream behavior. In Node docs, document internal byte/chunk watermarks as implementation details, cleanup cancellation of abandoned streams, and the fact that yielded buffers retained by user code are outside the bound. - -Change the large-file writable example to await `drain`: - -```ts -if (!output.write(chunk)) { - await once(output, "drain"); -} -``` - -Import `once` from `node:events` and preserve existing ESM/CommonJS style in that example. - -- [ ] **Step 4: Update the consolidated design** - -Replace `ConcurrentHashMap`/`ScriptRuntime.get` descriptions with core lifecycle records and leases. Add binding generations, callback TLS, bounded output flow, cancel-on-cleanup, explicit exception sentinels, and detach test hooks. Correct Python stream cleanup wording to match active-worker refusal plus generation-safe registration. - -- [ ] **Step 5: Verify documentation claims against symbols and tests** - -Search all documented C names against `NativeLib.java`, `_bind_abi`, and `addon.c`; remove stale or invented names. Confirm every new error phrase matches production source exactly. - -- [ ] **Step 6: Run whitespace verification and commit** - -```bash -git diff --check w-23692110-multi-engine-design...HEAD -git add docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md \ - docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md \ - native-lib/README.md native-lib/node/README.md native-lib/python/README.md \ - native-lib/python/src/dataweave/native.py -git commit -m "docs(native-lib): document hardened multi-engine contracts" -``` - -Expected: `git diff --check` exits `0` before commit. - -### Task 12: Full Verification, Review, Push, and PR - -**Files:** -- Modify only if verification or review reveals a defect in files already in scope. - -**Interfaces:** -- Consumes: every preceding task. -- Produces: clean, reviewed branch and PR to `w-23692110-multi-engine-design`. - -- [ ] **Step 1: Inspect branch state and commit range** - -```bash -git status --short -git log --oneline --decorate w-23692110-multi-engine-design..HEAD -git diff --stat w-23692110-multi-engine-design...HEAD -git diff --check w-23692110-multi-engine-design...HEAD -``` - -Expected: no unintended tracked changes, focused commits only, and no whitespace errors. - -- [ ] **Step 2: Run complete hosted Java verification** - -```bash -./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true -``` - -Expected: exit `0`. If this task still triggers `nativeCompile` through plugin wiring, use the GraalVM environment from the next step rather than falling back to JDK 17. - -- [ ] **Step 3: Build native library with GraalVM 24** - -```bash -export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" -export JAVA_HOME="$GRAALVM_HOME" -./gradlew native-lib:nativeCompile -PskipStripDebug=true -``` - -Expected: exit `0` with `dwlib.dylib` generated. - -- [ ] **Step 4: Run complete Python unit and integration lane** - -```bash -cd native-lib/python -DATAWEAVE_NATIVE_LIB="../build/native/nativeCompile/dwlib.dylib" \ - python3 -m pytest -m "unit or integration" -q -``` - -Expected: all selected tests pass with no hangs or fatal child exits. - -- [ ] **Step 5: Build and run complete Node unit/integration lanes** - -```bash -cd native-lib/node -npm install -npm run build:addon -npm run build:ts -npm run test:unit -npm run test:integration -``` - -Expected: all commands exit `0`; no unhandled rejections, Worker nonzero exits, or timeout hangs. - -- [ ] **Step 6: Run normal native-lib Gradle verification** - -```bash -export GRAALVM_HOME="/Users/lmariano/dev/mulesoft/data-weave-cli/.graalvm/graalvm-community-openjdk-24.0.2+11.1/Contents/Home" -export JAVA_HOME="$GRAALVM_HOME" -./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true -./gradlew build -PskipNodeTests=true -``` - -Expected: both commands exit `0`. The second command includes the repository's configured Python lane; Node was already run directly to preserve focused output. - -- [ ] **Step 7: Perform three-lens code review** - -Review the complete PR diff for: - -```text -General correctness: stale state, exception masking, duplicate cleanup, error timing. -Native/concurrency: lock order, lease/flow refcounts, callback thread affinity, env death, cancellation. -Security: pointer validation, callback input lengths, tenant data in logs, raw ABI nullability. -``` - -For each accepted finding, add a failing regression test first, implement the smallest correction, rerun the focused test, then rerun the affected module lane. Commit review fixes separately with a concise message. - -- [ ] **Step 8: Verify final clean evidence** - -```bash -git status --short -git diff --check w-23692110-multi-engine-design...HEAD -git log --oneline --decorate w-23692110-multi-engine-design..HEAD -git diff --stat w-23692110-multi-engine-design...HEAD -``` - -Expected: only ignored build artifacts may exist; no uncommitted source/doc changes; diff check exits `0`. - -- [ ] **Step 9: Push the branch** - -```bash -git push -u origin w-23692110-review-22-fixes -``` - -Expected: remote tracking branch created without force. - -- [ ] **Step 10: Create the PR targeting the multi-engine branch** - -Before creation, inspect remote tracking and the full range: - -```bash -git status --short -git branch -vv -git log --oneline origin/w-23692110-multi-engine-design..HEAD -git diff --stat origin/w-23692110-multi-engine-design...HEAD -``` - -Create the PR: - -```bash -gh pr create \ - --base w-23692110-multi-engine-design \ - --head w-23692110-review-22-fixes \ - --title "@W-23692110: Harden multi-engine lifecycle and streaming" \ - --body-file /tmp/pr157-review22-body.md -``` - -The body must summarize all eight resolved findings, list exact verification commands/results, call out the intentional destroy-blocking and callback-reentrancy contracts, and state that this PR layers onto PR #157 rather than targeting `master`. - -- [ ] **Step 11: Report the PR URL and residual risks** - -Return the PR URL, commit count, final test counts, and any unrun platform-only checks. Do not claim hosted CI passes until GitHub reports it. diff --git a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md index c82ff66d..5c1aea7d 100644 --- a/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md +++ b/docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md @@ -1,18 +1,20 @@ # Design: Multiple Isolated DataWeave Engines per Process (native-lib — Node & Python) -**Date:** 2026-08-07 (consolidated 2026-08-25; unified Node + Python 2026-08-26) -**Status:** Approved and implemented on `w-23692110-multi-engine-design` (PR #157) +**Date:** 2026-08-07 (consolidated 2026-08-25; unified Node + Python 2026-08-26; review-22 hardening folded in 2026-09-02) +**Status:** Approved and implemented on `w-23692110-multi-engine-design` (PR #157), with review-22 hardening finalized on `w-23692110-review-22-fixes` **Tracks:** GUS [W-23692110](https://gus.my.salesforce.com/lightning/r/ADM_Work__c/a07EE00002gS7SOYA0/view) — "Native-lib: support multiple DataWeave engine instances with independent module resolvers" **Related:** [docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md](./2026-08-04-nodejs-external-modules-design.md) (the design during which this limitation was discovered) > **About this document.** This is the single, consolidated design for the multi-engine > `native-lib` feature across **both** consumer bindings — Node and Python. It describes the -> **final state** as shipped on PR #157. The core feature (object-level engines behind opaque +> **final state** of PR #157 and its review-22 hardening. The core feature (object-level engines behind opaque > handles, one shared GraalVM isolate) is common to both bindings and is driven through the > identical `*_engine` C ABI. Two substantial bodies of work are folded in here rather than kept > as separate documents: the Node **concurrency & lifecycle model** (§6), hardened across a long > series of code reviews, and the **Python unification** (§7) that removed the `ScriptRuntime` -> singleton and moved Python off its former isolate-per-instance model onto the shared model. +> singleton and moved Python off its former isolate-per-instance model onto the shared model. The +> subsequent review-22 hardening is folded into the shared Java lifecycle, binding admission, Node +> output-flow, and detach-recovery contracts below rather than retained as a second specification. > A provenance map for git archaeology lives in the [Appendix](#appendix-hardening-provenance). > The product-facing `DataWeave` classes are pre-GA, so several internal contracts (async Node > `cleanup()`, the removed `*_with_resolver` and legacy-singleton C ABI) changed during this work @@ -748,27 +750,40 @@ uphold all six: ## 12.1 Final hardening contract -This section records final behavior that is intentionally narrower than a public API guarantee. +This section records review-22 hardening folded into this canonical final-state design. The +implementation details here are intentionally narrower than public API guarantees. Java registry +records use `LIVE`, `CLOSING`, and `DESTROYED` lifecycle states plus admitted-operation leases: +`acquire(handle)` admits only a live record, and `destroy(handle)` closes admission and waits for +existing leases before removal. Every exported Java C entrypoint has an explicit, allocation-free +exception sentinel: engine creation returns `0`, run entrypoints return `NULL`, and void entrypoints +return normally. Those sentinels are distinct from normal script or unknown-handle failures, which +remain allocated, non-`NULL` `success:false` JSON envelopes. + Both bindings use callback thread-local state to reject same-thread resolver/read/write callback -reentry into lifecycle or execution APIs with their public `DataWeaveError`. Both generation-bind -lazy stream and transform work; cleanup or reinitialization before consumption or admission rejects -the stale work rather than allowing it to execute on a replacement engine. +reentry into lifecycle or execution APIs with their public `DataWeaveError`. Both bind operations +to immutable `{handle, generation}` tokens, validate that token immediately at native admission, +and reject stale work after cleanup or reinitialization rather than allowing it to execute on a +replacement engine. Java leases independently preserve raw-ABI engine and callback-context lifetime +after binding admission succeeds. Node's native output bridge bounds its own outstanding bytes and chunks and uses a finite TSFN -queue. Buffers retained by user code after yield are outside that bound. Controller and sequence -credit values are internal correctness machinery, not a public BigInt sequence contract. Node -cleanup cancels and waits for abandoned active streams/transforms before engine destruction. +queue. Each output flow tracks private sequence-bound credits from native enqueue until the consumer +dequeues the chunk; cancellation releases outstanding credits and unblocks a producer. Buffers +retained by user code after yield are outside that bound. Controller and sequence credit values are +internal correctness machinery, not a public BigInt sequence contract. Node cleanup cancels and +waits for abandoned active streams/transforms before engine destruction. Python deliberately differs: it cannot force-cancel an active native call, so cleanup refuses while an active streaming worker is attached. Its registration validation is generation-safe; it does not adopt Node's cancellation behavior. **Test-only detach hooks.** The addon enables fault injection only when `DATAWEAVE_TEST_HOOKS` is -non-empty. A detach hook always performs a real detach first and can synthesize a failure only after -that detach succeeds; it does not model a physically stuck Graal thread. A real detach failure -poisons admission fail-closed. Cleanup abandons the old published generation, and a later fresh -initialization can recover with a new isolate. These hooks and their names are implementation/test -details, not stable APIs. +non-empty. Ordinary operation detaches pass through centralized status handling. A detach hook +always performs a real detach first and can synthesize a failure only after that detach succeeds; it +therefore does not model a physically stuck Graal thread. A nonzero ordinary detach status poisons +the published isolate generation and makes new admission fail closed. Cleanup abandons that +generation safely, and a later initialization can recover with a new isolate. These hooks are +internal, non-public test details, not stable APIs. ## 13. Follow-Up Work @@ -795,10 +810,11 @@ details, not stable APIs. ## Appendix: Hardening provenance -The Node concurrency & lifecycle model (§6) converged over a series of code-review rounds, and the -Python unification (§7) was implemented and reviewed task-by-task; each round's decisions are folded -into the sections above. This map exists only for git archaeology — the per-round and Python -unification design documents were consolidated into this file. +The Node concurrency & lifecycle model (§6) converged over a series of code-review rounds, the +Python unification (§7) was implemented and reviewed task-by-task, and review-22 hardening was +subsequently folded into the same final-state contracts. This map exists only for git archaeology; +the per-round, Python-unification, and review-22 remediation designs were consolidated into this +file. | Round(s) | Area folded into | Decision | |----------|------------------|----------| @@ -818,3 +834,4 @@ unification design documents were consolidated into this file. | Python unification (08-26) | §2, §5, §7, §8 (Layer 1/4), §10–§12 | Remove `ScriptRuntime` singleton + 3 legacy C entrypoints; Python onto shared refcounted isolate + handle engines via `*_engine` ABI; 3-arg ctx resolver trampoline. | | PR157 review 10 (08-27) | §6.3, §6.5, §7.2, §7.4 | Python `_release_isolate`/`_acquire_isolate` retryable-teardown model brought to parity with Node's `g_teardown_needed` (retains the live isolate on failed teardown instead of nulling globals); Node streaming/transform completion sentinel pre-allocated in synchronous setup (worker terminal path now allocation-free, closing a stranded-hang window); stranded-bridge free confirmed conditional on registry removal, with the non-`in_flight`-pinned residual window documented as reachable only via unsupported cross-Worker handle sharing / API misuse; raw `napi_initialize` validates its library-path argument synchronously; user-facing custom-module resolution scope (`run()`-only) documented in both READMEs, cross-referencing the existing streaming-resolver-guard tests. | | Python final review (08-26) | §7.2, §10 | Detach isolate bootstrap thread at create + attach-on-demand so cross-thread last-release teardown cannot hang; unregister resolver token on failed init. | +| PR157 review 22 (09-02) | §8, §10, §11, §12.1 | Java lifecycle records and leases; explicit C-entrypoint exception sentinels distinct from normal JSON errors; Node/Python same-thread callback TLS rejection and generation-safe immediate admission; Node finite-queue, sequence-credit output bounds with cancellation cleanup; Python active-worker cleanup refusal; centralized ordinary-detach poison handling, fail-closed admission, generation-safe abandonment/recovery, and internal real-detach-first fault injection. | diff --git a/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md b/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md deleted file mode 100644 index c84cfed8..00000000 --- a/docs/superpowers/specs/2026-09-02-pr157-review-22-remediation-design.md +++ /dev/null @@ -1,365 +0,0 @@ -# PR #157 Review 22 Remediation Design - -**Date:** 2026-09-02 -**Status:** Approved for implementation -**Base branch:** `w-23692110-multi-engine-design` at `6661b96` -**Fix branch:** `w-23692110-review-22-fixes` -**PR target:** `w-23692110-multi-engine-design` -**Source:** `docs/pr-157-follow-up-code-review-22.md` - -## Goal - -Resolve all eight open findings from PR #157 review 22 without changing valid-input behavior or the public Node and Python APIs. The result must prevent host-process termination, make engine destruction safe for direct C ABI consumers, bind accepted operations to one engine generation, bound Node streaming memory, cover detach-poison recovery, and pass repository whitespace checks. - -## Scope - -In scope: - -- Finding 1: reject same-OS-thread DataWeave lifecycle or execution reentrancy from a native host callback in Node and Python. -- Finding 2: contain every Java `@CEntryPoint` throwable behind an explicit ABI sentinel. -- Finding 3: make Python operation admission atomic with lifecycle generation validation. -- Finding 4: give every Java engine a core-owned lifecycle record and operation leases. -- Finding 5: bind Node and Python lazy streams to the generation present at stream creation. -- Finding 6: bound Node output buffering across both the native TSFN queue and the JavaScript chunk queue. -- Finding 7: add deterministic failure-injection coverage for Node's detach-poison transitions. -- Finding 8: remove the two whitespace errors in the PR diff. -- Update native-lib, Node, Python, and consolidated design documentation for changed contracts. - -Out of scope: - -- Separate GraalVM isolates per engine. -- Enabling custom module resolvers in background streaming or transform workers. -- Changing public `DataWeave` method names, parameters, or result envelopes. -- Retaining compatibility with removed pre-GA singleton C entrypoints. -- Making application-retained output buffers part of the binding's memory bound. -- Guaranteeing bounded total DataWeave runtime memory for non-deferred scripts that materialize output before callback delivery. - -## Global Constraints - -- Use the checked-in `./gradlew` wrapper and GraalVM Community Java 24 for native verification. -- Java remains source/target 17; Scala remains 2.12; Python remains 3.9+; Node remains 18+. -- Preserve exported C names, argument order, callback semantics, and existing JSON wire fields. -- Normal script failures continue to return non-null `{"success":false,...}` envelopes. -- Never let Java, JavaScript, or Python exceptions unwind across C callbacks. -- Every OS thread calling Graal attaches its own isolate thread and detaches afterward unless a successful teardown has invalidated the attachment. -- Node shared C lifecycle state remains guarded by `g_mutex`; per-stream flow state uses its own mutex with explicit lock ordering. -- Python module isolate state remains guarded by `_isolate_lock`; user callbacks run without module locks held. -- All behavior changes use test-first red-green cycles. - -## Architecture - -```mermaid -flowchart TD - A[Public Node or Python call] --> B[Capture immutable engine token] - B --> C{Native callback active on this OS thread?} - C -->|yes| D[Throw DataWeaveError] - C -->|no| E[Binding admission validation] - E --> F[Java core lease acquisition] - F -->|closing or absent| G[Unknown engine envelope] - F -->|live| H[Execute with fixed runtime and handle] - H --> I[Release core lease] - I --> J[Destroy may finish draining] -``` - -The binding token prevents work accepted by one object generation from migrating to a replacement engine. The Java lease independently protects the raw ABI and resolver context even when callers bypass the bindings. These mechanisms are deliberately additive: neither replaces Node bridge pinning, Python instance serialization, or isolate reference accounting. - -## 1. Java C Entrypoint Exception Containment - -### Problem - -The exported methods in `NativeLib` use GraalVM's default `CEntryPoint.FatalExceptionHandler`. A recoverable Java validation error, such as a null resolver callback, therefore prints a fatal error and exits the embedding process instead of returning to C. - -### Design - -Add one package-private `CEntryPointExceptionHandlers` support class containing three nested handler classes. Each nested class declares exactly one static `@Uninterruptible` handler method, performs no allocation or logging, and returns one ABI-category sentinel: - -| Category | Sentinel | Entrypoints | -|---|---|---| -| Engine handle | `0L` | `create_engine`, `create_engine_with_resolver` | -| Result pointer | null `CCharPointer` | all three `run_*_engine` methods | -| Void | return | `destroy_engine`, `free_cstring` | - -Expected validation failures should still be handled in the entrypoint body and represented as ordinary error envelopes when allocation remains safe. The custom exception handler is the non-allocating last resort for unexpected throwables or a failure while constructing an envelope. - -The ABI contract becomes explicit: - -- Engine handles are positive; `0` means creation failed and no engine was registered. -- A null run-result pointer means the entrypoint could not create a JSON result. The caller must not free it. -- Unknown, closing, or destroyed handles return the existing non-null `Unknown engine handle` envelope. -- `free_cstring(NULL)` remains a no-op. - -### Tests - -- A native subprocess calls `create_engine_with_resolver(thread, NULL, NULL)`, observes `0`, then creates and uses a valid engine in the same process. -- Native malformed-pointer coverage observes a null result instead of process exit where a safe deterministic trigger exists. -- A hosted reflection test verifies every export names a non-default handler of the correct category. -- Native compilation validates GraalVM's handler shape and `@Uninterruptible` requirements. - -## 2. Core Engine Lifecycle Records and Leases - -### Problem - -`ScriptRuntime.get(handle)` and `ScriptRuntime.destroy(handle)` are unrelated map operations. A raw C run can retain the Java runtime, then a concurrent destroy can remove the registry entry and return. The caller may free the resolver `ctx` while the admitted run later invokes it. - -### Design - -Change the Java registry value from a bare `ScriptRuntime` to an `EngineRecord` with this lifecycle: - -```mermaid -stateDiagram-v2 - [*] --> LIVE: register - LIVE --> LIVE: acquire or release lease - LIVE --> CLOSING: destroy closes admission - CLOSING --> CLOSING: wait for active leases - CLOSING --> DESTROYED: final lease released - DESTROYED --> [*]: remove exact record -``` - -`ScriptRuntime.acquire(long handle)` returns an `EngineLease` only when the record is `LIVE`. The state check and active-lease increment occur under the record monitor. `EngineLease` implements `AutoCloseable`, exposes the fixed `ScriptRuntime`, and releases exactly once. - -Every `run_*_engine` entrypoint acquires a lease before converting required pointers or invoking the runtime and closes it with try-with-resources. The lease spans resolver calls, streaming callbacks, transform feeder cleanup, and final result allocation. It can end before `free_cstring()` because the returned allocation no longer depends on the engine. - -`destroy_engine` atomically changes `LIVE` to `CLOSING`, rejects later acquisitions, and waits until all admitted leases close. It continues waiting through interruption and restores the interrupt status only after lifetime safety has been re-established. Concurrent destroy calls coordinate on the same record. Exact-record map removal occurs only after `DESTROYED`. - -The public raw ABI contract is: - -- `destroy_engine` is idempotent for unknown or already destroyed handles. -- `destroy_engine` can block until all previously admitted operations finish. -- Resolver and callback contexts must remain valid through `destroy_engine` return and may be freed afterward. -- Calling `destroy_engine` for an engine synchronously from one of that engine's callbacks is prohibited because the callback owns a lease that destroy must drain. - -Node's `engine_bridge_t.in_flight` remains necessary for N-API reference and resolver-bridge lifetime. Python operation serialization remains necessary for wrapper lifecycle and generation correctness. - -### Tests - -- Hosted Java tests prove admission rejection after close, multiple-lease drain, concurrent destroy, interrupted wait restoration, and lease release on thrown bodies. -- A raw ctypes subprocess blocks inside a real resolver callback, calls destroy on another OS thread, and proves destroy does not return until the callback and run finish. - -## 3. Native Callback Reentrancy Guards - -### Problem - -A synchronous resolver invokes host JavaScript or Python while the outer Graal call is active. If host code invokes DataWeave on another engine on the same OS thread, the nested call attaches and detaches that thread. The outer call then resumes with invalid Graal thread state and terminates the process. - -The same hazard applies to lifecycle operations and other synchronous host callbacks that can enter another engine. - -### Node Design - -Add addon-level OS-thread-local callback depth using `uv_key_t`, initialized through the existing `uv_once` setup. Enter the scope immediately around each direct host callback invocation and restore it on every status or exception path. - -Before any entrypoint attaches, detaches, creates, destroys, or admits DataWeave work, reject when callback depth is nonzero. The addon throws an error with a stable internal code such as `ERR_DATAWEAVE_CALLBACK_REENTRANCY`. The TypeScript layer maps that code to the public `DataWeaveError` class. - -The C guard is authoritative because raw addon users and duplicate package copies must not bypass it. A process-global boolean is forbidden because independent Worker OS threads must remain concurrent. - -### Python Design - -Add module-level `threading.local()` callback depth shared by all `NativeRuntime` instances. Apply the scope directly around user resolver, read, and write callback invocation. Check the scope before public lifecycle mutation, operation-token capture, and native serialized admission. - -The guard fails before waiting on another engine's lock. A global execution mutex or `RLock` is not used because it would either deadlock or permit the unsafe reentry. - -### Tests - -- Node and Python isolated child processes attempt cross-engine nested `run()` from a resolver, catch `DataWeaveError`, return valid module source, and prove the outer run still succeeds with no fatal stderr. -- Tests cover uncaught resolver reentry, nested initialization, and raw addon/native admission where feasible. -- Unit tests prove callback depth is OS-thread-local and restored after callback errors. - -## 4. Immutable Engine Generations - -### Problem - -Both bindings hold a mutable current handle. Python validates initialization before entering its operation lock. Node and Python lazy streams defer body execution until first iteration. Cleanup and reinitialization can therefore replace the engine between acceptance and admission, silently moving work from generation A to B. - -### Shared Contract - -Each successful engine initialization increments a monotonic binding-instance generation. Each operation captures an immutable token: - -```text -EngineOperationToken { handle, generation } -``` - -Generation never resets during cleanup or failed initialization. Handle alone is insufficient because a newly created isolate can restart Java static handle allocation. - -At native admission, the binding compares the token with the current initialized token while holding the lock that excludes lifecycle mutation. Native calls use the token's handle, never a later mutable field. - -The race has two valid outcomes: - -- Admission wins: work completes on its captured engine; cleanup waits or refuses while it is active. -- Cleanup wins: later token validation raises `DataWeaveError` for a stale engine generation; the replacement engine is never called. - -### Python Design - -Add frozen internal `_EngineOperation(handle: int, generation: int)` state to `NativeRuntime`. Public buffered, callback, transform, and stream methods capture this token synchronously. `_serialized_native_operation(expected)` validates it under the per-instance operation lock and yields the immutable token. - -Stream worker registration validates the token atomically with `_stream_workers_lock` registration. If registration wins, cleanup sees the worker and follows current active-worker policy. If cleanup wins, registration rejects as stale. Lock order is `_stream_workers_lock` before the brief operation-token validation; native execution never reacquires `_stream_workers_lock` while holding the operation lock. - -### Node Design - -Add `engineGeneration` and `EngineOperationToken`. Convert `runStreaming` and `runTransform` from public async-generator methods into ordinary methods that capture the token at call time and return private async generators. The private generator validates the token immediately before synchronous native admission and invokes FFI with `token.handle`. - -`runTransform` validates the same token before and after async input pre-buffering. The second check must occur immediately before FFI admission. - -This intentionally changes stale or uninitialized stream failure timing to method call or first pull as documented by the concrete path; normal stream chunks and terminal metadata do not change. - -### Tests - -- Deterministic Python paused-admission test captures generation A, performs cleanup/reinitialize to B, resumes, and proves no call reaches handle B. -- Node and Python tests create a lazy stream, cleanup/reinitialize, then consume it and receive `DataWeaveError` before native invocation. -- Handle-reuse tests prove generation, not only numeric handle, controls identity. -- Transform tests pause during async pre-buffering and reject after generation replacement. -- Complementary tests prove already-admitted work finishes on its original engine. - -## 5. End-to-End Node Output Backpressure - -### Problem - -The output TSFNs use `max_queue_size = 0`, and `streamFromNative` appends delivered buffers to an unbounded array. The native producer can outrun a paused consumer and retain output-sized memory even if one of those queues is later bounded independently. - -### Design - -Retain the push architecture and add one reference-counted `output_flow_t` per stream or transform operation. It tracks outstanding bytes and chunks from immediately before TSFN enqueue until the async generator dequeues the corresponding buffer. - -```mermaid -sequenceDiagram - participant P as Native producer - participant F as Flow credits - participant Q as Bounded TSFN - participant J as JS chunk queue - participant C as Async consumer - P->>F: reserve bytes and one chunk - F-->>P: wait only on producer thread if full - P->>Q: enqueue payload - Q->>J: deliver Buffer on JS thread - C->>J: dequeue Buffer - C->>F: acknowledge bytes and one chunk - F-->>P: resume below low watermark -``` - -Internal defaults are fixed initially rather than public configuration: - -- High watermark: 1 MiB or 128 chunks. -- Low watermark: 512 KiB and 64 chunks. -- A single oversized chunk is admitted when the window is empty so it cannot deadlock permanently. - -The output TSFN receives a finite queue capacity with room for normal outstanding chunks and the terminal sentinel. The native producer may wait on a per-flow condition variable; the JS thread only performs short acknowledge/cancel updates and never waits. - -The internal FFI start contract returns an operation controller with: - -```ts -interface NativeStreamingOperation { - readonly completion: Promise; - acknowledge(bytes: number): void; - cancel(): void; - close(): void; -} -``` - -`streamFromNative` acknowledges a chunk when dequeuing it for delivery. This bounds binding-owned memory but not buffers retained by application code after `yield`. - -Cancellation is mandatory because a producer blocked on credit cannot finish if the consumer abandons the stream. Generator `finally`, early `return()`, DataWeave cleanup, env teardown, and JS callback failure all cancel the flow, signal the producer, release queued credit exactly once, and allow native completion to settle. Cleanup semantics become cancel abandoned streams and drain their native completion, never wait indefinitely for consumer pulls. - -Native flow lock ordering is: - -- Never wait on the flow condition while holding `g_mutex`. -- If a path needs both locks, acquire `g_mutex` before the flow mutex and release the flow mutex before later global completion accounting. -- Refcounted flow ownership prevents callbacks, worker completion, cancellation, or TSFN finalization from freeing shared state twice. - -### Tests - -- TypeScript unit tests verify acknowledgment occurs only at dequeue, not JS enqueue; early return cancels; rejection drains accounted chunks; close/cancel are idempotent. -- Native test hooks expose current and peak outstanding bytes/chunks, pause, and cancellation state. -- Real integration tests pause a large deferred stream and transform, assert peak credits stay within the configured limit plus the one-oversized-chunk allowance, then resume and verify ordered complete output. -- Early generator return and cleanup of a paused stream complete under a bounded timeout and permit healthy reinitialization. -- The Node README writable-stream example waits for `drain` when `write()` returns false. - -## 6. Detach-Poison Failure Injection - -### Problem - -The addon now poisons an isolate when ordinary detach returns nonzero, but no test forces the status. Cleanup skipping teardown, isolate abandonment, and fresh-isolate recovery are unverified. - -### Design - -Centralize ordinary detach calls behind `detach_thread_checked(detach_site_t, thread)`. Under `DATAWEAVE_TEST_HOOKS`, a site-specific one-shot injection calls the real detach first and, when it succeeds, substitutes a nonzero observed status. Production builds remain a thin wrapper over the real function. - -Test-only counters record forced failures, isolate creation, teardown attempts, and isolate abandonment. Hooks expose state without making test behavior part of the product ABI. - -The safe hook verifies the addon's response to a detach status; it does not claim to reproduce every physical consequence of a thread that truly remained attached. - -Once poisoned, new DataWeave admission should fail closed rather than assume the isolate remains safe for new work. Already-produced triggering results may surface, active operations drain, final cleanup abandons the isolate without teardown, and later initialization creates a fresh isolate. - -### Tests - -- Child-process synchronous-run test forces one detach failure, observes poison, completes cleanup without teardown or hang, reinitializes, and runs successfully on a fresh isolate. -- Child-process transform test forces a background detach failure while final cleanup is waiting, proving deferred cleanup skips teardown and resolves. -- Lower-cost cases exercise create, resolver-create, bridge-finalize, and unknown-destroy detach sites where deterministic setup exists. - -## 7. Whitespace and Documentation - -Remove trailing spaces in `docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md` and the extra final blank line in `native-lib/python/src/dataweave/native.py` without unrelated formatting churn. - -Update: - -- `native-lib/README.md` with C sentinel, lease/drain, callback-context, and destroy-blocking contracts. -- `native-lib/node/README.md` with callback reentrancy, bounded streaming, cancellation, and writable `drain` guidance. -- `native-lib/python/README.md` with callback reentrancy and stale-generation behavior. -- `docs/superpowers/specs/2026-08-07-native-lib-multi-engine-design.md` so the final-state architecture uses core leases, immutable generation tokens, bounded flow control, and the new test-hook posture. - -## Implementation Sequence - -```mermaid -flowchart TD - A[Java lifecycle tests] --> B[Core leases] - C[Native exception tests] --> D[C entrypoint handlers] - B --> E[Raw ABI concurrency verification] - D --> E - F[Python red tests] --> G[Callback and generation guards] - H[Node callback and generation red tests] --> I[Node guards and tokens] - J[Stream credit unit tests] --> K[Native credit protocol] - K --> L[Slow-consumer integration tests] - M[Detach injection red tests] --> N[Central detach wrapper and recovery] - E --> O[Docs and full native verification] - G --> O - I --> O - L --> O - N --> O -``` - -Use focused commits for independently reviewable behavior. The final PR contains the complete cohesive hardening set and targets `w-23692110-multi-engine-design`; do not squash unless requested during review. - -## Verification Matrix - -| Layer | Required verification | -|---|---| -| Java hosted | Focused lifecycle and existing `native-lib:test` suites | -| Native image | `native-lib:nativeCompile` with GraalVM Community Java 24 | -| Raw ABI | Isolated ctypes exception and resolver-context lease subprocess tests | -| Node unit | Generation, stream-credit, cancellation, and error mapping tests | -| Node integration | Resolver reentry, stale streams, slow consumer, cleanup, poison recovery | -| Node typecheck | `npm run build:ts` | -| Python unit | Token admission, callback TLS, worker registration, lifecycle tests | -| Python integration | Resolver reentry and stale real-native streams | -| Repository | `git diff --check w-23692110-multi-engine-design...HEAD` | - -Run the smallest focused test after each red-green cycle, then the nearest module suite. Before PR creation, run Java, Node, Python, native-image, and diff verification from a clean branch and review every commit in the PR range. - -## Risks and Mitigations - -- **Destroy self-deadlock:** same-engine callback destroy waits for its own lease. Reject binding callback reentrancy and document the raw ABI prohibition. -- **Lease leak:** one missed close blocks destroy forever. Use try-with-resources in every entrypoint and tests that throw inside the leased body. -- **Flow-control deadlock:** waiting from the JS thread or while holding `g_mutex` prevents progress. Only the native producer waits, with explicit lock ordering and cancellation broadcasts. -- **Double free/use-after-free:** worker, TSFN callback, generator cancellation, and finalizer share flow state. Use reference counting and idempotent cancel/close transitions. -- **Generation false acceptance:** numeric handles can repeat after isolate replacement. Compare both monotonic generation and handle. -- **Fault-hook overclaim:** real detach then substitute failure validates status handling but not a physically attached dead thread. State this limitation in tests and docs. -- **Throughput regression:** high/low watermarks intentionally slow fast producers behind slow consumers. Integration tests verify correctness; benchmark only if normal-consumer throughput changes materially. -- **Native handler fragility:** GraalVM custom exception handlers have strict shape requirements. Keep handlers allocation-free and prove them through native compilation and subprocess execution. - -## Success Criteria - -- The three previously reproduced exit-99 cases remain alive and return documented errors or sentinels. -- Direct raw-ABI destroy cannot return while an admitted resolver callback may still use its context. -- No buffered, callback, streaming, or transform operation can migrate to a replacement engine generation. -- Node binding-owned output buffering remains within configured byte/chunk watermarks for paused consumers. -- Abandoned or cleanup-cancelled Node streams settle without deadlock. -- Forced ordinary detach failures poison and abandon the old isolate, skip unsafe teardown, and allow fresh initialization. -- Existing valid-input Node, Python, Java, native, and TCK behavior remains green. -- `git diff --check` passes for the full PR range.