@W-23692110: Harden multi-engine lifecycle and streaming

docs/superpowers/specs/2026-08-04-nodejs-external-modules-design.md

@@ -1,7 +1,7 @@
 # Design: External DataWeave Module Support in Node.js Binding
 
-**Date:** 2026-08-04  
-**Status:** Superseded  
+**Date:** 2026-08-04
+**Status:** Superseded
 **Related Proposal:** [docs/proposals/nodejs-external-modules.md](../../proposals/nodejs-external-modules.md)
 
 > **⚠️ Superseded (2026-08-31).** This document describes the original
@@ -366,20 +366,20 @@ public interface ResolveModuleCallback extends CFunctionPointer {
 ```java
 public class CallbackWeaveResourceResolver implements WeaveResourceResolver {
     private final ResolveModuleCallback callback;
-    
+
     public CallbackWeaveResourceResolver(ResolveModuleCallback callback) {
         this.callback = callback;
     }
-    
+
     @Override
     public Option<WeaveResource> resolve(ResourceDescriptor descriptor) {
         CCharPointer pathPtr = CTypeConversion.toCString(descriptor.path()).get();
         CCharPointer resultPtr = callback.invoke(CurrentIsolate.getCurrentThread(), pathPtr);
-        
+
         if (resultPtr.isNull()) {
             return Option.empty();  // Resolver returned null
         }
-        
+
         String source = CTypeConversion.toJavaString(resultPtr);
         // Note: host must free resultPtr after this returns
         return Option.apply(new StringWeaveResource(descriptor.path(), source));
@@ -462,7 +462,7 @@ static char* resolve_module_callback(void* thread, const char* module_path) {
     // Return result_source (or NULL)
 }
 
-static void resolver_js_callback(napi_env env, napi_value js_callback, 
+static void resolver_js_callback(napi_env env, napi_value js_callback,
                                   void* context, void* data) {
     // Call JS: result = resolveModule(modulePath)
     // Extract result string or null

native-lib/README.md

@@ -68,6 +68,24 @@ isolate for the process lifetime, and lets a future initialization build a fresh
 isolate. This ref-counting and teardown policy lives in the binding code, not in
 the dwlib engine ABI.
 
+## Raw engine ABI contract
+
+The exported engine entrypoints are `create_engine`,
+`create_engine_with_resolver`, `destroy_engine`, `run_script_engine`,
+`run_script_callback_engine`, and `run_script_input_output_callback_engine`.
+`create_engine` and `create_engine_with_resolver` return `0` when their Java C
+entrypoint fails. The three `run_*_engine` entrypoints return `NULL` when their
+Java C entrypoint fails; callers must not pass `NULL` to `free_cstring`. Those
+sentinels are distinct from normal DataWeave failures: a script error is a
+non-`NULL` JSON envelope with `success:false`, and that allocated result must be
+freed normally.
+
+`destroy_engine` closes admission for the handle and blocks until operations
+already admitted to that engine drain. Resolver, read, and write callback
+contexts must remain valid until `destroy_engine` returns. A callback must not
+call `destroy_engine` synchronously for the engine invoking it: that operation
+holds an admitted lease and would wait for itself to finish.
+
 ## Building with Gradle
 
 ### Prerequisites

native-lib/node/README.md

@@ -232,6 +232,26 @@ import { cleanup } from 'dataweave-native';
 await cleanup();
 ```
 
+### Callback and stream lifecycle
+
+Resolver, read, and write callbacks must not call DataWeave lifecycle or
+execution APIs on the same thread. The binding rejects that reentry with a
+public `DataWeaveError` instead of recursively entering the native runtime.
+Native addon callers receive the message `DataWeave native methods cannot be
+called from a native callback` and code `ERR_DATAWEAVE_CALLBACK_REENTRANCY`.
+
+Streaming and transform work captures the initialized engine generation. If
+cleanup or reinitialization happens before it is consumed or admitted, it fails
+with `DataWeaveError: DataWeave operation belongs to a stale engine generation.`
+and never runs against the replacement engine. Cleanup cancels and waits for
+abandoned active streams and transforms before destroying their engine.
+
+The addon uses bounded native output buffering. Its byte and chunk watermarks,
+finite thread-safe-function queue, and controller/sequence credit bookkeeping
+are implementation details, not public configuration or a BigInt sequence API.
+The bound excludes `Buffer` objects retained by application code after a chunk
+is yielded.
+
 ### Class-Based API
 
 For more control, use the `DataWeave` class directly:
@@ -411,8 +431,9 @@ console.log(result.getString());  // "300"
 ### Streaming Large Files
 
 ```javascript
-import { runTransform } from 'dataweave-native';
-import { readFileSync, createWriteStream } from 'fs';
+import { once } from "node:events";
+import { readFileSync, createWriteStream } from "node:fs";
+import { runTransform } from "dataweave-native";
 
 const script = `
 %dw 2.0
@@ -439,7 +460,9 @@ const generator = runTransform(
 const output = createWriteStream('filtered.json');
 
 for await (const chunk of generator) {
-  output.write(chunk);
+  if (!output.write(chunk)) {
+    await once(output, "drain");
+  }
 }
 
 output.end();