feat(workflow-operator): export the source operators as Python by kz930 · Pull Request #8341 · apache/texera

kz930 · 2026-09-02T18:18:44Z

What changes were proposed in this PR?

Nine source operators implement StandaloneCodeGenerator, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in: the CSV family, JSON Lines, Arrow, plain text, and the two that read a file named at run time.

A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory. Each source writes a placeholder where its file should be named and declares the path it reads, and the translator assigns the names once it can see the whole plan: the last segment of the path where that is free, and a numbered form of it where two sources read different files whose paths end the same way. The last segment rather than a parse of the whole string, because the resolver percent-encodes the file-relative parts but leaves the repository and version names as the user typed them, and a dataset version with a space in its name makes new URI throw before any code is generated. That is what makes an exported script portable, and it is equally its one precondition: the data has to sit beside the script, under the names the script states.

Two sources are reported as unverifiable rather than exported blind. File Scan takes its filenames from an input port at run time, which a source harness has nothing to feed. URL Fetcher reads a live URL, so two runs are not required to agree and a comparison would only measure the network.

Arrow is read into pandas' nullable dtypes rather than its numpy ones. Arrow says of every value whether it is there, and a numpy column has nowhere to put that: a missing double and a stored NaN both land on NaN, and a missing integer costs the column its integer type. The engine reads the file's own answer, so a set operation keeps a NaN row a null row does not match, a holed integer column is still made of integers, and a long past 2^53 is not rounded on its way through a float. The other sources state the types they need one at a time, their formats having no way to say it once.

A file also states the width and the sign of each of its numbers, and Texera has no column narrower than a double or a 32-bit integer. A single-precision or 16-bit column was arriving null, the parse having no case for the Float and Short those vectors hand back, and an unsigned one arrived as the -1 its storage counts down to. Each now reads as the Texera type that holds it, an unsigned 32-bit column as a long and an unsigned 64-bit one refused for want of a wider one, and the script normalizes the same widths so a later sum comes to the same number on both sides. This is the rule the Parquet source already follows. The end of an exported scan window is counted in Long for the same kind of reason: two Ints the operator accepts can add up past what an Int holds, and the slice came out negative.

Two of the changes made in review are deliberately not the literal instruction, because the three CSV readers do not agree with each other about a blank field. Old CSV sets keep_default_na=False but names no missing value at all: scala-csv hands a blank back as the empty string rather than as a null, so the na_values the main CSV source needs would null a value this reader keeps. Parallel CSV asks for such a column as text rather than as a nullable integer: its schema is read with scala-csv, where one blank types the whole column as text, while its executor nulls the blank and parses the rest as that text. Each fix follows the reader it belongs to rather than the one next to it.

Any related issues, documentation, discussions?

Part of #8325, 12 of 27; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here.

Closes #8415, the task this change is the whole of.

Three engine defects were found while matching these readers, and all three are fixed here rather than worked around: none could be closed by the export alone. #8596, where a file scan decoded UTF-8 whatever its Encoding field said, because the field the panel writes never reached the executor. #8598, where a file scan extracting an archive could not also include the filename, because the line-by-line branch emitted a one-field row against the two-column schema its own flag declares. #8602, where a scan window of no rows left the type inference nothing to read, so three readers declared a schema of no columns and a fourth threw.

Closes #8596. Closes #8598. Closes #8602.

How was this PR tested?

Each operator asserts the block it emits in its own spec. Where a difference is one of behaviour rather than of text, the spec runs the engine's own executor and the generated script over the same file and compares what each produced: the boolean parsing rules, the entries read out of an archive, the row window the parallel reader ignores, and the nullable long each source has to keep exact. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.

Arrow is the one worth calling out. Its column of timestamps only matches once #7672 is in: before that the engine read the file's numbers through the JVM's zone while pd.read_feather read the wall clock the file states, and the two differed by whatever the machine was set to. That change has since landed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…ython script A workflow can be read in the editor but not taken away: there is no form of it that runs anywhere else, so a user who wants to keep a pipeline, hand it to someone without Texera, or step through it in a notebook has nothing to take. This adds the seam for one and the first few operators through it. An operator says how it reads outside the engine by implementing `StandaloneCodeGenerator`, returning a block of pandas that names its inputs and outputs as `in1df` / `out1df`. The translator walks the plan in topological order, gives every port a variable, substitutes those placeholders, and prints the leaves; `inAlldf` stands for the whole list of upstreams, which is what a variadic port like Union's needs, since any fixed count the code stated would be wrong for some workflow. An operator with no generator yet leaves a commented TODO rather than a line that looks like it works. `GET /workflow-to-python` on the compiling service returns the script for a plan it is given. Five operators implement it here — Distinct, Limit, Projection, Filter and Union — chosen to cover the shapes the translator has to handle: a single input, a config-driven one, one that renames columns, one that builds a predicate, and the variadic port. The rest of the operator set follows in later changes. `pyStringLiteral` renders a value as a Python literal with the escaping that keeps a quote or a newline in a column name from ending the literal early. The generators cannot use the runtime's decode expression, which needs an operator instance to decode through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…re the files The standalone export claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a LogicalOp the way the engine does — compiling it to a physical plan and driving the executor — but outside a workflow, against JSONL files rather than a live upstream. `PyOpExecHarness` does the same for a Python operator, through the worker the engine uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, and runs it. `TupleIO` is what the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a number and the two paths disagree over a difference neither operator made. Both runners produce files, not assertions, so what to make of a difference is left to a later change. What is here is enough to run one operator both ways and see that the answers match, which is what the spec does with Distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…he operator it came from The standalone export claims a generated script does what the operator does. This is what checks it, for every operator, on every configuration the operator offers. An operator is run twice. `OpExecHarness` drives it the way the engine does, compiled to a physical plan but outside a workflow, reading JSONL files rather than a live upstream; `PyOpExecHarness` does the same for a Python operator through the worker the engine uses. `StandaloneRunner` takes the other path, wrapping the operator's standalone code in a script that binds the same files. Both write files, and `Comparator` reads them back: order-insensitive by default, since the engine interleaves across workers and only the sort family promises an order. A visualization is compared as a figure rather than as a frame. What to run an operator ON is decided rather than written by hand for each. `ConfigGenerator` reads the operator's own schema — its enums, defaults, declared ranges and column pickers — and produces a base configuration plus one variant per branch the operator offers, so a switch nobody thought to try is still tried. `CanonicalFixture` is the table they run against, one column per shape an operator might ask for. `CuratedHandlers` is the escape hatch for an operator whose input cannot be derived, and `TransformVerificationRunner` decides which of the three tiers each operator takes and reports what it could not run and why. `LogicalOp.orderSensitive` and `@SampleColumn` are the two things the operators had to say for this to read them: whether row order is part of the contract, and which column a field should be pointed at when the first unused one would be a poor choice. Most of the operator set does not implement the generator yet — it arrives a family at a time — and the runner reports each of those rather than passing over it. The tier assertions for a family land with the change that gives that family its generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rces

Ten sources say how they read outside the engine, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in: the CSV family, JSON Lines, Arrow, plain text, and the two that read a file named at runtime. A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory by the name the URI ended with. That is what makes an exported script portable, and it is also its one precondition: the data has to sit beside the script. Two are reported as unverifiable rather than exported blind. File Scan takes its filenames from an input port at run time, which a source harness has nothing to feed; URL Fetcher reads a live URL, so no two runs are required to agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-02T18:20:13Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @roshiiiz, @aglinxinyuan, @eugenegujing
You can notify them by mentioning @roshiiiz, @aglinxinyuan, @eugenegujing in a comment.

codecov-commenter · 2026-09-02T18:23:33Z

Codecov Report

❌ Patch coverage is 62.31454% with 127 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.52%. Comparing base (bf356f5) to head (ce52ee2).
⚠️ Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
...rator/source/scan/json/JSONLScanSourceOpDesc.scala	10.63%	42 Missing ⚠️
...erator/source/scan/file/FileScanSourceOpDesc.scala	37.50%	27 Missing and 8 partials ⚠️
...cala/org/apache/texera/amber/util/ArrowUtils.scala	29.41%	5 Missing and 7 partials ⚠️
...rator/source/scan/text/TextInputSourceOpDesc.scala	62.50%	3 Missing and 6 partials ⚠️
.../source/scan/csv/ParallelCSVScanSourceOpDesc.scala	78.57%	2 Missing and 4 partials ⚠️
...or/source/scan/csvOld/CSVOldScanSourceOpDesc.scala	78.57%	2 Missing and 4 partials ⚠️
...operator/source/scan/arrow/ArrowSourceOpDesc.scala	66.66%	4 Missing and 1 partial ⚠️
...ber/operator/source/scan/file/FileScanOpDesc.scala	91.37%	0 Missing and 5 partials ⚠️
...operator/source/scan/csv/CSVScanSourceOpDesc.scala	89.65%	1 Missing and 2 partials ⚠️
.../amber/translator/WorkflowToPythonTranslator.scala	80.00%	0 Missing and 2 partials ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8341      +/-   ##
============================================
- Coverage     93.50%   92.52%   -0.99%     
- Complexity     4882     4948      +66     
============================================
  Files          1220     1221       +1     
  Lines         50788    51031     +243     
  Branches       6262     6290      +28     
============================================
- Hits          47489    47215     -274     
- Misses         1746     2228     +482     
- Partials       1553     1588      +35

Flag	Coverage Δ		*Carryforward flag
access-control-service	`71.78% <ø> (ø)`
agent-service	`99.32% <ø> (ø)`		Carriedforward from 3d1a4be
amber	`88.12% <61.77%> (-0.99%)`	⬇️
computing-unit-managing-service	`55.20% <ø> (-21.95%)`	⬇️
config-service	`87.37% <ø> (+0.12%)`	⬆️
file-service	`81.53% <ø> (ø)`
frontend	`96.70% <ø> (+<0.01%)`	⬆️	Carriedforward from 3d1a4be
notebook-migration-service	`83.73% <ø> (ø)`
pyamber	`98.47% <ø> (ø)`		Carriedforward from 3d1a4be
workflow-compiling-service	`74.43% <80.00%> (+0.33%)`	⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-09-02T18:27:07Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 10 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main bf356f5 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	376	0.229	25,939/36,367/36,367 us	🔴 +22.5% / 🔴 +130.2%
🔴	bs=100 sw=10 sl=64	815	0.497	119,106/152,385/152,385 us	🔴 +8.0% / 🔴 +40.2%
⚪	bs=1000 sw=10 sl=64	930	0.568	1,070,293/1,165,474/1,165,474 us	⚪ within ±5% / 🔴 +12.4%

Baseline details

Latest main bf356f5 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	376 tuples/sec	432 tuples/sec	767.43 tuples/sec	-13.0%	-51.0%
bs=10 sw=10 sl=64	MB/s	0.229 MB/s	0.263 MB/s	0.468 MB/s	-12.9%	-51.1%
bs=10 sw=10 sl=64	p50	25,939 us	22,820 us	12,880 us	+13.7%	+101.4%
bs=10 sw=10 sl=64	p95	36,367 us	29,693 us	15,801 us	+22.5%	+130.2%
bs=10 sw=10 sl=64	p99	36,367 us	29,693 us	19,767 us	+22.5%	+84.0%
bs=100 sw=10 sl=64	throughput	815 tuples/sec	862 tuples/sec	988.73 tuples/sec	-5.5%	-17.6%
bs=100 sw=10 sl=64	MB/s	0.497 MB/s	0.526 MB/s	0.603 MB/s	-5.5%	-17.6%
bs=100 sw=10 sl=64	p50	119,106 us	111,931 us	102,684 us	+6.4%	+16.0%
bs=100 sw=10 sl=64	p95	152,385 us	141,104 us	108,712 us	+8.0%	+40.2%
bs=100 sw=10 sl=64	p99	152,385 us	141,104 us	118,731 us	+8.0%	+28.3%
bs=1000 sw=10 sl=64	throughput	930 tuples/sec	945 tuples/sec	1,022 tuples/sec	-1.6%	-9.0%
bs=1000 sw=10 sl=64	MB/s	0.568 MB/s	0.577 MB/s	0.624 MB/s	-1.6%	-9.0%
bs=1000 sw=10 sl=64	p50	1,070,293 us	1,052,467 us	999,086 us	+1.7%	+7.1%
bs=1000 sw=10 sl=64	p95	1,165,474 us	1,192,246 us	1,037,033 us	-2.2%	+12.4%
bs=1000 sw=10 sl=64	p99	1,165,474 us	1,192,246 us	1,066,123 us	-2.2%	+9.3%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,532.37,200,128000,376,0.229,25938.59,36367.36,36367.36
1,100,10,64,20,2453.95,2000,1280000,815,0.497,119105.53,152385.18,152385.18
2,1000,10,64,20,21498.50,20000,12800000,930,0.568,1070293.05,1165473.70,1165473.70

…f the one without The split this change relies on was declared but never wired. The specs carry `@IntegrationTest` and `build.sbt` reads `WCS_TEST_FILTER` to act on it, but nothing set that variable, so the filter was a no-op and the specs that fork Python ran in the job that provisions none — failing on `No module named 'pandas'` rather than on anything they were testing. The platform job now sets `skip-integration`, which excludes them. The platform-integration job sets `integration-only` and provisions what they need: Python 3.12, amber's requirements, protoc, and the generated proto bindings, which are gitignored and so have to be regenerated before a forked driver can import pyamber. Every step is guarded on the service, so the other entries in that matrix are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…into feat/standalone-sources

… have one The example was an operator another batch gives a generator to, so the assertion held only until that batch landed. A Python UDF holds whatever order these land in: its body is written by whoever drops the operator, so there is nothing for a generator to emit. The word cloud assertion goes for the same reason. It says the operator is withheld, which a later batch stops being true once its placement is seeded, and the prediction op alone already covers what the test is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything here that is not a source operator belongs to apache#8327 and was carried only so this branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does: what is left is the thirteen files this change is actually about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch held an older copy of both and changes neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

The source exports now follow the native slicing, parsing, column naming, and decoding behavior. Looks good.

carloea2

The standalone boolean readers do not match the native reader. The native path accepts 1 as true and rejects invalid text. The standalone path treats both as false. Please use the same parsing rules and add tests for 1 and invalid text.

carloea2

There are two more source mismatches. Parallel CSV applies offset and limit only in standalone mode, while the native executor ignores them. Also every source path is reduced to its filename, so two different files named data.csv both read the same local file. Please keep the paths distinct and make Parallel CSV behavior match.

carloea2

One more source case is different. With extract enabled, the native path reads files inside the archive. Standalone only prints a warning and then opens the archive itself. Please implement extraction or stop export with a clear unsupported error, and add an archive test.

carloea2

Parallel CSV and Old CSV still use pandas defaults. Literal NA becomes null, and a blank header becomes an Unnamed column. The native readers keep NA and use generated column names. Please apply the same fixes as the main CSV source and add tests for both readers.

carloea2

Nullable large integers can lose precision in the exported CSV and JSONL sources. For example, 9007199254740993 with one null is widened through a float and becomes 9007199254740992. The native source keeps the exact value. Please preserve nullable long values and add this case to the source tests.

Arrow says of every value whether it is there. pandas' numpy dtypes have nowhere to put that: a missing double and a stored NaN both land on NaN, and a missing integer costs the column its integer type. The engine reads the file's own answer, so a set operation keeps a NaN row that a null row does not match, and a holed integer column is still made of integers. Only Arrow among these sources carries the distinction. CSV, JSON Lines and the text sources cannot express it, so they are left as they are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…om a NaN Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rces Five places where a source's exported Python answered differently from the operator the workflow ran. A boolean line is read by parseField, which takes "true" and "false" in any case and then an integer that is true only at 1, and refuses anything else. Comparing the lowercased line to "true" called 1 false and passed text the engine refuses off as a row of false. The three line readers now share one parser, emitted once per script. Parallel CSV inherits a limit and an offset its executor never reads, so slicing in the export handed back fewer rows than the workflow produced. The window is dropped, and said to be dropped where it was set. Every source named its file by the last segment of its path, so two sources reading different files both opened one of them and nothing said so. A source now writes a placeholder and declares the path it reads; the translator hands each distinct path a name, reusing the last segment when it is free and numbering it when it is not, the way a chart's output file is numbered. With extract on, the engine reads the files inside the archive. The export printed a warning and opened the archive itself. It now walks the zip entries the same way, skipping the ones macOS adds and taking the entry's own name for the filename column. Parallel CSV and Old CSV read a literal NA as a null and a blank header as an Unnamed column, where their readers keep both. They keep them now, by the same route the main CSV source took, except that Old CSV names no missing value at all: scala-csv hands a blank back as "", not as a null. A nullable long lost precision in four sources, for three different reasons. The main CSV and Arrow sources widen it through a float, so 9007199254740993 came back as ...992; they ask for the nullable integer instead. read_json rounds before any dtype can apply, so the JSONL source rebuilds those columns from an exact parse of the same lines. Parallel CSV types such a column STRING and needed the text, not an integer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…at/standalone-sources # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/arrow/ArrowSourceOpDesc.scala # common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/arrow/ArrowSourceOpDescSpec.scala

kz930 · 2026-09-18T20:28:14Z

All five are fixed in 32173ef6f.

Two are deliberately not the literal "same as the main CSV source", because the three readers disagree about a blank field. Old CSV names no missing value: scala-csv hands a blank back as the empty string, so na_values would null a value it keeps. Parallel CSV asks for the large-integer column as text rather than as a nullable integer, because its schema types that column text while its executor nulls the blank.

Two engine defects are filed rather than worked around: #8596 and #8598. The export follows the panel on the first, so it diverges from the engine until that one is fixed.

carloea2

JSONL array flattening still differs. With flatten enabled and input {"items":[{"id":1},{"id":2}]}, native creates items1.id and items2.id. The export keeps one items column containing the list, so a following projection of items1.id fails. I reproduced the exported result with pandas 2.2.3. Please flatten arrays using the native column names and add this case.

pandas reads a negative bound from the end where the native readers read it as no skip or no rows: iloc[-1:] is the last row, and read_csv rejects a negative nrows outright. The Arrow export and the two CSV exports now take the window from the clamped value, so the script answers what the executor answers for a value only the property editor refuses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…names The executor gives every element of a nested array a column of its own, named for its position counted from one, so {"items":[{"id":1},{"id":2}]} is items1.id and items2.id and the schema declares those. json_normalize opens a nested object and leaves an array whole, so the export handed the plan one items column holding a list and a step reading items1.id found no such column. The script now flattens each record the way JSONToMap flattens it and leaves json_normalize records that are flat by then. The exact re-read a long needs looks its column up in that flattened record, which also settles a long nested inside an array: the name items1.id is a key there, where splitting it on the dot looked for an items1 no record holds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-18T23:35:33Z

Fixed in a2290cdfb: the script now flattens each record the way JSONToMap does, so an array element takes its parent's name and its position counted from one, and both sides call the column items1.id. The new tests run the generated script against the executor's own answer, for the array case and for a long nested inside one.

A JSONL file states no column order, so the operator sorts the names it found and the rows the workflow sees follow that. read_json keeps the order the first record happened to use, so the export handed the plan the same columns in another order, which is what a positional read downstream and a file export both go by. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A single-precision or 16-bit column arrived null. `parseField` had no case for the Float and Short those vectors hand back, so the parse threw and the Arrow source's own catch wrote null for the whole column. Texera writes every double it owns as eight bytes and every integer as 32 or 64 bits, which is why nothing had reached the gap before; a file written elsewhere from a numpy float32 or int16 array does. The exported script kept the file's width where the executor reads the Texera column's, so 16777216 and 1 summed to 16777216 against the 16777217 doubles give. It now normalizes the two widths, as the Parquet source already does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…they count to Arrow's unsigned vectors hand back the raw storage, signed: a column counting to 4294967295 arrived as -1, and an unsigned 16-bit one arrived null, its Character being no kind of integer to the parse. The mapping read only the bit width and never the sign, so there was no width above to read the value into. Each unsigned column now takes the Texera type its own values need, as the Parquet source's mapping already does, and an unsigned 64-bit one is refused rather than read as the -1 it is stored as. The exported script lands on the same types. The two upstream specs pinning the old mapping move with it, and the one refusing an 8-bit width now asserts the INTEGER it reads as. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both bounds are Ints the operator accepts, and their sum is not one. The Arrow window ended at a negative, which `iloc` reads from the end, so it asked for all but the last two rows where the executor takes every row from the offset on. The two CSV readers add 1 to the offset to step past the header, which overflows at the largest offset and leaves an empty range, skipping nothing where the executor's drop keeps no rows. Each addition is now carried in Long, as the Parquet source already does. JSONL and the file scans take two independent slices and add nothing, so they were already right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@JsonIgnoreProperties

… names The panel writes `encoding`, but the executor decoded with the inherited `fileEncoding`, which this descriptor names in its own @JsonIgnoreProperties. It never survived the trip, so it was always its default and choosing any other charset changed nothing: a UTF-16 file came back as its bytes read as UTF-8. The executor reads the field the panel writes. The name stays as the panel spells it, a saved workflow carrying `encoding` and not the other. The export already followed the panel, so its note about parting from the engine goes with the defect. The spec that claimed to read US_ASCII was setting the field the executor never saw, and now sets the one it does. Closes apache#8596 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… asked to A file scan with Extract and Include Filename on could not run at all in any attribute type that reads line by line, which is the default. The two halves disagreed about how wide a row is: the schema prepends a `filename` column whenever the flag is set, while the line-by-line branch emitted the value alone, so the first tuple was one field against two columns and could not be built. Include Filename is only offered once Extract is on, so every configuration reaching it is one the panel invites. The branch now pairs each entry with its name, as the single-value branch already did. Both exports follow: their `&& isSingle` was there to mirror the defect, and the spec pinning that a line-mode export carries no filename column now pins the column it does carry. Closes apache#8598 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

Flattened timestamp fields still come out as text. With a timestamp inside meta.when, read_json is asked to convert meta.when before flattening creates that column. The value stays a string. I reproduced a sort that puts January 2025 before March 2024 because of this. Please convert declared timestamp columns after flattening and add a nested date test.

…s it `read_json` was handed the names the schema gives nested values, but under flattening those columns do not exist yet: the file still holds the object around them, so it found nothing to convert and `json_normalize` built the column out of text. A plan sorting on it read January 2025 as coming before March 2024, where the executor, which parses the value on its way into the tuple, had them the other way around. `convert_dates` now goes to the reader only when the names it is given are the file's own. Under flattening the declared timestamp columns are converted once the frame that holds them exists. A format is inferred per value, as this operator's own parser reads each value on its own, so a column that wrote the same instant two ways still converts whole; `read_json` was already lenient that way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-19T08:11:50Z

Confirmed and fixed in f3d0db2: convert_dates now only names columns the file itself has, and under flattening the declared timestamp columns are converted once json_normalize has built them. The new test pins a nested timestamp whose text order and instant order disagree, checked against both the executor and the generated script.

… rows The Limit bounds the rows a file scan outputs, and it also bounded the sample `sourceSchema` reads to infer their types, so a Limit of 0 left the inference nothing to look at. The three readers then failed differently on the same file: the CSV and JSONL scans declared a schema with no attributes at all, and the old CSV scan threw, taking its names from the header row and its types from the sample and then asking an empty array for the first one. The parallel CSV scan threw for the same reason. A file's columns do not depend on how many of its rows were asked for, so a window of no rows reads the sample it would have read without one. A Limit of 1 or more is unchanged, still typing the columns from the rows the operator will actually emit. Closes apache#8602 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A file pandas wrote from a frame keyed by one of its columns says so in its schema, and pandas reads those columns back as the frame's index rather than as columns. The executor reads the columns the file states and has no notion of an index, so a file written from a frame keyed by `customer_id` kept that column on the one side and dropped it on the other, where an operator naming it raised. The columns the file states are asked of it and the index put back under those names, which is `__index_level_0__` for an index that had none, and in the file's own order, which is where pandas wrote them: last. Before the widths are looked at, a column restored from an index being as narrow as any other. The Parquet source parted the same way over the same note. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 and others added 5 commits September 1, 2026 16:23

Merge remote-tracking branch 'upstream/main' into feat/standalone-sou…

97e05ec

…rces

github-actions Bot assigned kz930 Sep 2, 2026

github-actions Bot added feature dependencies Pull requests that update a dependency file common platform Non-amber Scala service paths labels Sep 2, 2026

kz930 and others added 2 commits September 2, 2026 11:32

Merge remote-tracking branch 'myfork/feat/standalone-verify-harness' …

9be0dbd

…into feat/standalone-sources

github-actions Bot added the ci changes related to CI label Sep 2, 2026

kz930 and others added 3 commits September 2, 2026 16:10

Merge upstream/main

3f07020

chore: leave the two verify files to apache#8327 as well

df10bd8

This branch held an older copy of both and changes neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions Bot removed dependencies Pull requests that update a dependency file ci changes related to CI platform Non-amber Scala service paths labels Sep 2, 2026

kz930 mentioned this pull request Sep 2, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

27 tasks

carloea2 approved these changes Sep 18, 2026

View reviewed changes

carloea2 suggested changes Sep 18, 2026

View reviewed changes

kz930 and others added 2 commits September 18, 2026 12:52

test(operator): pin that an Arrow read keeps a missing value apart fr…

fe5e8c0

…om a NaN Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 mentioned this pull request Sep 18, 2026

feat(workflow-operator): export the relational and sampling operators #8506

Draft

kz930 and others added 2 commits September 18, 2026 13:19

github-actions Bot added the platform Non-amber Scala service paths label Sep 18, 2026

carloea2 suggested changes Sep 18, 2026

View reviewed changes

kz930 and others added 2 commits September 18, 2026 15:35

kz930 and others added 6 commits September 18, 2026 17:05

carloea2 suggested changes Sep 19, 2026

View reviewed changes

kz930 mentioned this pull request Sep 19, 2026

A file scan with Limit 0 loses the file's columns, and the CSVOld scan throws #8602

Open

VuMartin mentioned this pull request Sep 19, 2026

fix(workflow-operator): sample at least one row when Limit is 0 #8608

Closed

Conversation

kz930 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Sep 18, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kz930 commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

codecov-commenter commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

kz930 commented Sep 18, 2026 •

edited

Loading