Skip to content

fix(workflow-operator): sample at least one row when Limit is 0 - #8608

Closed
VuMartin wants to merge 2 commits into
apache:mainfrom
VuMartin:fix/file-scan-limit-zero
Closed

VuMartin wants to merge 2 commits into
apache:mainfrom
VuMartin:fix/file-scan-limit-zero

Conversation

@VuMartin

@VuMartin VuMartin commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Fix schema inference for CSV and JSONL file-scan operators when Limit = 0.

Previously, the scan limit was also used to determine how many rows were sampled for schema inference. When the limit was 0, no rows were sampled, causing schema inference to produce an empty schema or, for the old CSV scanner, an ArrayIndexOutOfBoundsException.

This change ensures schema inference samples at least one row when the scan limit is 0, while the actual scan output remains limited to zero rows.

Before:
image

After:
image

Any related issues, documentation, discussions?

How was this PR tested?

Added regression tests verifying that CSV, Parallel CSV, CSVOld, and JSONL file-scan operators still infer the expected columns when Limit = 0.

The CSV and JSONL test suites were run with:

sbt 'WorkflowOperator / Test / testOnly org.apache.texera.amber.operator.source.scan.csv.CSVScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.json.JSONLScanSourceOpDescSpec'

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Sonnet 5)

@Yicong-Huang Yicong-Huang added release/v1.3 back porting to release/v1.3 release/v1.2 back porting to release/v1.2 labels Sep 19, 2026
@github-actions
github-actions Bot requested review from mengw15 and xuang7 September 19, 2026 10:53
@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. A release/* label nominates a backport target; the branch's release manager approving this PR is what sends the fix there. The required Backport Approvals check stays red until every label below is approved, so each manager either approves or removes their own label — which is why the labels left on a merged PR are exactly the branches it reached.

Release branch Analysis
release/v1.3 Already labeled — this fix is queued to backport here. @mengw15 decides: approving sends the fix here, removing this label declines it. The merge waits on one or the other.
release/v1.2 Already labeled — this fix is queued to backport here. @xuang7 decides: approving sends the fix here, removing this label declines it. The merge waits on one or the other.

Auto-label run.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kz930, @roshiiiz
    You can notify them by mentioning @kz930, @roshiiiz in a comment.

@codecov-commenter

codecov-commenter commented Sep 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.76%. Comparing base (e7d1676) to head (db104f7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8608   +/-   ##
=========================================
  Coverage     92.76%   92.76%           
- Complexity     4897     4898    +1     
=========================================
  Files          1236     1236           
  Lines         52155    52157    +2     
  Branches       6409     6410    +1     
=========================================
+ Hits          48380    48382    +2     
  Misses         2194     2194           
  Partials       1581     1581           
Flag Coverage Δ *Carryforward flag
access-control-service 71.78% <ø> (ø)
agent-service 99.16% <ø> (ø) Carriedforward from e7d1676
amber 88.60% <100.00%> (+<0.01%) ⬆️
computing-unit-managing-service 55.20% <ø> (ø)
config-service 87.37% <ø> (ø)
file-service 81.53% <ø> (ø)
frontend 96.57% <ø> (ø) Carriedforward from e7d1676
notebook-migration-service 83.73% <ø> (ø)
pyamber 98.41% <ø> (ø) Carriedforward from e7d1676
workflow-compiling-service 74.09% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 10 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main 957b6c9 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 365 0.223 26,090/34,235/34,235 us 🔴 +20.5% / 🔴 +122.8%
🔴 bs=100 sw=10 sl=64 749 0.457 129,807/171,118/171,118 us 🔴 +20.5% / 🔴 +63.7%
bs=1000 sw=10 sl=64 904 0.552 1,100,943/1,153,418/1,153,418 us ⚪ within ±5% / 🔴 -15.5%
Baseline details

Latest main 957b6c9 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 365 tuples/sec 420 tuples/sec 801.54 tuples/sec -13.1% -54.5%
bs=10 sw=10 sl=64 MB/s 0.223 MB/s 0.256 MB/s 0.489 MB/s -12.9% -54.4%
bs=10 sw=10 sl=64 p50 26,090 us 21,647 us 12,422 us +20.5% +110.0%
bs=10 sw=10 sl=64 p95 34,235 us 30,787 us 15,365 us +11.2% +122.8%
bs=10 sw=10 sl=64 p99 34,235 us 30,787 us 18,987 us +11.2% +80.3%
bs=100 sw=10 sl=64 throughput 749 tuples/sec 820 tuples/sec 1,039 tuples/sec -8.7% -27.9%
bs=100 sw=10 sl=64 MB/s 0.457 MB/s 0.5 MB/s 0.634 MB/s -8.6% -28.0%
bs=100 sw=10 sl=64 p50 129,807 us 122,626 us 98,524 us +5.9% +31.8%
bs=100 sw=10 sl=64 p95 171,118 us 142,009 us 104,534 us +20.5% +63.7%
bs=100 sw=10 sl=64 p99 171,118 us 142,009 us 115,146 us +20.5% +48.6%
bs=1000 sw=10 sl=64 throughput 904 tuples/sec 917 tuples/sec 1,070 tuples/sec -1.4% -15.5%
bs=1000 sw=10 sl=64 MB/s 0.552 MB/s 0.56 MB/s 0.653 MB/s -1.4% -15.5%
bs=1000 sw=10 sl=64 p50 1,100,943 us 1,100,059 us 963,083 us +0.1% +14.3%
bs=1000 sw=10 sl=64 p95 1,153,418 us 1,148,822 us 1,000,730 us +0.4% +15.3%
bs=1000 sw=10 sl=64 p99 1,153,418 us 1,148,822 us 1,026,043 us +0.4% +12.4%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,548.38,200,128000,365,0.223,26090.26,34234.81,34234.81
1,100,10,64,20,2671.18,2000,1280000,749,0.457,129806.87,171117.55,171117.55
2,1000,10,64,20,22129.22,20000,12800000,904,0.552,1100942.51,1153418.02,1153418.02

@VuMartin

Copy link
Copy Markdown
Contributor Author

Closing this since #8341 already addresses #8602.

@VuMartin VuMartin closed this Sep 19, 2026
@xuang7 xuang7 removed the release/v1.2 back porting to release/v1.2 label Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common fix release/v1.3 back porting to release/v1.3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants