Conversation
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8608 +/- ##
=========================================
Coverage 92.76% 92.76%
- Complexity 4897 4898 +1
=========================================
Files 1236 1236
Lines 52155 52157 +2
Branches 6409 6410 +1
=========================================
+ Hits 48380 48382 +2
Misses 2194 2194
Partials 1581 1581
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 365 | 0.223 | 26,090/34,235/34,235 us | 🔴 +20.5% / 🔴 +122.8% |
| 🔴 | bs=100 sw=10 sl=64 | 749 | 0.457 | 129,807/171,118/171,118 us | 🔴 +20.5% / 🔴 +63.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 904 | 0.552 | 1,100,943/1,153,418/1,153,418 us | ⚪ within ±5% / 🔴 -15.5% |
Baseline details
Latest main 957b6c9 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 365 tuples/sec | 420 tuples/sec | 801.54 tuples/sec | -13.1% | -54.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.223 MB/s | 0.256 MB/s | 0.489 MB/s | -12.9% | -54.4% |
| bs=10 sw=10 sl=64 | p50 | 26,090 us | 21,647 us | 12,422 us | +20.5% | +110.0% |
| bs=10 sw=10 sl=64 | p95 | 34,235 us | 30,787 us | 15,365 us | +11.2% | +122.8% |
| bs=10 sw=10 sl=64 | p99 | 34,235 us | 30,787 us | 18,987 us | +11.2% | +80.3% |
| bs=100 sw=10 sl=64 | throughput | 749 tuples/sec | 820 tuples/sec | 1,039 tuples/sec | -8.7% | -27.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.457 MB/s | 0.5 MB/s | 0.634 MB/s | -8.6% | -28.0% |
| bs=100 sw=10 sl=64 | p50 | 129,807 us | 122,626 us | 98,524 us | +5.9% | +31.8% |
| bs=100 sw=10 sl=64 | p95 | 171,118 us | 142,009 us | 104,534 us | +20.5% | +63.7% |
| bs=100 sw=10 sl=64 | p99 | 171,118 us | 142,009 us | 115,146 us | +20.5% | +48.6% |
| bs=1000 sw=10 sl=64 | throughput | 904 tuples/sec | 917 tuples/sec | 1,070 tuples/sec | -1.4% | -15.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.552 MB/s | 0.56 MB/s | 0.653 MB/s | -1.4% | -15.5% |
| bs=1000 sw=10 sl=64 | p50 | 1,100,943 us | 1,100,059 us | 963,083 us | +0.1% | +14.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,153,418 us | 1,148,822 us | 1,000,730 us | +0.4% | +15.3% |
| bs=1000 sw=10 sl=64 | p99 | 1,153,418 us | 1,148,822 us | 1,026,043 us | +0.4% | +12.4% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,548.38,200,128000,365,0.223,26090.26,34234.81,34234.81
1,100,10,64,20,2671.18,2000,1280000,749,0.457,129806.87,171117.55,171117.55
2,1000,10,64,20,22129.22,20000,12800000,904,0.552,1100942.51,1153418.02,1153418.02
What changes were proposed in this PR?
Fix schema inference for CSV and JSONL file-scan operators when
Limit = 0.Previously, the scan limit was also used to determine how many rows were sampled for schema inference. When the limit was
0, no rows were sampled, causing schema inference to produce an empty schema or, for the old CSV scanner, anArrayIndexOutOfBoundsException.This change ensures schema inference samples at least one row when the scan limit is
0, while the actual scan output remains limited to zero rows.Before:

After:

Any related issues, documentation, discussions?
How was this PR tested?
Added regression tests verifying that CSV, Parallel CSV, CSVOld, and JSONL file-scan operators still infer the expected columns when
Limit = 0.The CSV and JSONL test suites were run with:
sbt 'WorkflowOperator / Test / testOnly org.apache.texera.amber.operator.source.scan.csv.CSVScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.json.JSONLScanSourceOpDescSpec'Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Sonnet 5)