Remove the unstable encodings feature - #9710
Conversation
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | cuda/bitpacked_u8/unpack/3bw[100M] |
299.3 µs | 345.9 µs | -13.46% |
| ❌ | WallTime | mul_i32_nonnull_avx512 |
7.1 µs | 7.9 µs | -10.21% |
| ⚡ | WallTime | words_gather_dispatch_avx2[1024] |
46 ns | 17 ns | ×2.7 |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/remove-unstable-encodings (53b9a4d) with develop (a13c246)3
Footnotes
-
176 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
-
No successful run was found on
develop(9fdaa38) during the generation of this report, so a13c246 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
2649c0b to
7102677
Compare
82cdf1c to
9db048d
Compare
0af6657 to
4390970
Compare
2da0446 to
3c62929
Compare
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.763x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (0.918x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.913x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.958x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.941x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.979x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (1.016x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.076x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.893x ✅, 3↑ 0↓)
duckdb / vortex-compact / ns (0.959x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 1↓)
File Size Changes (2 files changed, +0.3% overall, 1↑ 1↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.716x ✅, 10↑ 0↓)
datafusion / vortex-compact / ns (1.009x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.735x ✅, 7↑ 0↓)
duckdb / vortex-compact / ns (1.014x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 1↑ 1↓)
File Size Changes (200 files changed, +0.2% overall, 118↑ 82↓)
Totals:
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (6)
Powered by Polar Signals Cloud |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.875x ✅, 5↑ 0↓)
vortex / vortex-file-compressed / % (1.086x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
File Size Changes (16 files changed, +6.0% overall, 9↑ 7↓)
Totals:
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.957x ➖, 5↑ 0↓)
vortex / vortex-file-compressed / bytes (1.021x ➖, 0↑ 2↓)
vortex / vortex-file-compressed / ratio (0.975x ➖, 5↑ 2↓)
vortex / parquet / ns (1.005x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.003x ➖, 0↑ 2↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +8.6% overall, 1↑ 0↓)
Totals:
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (0.899x ✅, 3↑ 0↓)
random-access / vortex-file-compressed / ns (0.980x ➖, 2↑ 0↓)
random-access / parquet / ns (0.993x ➖, 0↑ 0↓)
random-access / lance / ns (0.988x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.068x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (1.079x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.924x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.079x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.888x ✅, 14↑ 0↓)
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.026x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.940x ➖, 10↑ 2↓)
duckdb / vortex-compact / ns (1.000x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
File Size Changes (200 files changed, +2.3% overall, 145↑ 55↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 4↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.968x ➖, 10↑ 3↓)
duckdb / vortex-compact / ns (1.006x ➖, 3↑ 4↓)
duckdb / parquet / ns (0.994x ➖, 2↑ 3↓)
File Size Changes (48 files changed, +1.6% overall, 23↑ 25↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 1↓)
File Size Changes (16 files changed, +6.5% overall, 9↑ 7↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.870x ✅, 5↑ 0↓)
duckdb / vortex-compact / ns (1.025x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
File Size Changes (2 files changed, +10.1% overall, 1↑ 1↓)
Totals:
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.009x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.987x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -0.0% overall, 1↑ 8↓)
Totals:
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.805x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.971x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.950x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.870x ➖, 1↑ 0↓)
|
Use enabled editions to filter the default writer and golden compressor configurations. Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
OnPair compression brings the customer fixture below DataFusion’s 1 MiB file splitting threshold. Refresh the eight affected physical plans while preserving query results and normalized byte ranges. Signed-off-by: "Robert Kruszewski" <github@robertk.io>
6472998 to
53b9a4d
Compare
| .map_err(|error| vortex_err!("{error}")) | ||
| .vortex_expect("Zstd edition declaration is valid"); |
| #[cfg(feature = "zstd")] | ||
| pub use zstd_buffers::ZstdBuffersScheme; |
There was a problem hiding this comment.
shall we move this into encodings?
unstable_encodings was only ever necessary to let us try things out without immediately introducing stability guarantees. We have editions now, that's the way things should be restricted, not via compile flags