Skip to content

Firebolt: do a full restart before every cold run #1666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexey-milovidov merged 3 commits into from
Aug 29, 2026
+1,629 −45
Merged

Firebolt: do a full restart before every cold run #1666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f7e2073
Firebolt: do a full restart before every cold run
alexey-milovidov Aug 29, 2026
2d23c1b
Add benchmark results for firebolt, firebolt-parquet, firebolt-parque…
github-actions[bot] Aug 29, 2026
c238d75
Add benchmark results for firebolt, firebolt-parquet, firebolt-parque…
github-actions[bot] Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

Changes in the benchmark methodology or presentation, as well as major news.

### 2026-08-29
Firebolt now does a true cold run. The three self-hosted firebolt-core entries (`firebolt`, `firebolt-parquet`, `firebolt-parquet-partitioned`) set `BENCH_RESTARTABLE=no` and therefore only got their page cache flushed before each first run; they now go through the common runner's full `./stop` → `drop_caches` → `./start` cycle like every other daemon, and lost the `no-cold` tag. The engine handles `SIGTERM` and shuts down cleanly, and the database lives on a bind-mounted volume, so it is still there when the container is started back up. The result files produced before this change keep the tag. The managed-cloud Firebolt results from 2025-06-07 also keep it: those ran against the hosted service, which cannot be restarted.

### 2026-08-28
Renamed the `lukewarm-cold-run` tag to `no-cold` — a shorter name for the same thing: an entry whose first run of each query is not a true cold run, because the system is not restarted before it. The classification was also brought up to date. The 37 systems that the common runner in `lib/` already stops, page-cache-flushes and restarts before every cold run lost the tag, as did the 82 result files it had already produced; the managed services that clear nothing at all before a first run (ClickHouse Cloud, Databricks, MotherDuck, Hologres, AlloyDB) gained it, as [#1646](https://github.com/ClickHouse/ClickBench/pull/1646) requires.

Expand Down
1 change: 0 additions & 1 deletion firebolt-parquet-partitioned/benchmark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/bin/bash
# Thin shim — actual flow is in lib/benchmark-common.sh.
export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-partitioned"
export BENCH_RESTARTABLE=no
exec ../lib/benchmark-common.sh
60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c6a.2xlarge.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c6a.2xlarge",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 9,
"data_size": 14737666736,
"concurrent_qps": 0.077,
"concurrent_error_ratio": 0.981,
"result": [
[0.094, 0.022, 0.023],
[2.423, 0.173, 0.191],
[4.371, 0.379, 0.362],
[3.426, 0.283, 0.284],
[3.619, 0.79, 0.792],
[4.083, 1.218, 1.215],
[2.439, 0.202, 0.216],
[2.413, 0.203, 0.17],
[3.981, 0.99, 0.988],
[8.617, 1.525, 1.571],
[6.237, 0.73, 0.667],
[6.239, 0.806, 0.792],
[4.056, 1.093, 1.142],
[8.069, 1.813, 1.767],
[4.021, 1.334, 1.341],
[3.487, 0.827, 0.819],
[7.856, 2.299, 2.333],
[7.563, 2.168, 1.989],
[11.142, 5.996, 5.91],
[2.589, 0.192, 0.195],
[12.48, 3.529, 3.392],
[16.597, 4.208, 4.283],
[28.807, 7.325, 7.324],
[56.117, 29.226, 27.957],
[8.151, 1.146, 1.135],
[3.757, 0.82, 0.794],
[8.154, 1.134, 1.13],
[14.163, 4.168, 4.322],
[11.451, 6.073, 6.027],
[2.495, 0.269, 0.205],
[9.019, 1.574, 1.571],
[14.495, 1.858, 1.806],
[13.098, 3.892, 3.881],
[12.972, 5.502, 6.689],
[13.16, 5.068, 5.202],
[3.369, 1.078, 1.042],
[0.278, 0.103, 0.084],
[0.205, 0.099, 0.072],
[0.229, 0.077, 0.082],
[0.516, 0.316, 0.257],
[0.185, 0.042, 0.052],
[0.187, 0.045, 0.036],
[0.201, 0.06, 0.068]
]
}

60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c6a.4xlarge.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c6a.4xlarge",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 20,
"data_size": 14737666736,
"concurrent_qps": 0.507,
"concurrent_error_ratio": 0,
"result": [
[0.116, 0.018, 0.017],
[2.393, 0.128, 0.132],
[4.371, 0.253, 0.251],
[3.43, 0.245, 0.244],
[3.545, 0.5, 0.531],
[4.044, 0.853, 0.762],
[2.422, 0.154, 0.147],
[2.428, 0.128, 0.128],
[3.902, 0.671, 0.66],
[8.472, 1.001, 0.959],
[6.165, 0.479, 0.466],
[6.211, 0.556, 0.541],
[4.082, 0.762, 0.761],
[7.881, 1.194, 1.26],
[4.127, 0.871, 0.932],
[3.619, 0.668, 0.673],
[7.637, 1.374, 1.408],
[7.583, 1.277, 1.206],
[10.678, 3.166, 3.144],
[2.602, 0.203, 0.2],
[12.444, 3.237, 3.238],
[16.554, 3.57, 3.604],
[28.786, 6.662, 6.657],
[56.443, 17.339, 17.695],
[8.15, 0.852, 0.835],
[3.792, 0.594, 0.6],
[8.169, 0.841, 0.84],
[14.147, 3.972, 3.95],
[11.166, 4.488, 4.493],
[2.498, 0.162, 0.163],
[8.97, 1.077, 1.031],
[14.437, 1.337, 1.282],
[12.401, 2.278, 2.239],
[12.908, 4.103, 4.236],
[12.9, 4.17, 4.217],
[3.208, 0.647, 0.675],
[0.306, 0.087, 0.104],
[0.509, 0.077, 0.076],
[0.405, 0.09, 0.081],
[0.514, 0.264, 0.239],
[0.248, 0.04, 0.037],
[0.192, 0.042, 0.039],
[0.202, 0.073, 0.066]
]
}

60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c6a.large.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c6a.large",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 0,
"data_size": 14737666736,
"concurrent_qps": 0.128,
"concurrent_error_ratio": 0.962,
"result": [
[0.129, 0.084, 0.085],
[2.335, 0.574, 0.618],
[4.289, 1.105, 0.991],
[3.341, 1.017, 1.065],
[4.053, 2.813, 2.771],
[5.829, 4.861, 5.048],
[2.339, 0.729, 0.608],
[2.33, 0.556, 0.557],
[5.018, 4.478, 4.356],
[12.906, 12.397, 13.333],
[6.148, 2.257, 2.09],
[6.191, 2.605, 2.561],
[5.466, 4.168, 4.116],
[14.388, 13.759, 14.062],
[6.706, 6.592, 6.485],
[4.68, 3.793, 3.649],
[15.803, 15.841, 15.889],
[12.109, 12.412, 12.216],
[31.197, 31.838, 31.617],
[2.606, 0.7, 0.69],
[25.9, 14.821, 13.019],
[25.334, 27.082, 27.631],
[33.646, 36.495, 36.804],
[100.557, 101.008, 101.917],
[8.209, 3.65, 3.622],
[3.843, 3.385, 3.456],
[8.209, 4.187, 4.176],
[30.083, 29.935, 31.223],
[23.92, 33.414, 32.888],
[2.507, 0.742, 0.726],
[9.979, 8.988, 10.317],
[18.479, 18.316, 19.22],
[49.739, 49.53, 49.829],
[46.429, null, null],
[47.26, null, null],
[3.777, 3.478, 3.373],
[0.622, 0.526, 0.509],
[0.461, 0.226, 0.227],
[0.616, 0.411, 0.378],
[1.185, 0.606, 0.553],
[0.331, 0.106, 0.115],
[0.292, 0.074, 0.083],
[0.339, 0.117, 0.117]
]
}

60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c6a.metal.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c6a.metal",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 63,
"data_size": 14737666736,
"concurrent_qps": 3.307,
"concurrent_error_ratio": 0,
"result": [
[0.315, 0.035, 0.037],
[3.221, 0.069, 0.07],
[5.1, 0.13, 0.109],
[4.152, 0.092, 0.1],
[4.278, 0.193, 0.18],
[4.714, 0.241, 0.207],
[2.181, 0.07, 0.068],
[2.19, 0.068, 0.078],
[3.602, 0.186, 0.197],
[7.909, 0.302, 0.258],
[5.706, 0.157, 0.143],
[5.715, 0.159, 0.151],
[3.741, 0.269, 0.309],
[7.242, 0.322, 0.3],
[3.814, 0.226, 0.226],
[3.333, 0.25, 0.167],
[7.246, 0.372, 0.35],
[7.218, 0.334, 0.35],
[10.066, 0.642, 0.556],
[2.481, 0.091, 0.118],
[12.357, 0.794, 0.721],
[16.319, 0.891, 0.798],
[28.541, 1.439, 1.332],
[55.638, 4.035, 3.529],
[7.843, 0.341, 0.236],
[3.696, 0.195, 0.149],
[7.893, 0.254, 0.267],
[14.095, 1.031, 0.868],
[10.963, 0.945, 0.914],
[2.359, 0.089, 0.092],
[8.671, 0.266, 0.242],
[14.043, 0.418, 0.385],
[11.38, 0.631, 0.661],
[12.614, 1.03, 0.942],
[12.553, 1.051, 0.912],
[2.989, 0.14, 0.173],
[0.259, 0.131, 0.128],
[0.219, 0.111, 0.098],
[0.242, 0.126, 0.111],
[0.43, 0.278, 0.264],
[0.205, 0.067, 0.068],
[0.186, 0.071, 0.067],
[0.205, 0.095, 0.093]
]
}

60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c6a.xlarge.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c6a.xlarge",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 0,
"data_size": 14737666736,
"concurrent_qps": 0.088,
"concurrent_error_ratio": 0.019,
"result": [
[0.128, 0.034, 0.034],
[2.432, 0.246, 0.266],
[4.277, 0.571, 0.579],
[3.34, 0.496, 0.444],
[3.699, 1.371, 1.362],
[3.976, 2.43, 2.366],
[2.339, 0.339, 0.342],
[2.328, 0.271, 0.276],
[4.04, 1.785, 1.767],
[8.768, 2.852, 2.825],
[6.109, 1.155, 1.21],
[6.137, 1.39, 1.373],
[3.966, 2.033, 2.008],
[8.507, 3.285, 3.362],
[4.133, 2.381, 2.344],
[3.651, 1.568, 1.531],
[8.372, 4.421, 4.342],
[7.781, 3.822, 4.167],
[12.378, 11.732, 11.838],
[2.591, 0.33, 0.294],
[12.644, 6.084, 6.669],
[16.51, 7.558, 7.479],
[29.002, 17.822, 16.66],
[56.481, 52.184, 52.965],
[8.195, 2.021, 2.003],
[3.825, 1.399, 1.581],
[8.211, 1.955, 2.006],
[14.166, 7.744, 8.076],
[11.562, 11.317, 11.599],
[2.513, 0.351, 0.368],
[9.242, 2.923, 2.858],
[14.697, 3.296, 3.293],
[21.072, 16.263, 17.321],
[18.272, 17.46, 16.843],
[null, 14.405, 17.407],
[3.37, 1.877, 1.789],
[0.302, 0.132, 0.15],
[0.296, 0.236, 0.128],
[0.281, 0.205, 0.128],
[0.529, 0.305, 0.308],
[0.245, 0.054, 0.048],
[0.192, 0.052, 0.059],
[0.218, 0.103, 0.078]
]
}

60 changes: 60 additions & 0 deletions firebolt-parquet-partitioned/results/20260829/c7a.metal-48xl.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"system": "Firebolt (Parquet, partitioned)",
"date": "2026-08-29",
"machine": "c7a.metal-48xl",
"cluster_size": 1,
"proprietary": "yes",
"hardware": "cpu",
"tuned": "no",
"tags": ["C++","column-oriented","PostgreSQL compatible","ClickHouse derivative","stateless"],
"load_time": 62,
"data_size": 14737666736,
"concurrent_qps": 6.018,
"concurrent_error_ratio": 0,
"result": [
[0.25, 0.149, 0.209],
[2.354, 0.129, 0.212],
[4.345, 0.168, 0.193],
[3.384, 0.144, 0.204],
[3.457, 0.347, 0.35],
[3.902, 0.249, 0.273],
[2.38, 0.146, 0.215],
[2.402, 0.136, 0.239],
[3.701, 0.22, 0.212],
[8.174, 0.291, 0.226],
[5.79, 0.182, 0.269],
[5.817, 0.194, 0.2],
[3.754, 0.211, 0.236],
[7.226, 0.381, 0.381],
[3.964, 0.254, 0.25],
[3.308, 0.212, 0.275],
[7.271, 0.344, 0.357],
[7.327, 0.322, 0.266],
[10.249, 0.523, 0.457],
[2.522, 0.1, 0.15],
[12.39, 0.49, 0.415],
[16.285, 0.57, 0.508],
[28.656, 0.827, 0.797],
[55.891, 2.779, 2.655],
[7.838, 0.257, 0.157],
[3.707, 0.202, 0.159],
[7.899, 0.173, 0.188],
[14.014, 0.639, 0.519],
[10.938, 0.617, 0.53],
[2.412, 0.245, 0.207],
[8.694, 0.269, 0.277],
[13.931, 0.331, 0.253],
[11.496, 0.519, 0.445],
[12.606, 0.717, 0.667],
[12.631, 0.775, 0.767],
[2.95, 0.213, 0.589],
[0.46, 0.204, 0.194],
[0.385, 0.157, 0.161],
[0.36, 0.167, 0.156],
[0.535, 0.34, 0.334],
[0.333, 0.188, 0.127],
[0.261, 0.12, 0.143],
[0.325, 0.149, 0.141]
]
}

Loading