Firebolt: do a full restart before every cold run

[alexey-milovidov]

Follow-up to #1648 (#1648 (comment)): rather than leaving the three self-hosted firebolt-core entries tagged no-cold, make them do a true cold run.

firebolt, firebolt-parquet and firebolt-parquet-partitioned set BENCH_RESTARTABLE=no, so lib/benchmark-common.sh skipped the ./stop → drop_caches → ./start cycle and only flushed the page cache before the first run of each query — a lukewarm cold run per README section 2.b.

Nothing forces that: the engine data directory is a host bind mount (fb-volume), so the clickbench database is still there after the container is stopped and started back up. This drops the BENCH_RESTARTABLE override so the common runner restarts the engine before every cold run like it does for every other daemon, and drops no-cold from the three templates.

./stop gets -t 120 instead of docker's 10 s default grace period. The shutdown is prompt, so docker stop returns as soon as the engine exits and the longer window costs nothing; without it a slower flush on the full dataset would be SIGKILLed, and the recovery work on the next ./start would land inside the next query's cold measurement.

Verified locally

Against ghcr.io/firebolt-db/engine:dev (arm64) with a 10M-row ingest:

• the engine handles SIGTERM — clean shutdown log (Shutting down the server.), container exit code 0;

• the clickbench database and its row count survive ./stop + ./start (10,000,000 rows before and after);

• ./check starts failing right after ./stop, so bench_wait_stopped returns immediately instead of burning its 60 s timeout;

• the first try after a restart is 10–20x slower than the two warm tries that follow, i.e. the cold number is actually cold:

  query	try 1 (after restart)	try 2	try 3
  SELECT COUNT(*)	0.033	0.0028	0.0029
  SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth)	0.289	0.0148	0.0144

Wall-clock cost of the extra 43 restarts could not be measured on the box used for the local check (an overloaded shared dev machine where the docker daemon alone takes minutes per operation); the run on the benchmark machines will show it.

Results

The existing result files keep no-cold — they were measured without a restart. The 2025-06-07 managed-cloud results keep it too: that is the hosted service, which cannot be restarted. Fresh results for the three entries come from this PR's benchmark run; add a machine:all label to re-run the whole matrix instead of just c6a.4xlarge.

impala is the other local daemon #1648 tagged, and it stays tagged: restarting catalogd would empty Impala's in-memory catalog.

🤖 Generated with Claude Code

[github-actions]

The run of firebolt on t3a.small did not produce results.

Logs:

• firebolt on t3a.small: benchmark log, cloud-init log

[github-actions]

Results for firebolt are ready for: c6a.2xlarge, c6a.4xlarge, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl.
Results for firebolt-parquet are ready for: c6a.2xlarge, c6a.4xlarge, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl, t3a.small.
Results for firebolt-parquet-partitioned are ready for: c6a.2xlarge, c6a.4xlarge, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl, t3a.small.
The result files are committed as 2d23c1b.

Logs:

• firebolt-parquet on t3a.small: benchmark log, cloud-init log
• firebolt on c6a.metal: benchmark log, cloud-init log
• firebolt on c8g.metal-48xl: benchmark log, cloud-init log
• firebolt on c7a.metal-48xl: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c6a.metal: benchmark log, cloud-init log
• firebolt on c6a.4xlarge: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c7a.metal-48xl: benchmark log, cloud-init log
• firebolt on c8g.4xlarge: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c8g.4xlarge: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c6a.4xlarge: benchmark log, cloud-init log
• firebolt-parquet on c6a.4xlarge: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c8g.metal-48xl: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c6a.2xlarge: benchmark log, cloud-init log
• firebolt on c6a.2xlarge: benchmark log, cloud-init log
• firebolt on c6a.xlarge: benchmark log, cloud-init log
• firebolt-parquet on c6a.xlarge: benchmark log, cloud-init log
• firebolt-parquet on c8g.4xlarge: benchmark log, cloud-init log
• firebolt-parquet on c6a.2xlarge: benchmark log, cloud-init log
• firebolt-parquet on c6a.metal: benchmark log, cloud-init log
• firebolt-parquet on c8g.metal-48xl: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c6a.xlarge: benchmark log, cloud-init log
• firebolt-parquet on c7a.metal-48xl: benchmark log, cloud-init log
• firebolt-parquet-partitioned on t3a.small: benchmark log, cloud-init log

[github-actions]

Results for firebolt are ready for: c6a.large.
Results for firebolt-parquet are ready for: c6a.large.
Results for firebolt-parquet-partitioned are ready for: c6a.large.
The result files are committed as c238d75.

Logs:

• firebolt-parquet on c6a.large: benchmark log, cloud-init log
• firebolt on c6a.large: benchmark log, cloud-init log
• firebolt-parquet-partitioned on c6a.large: benchmark log, cloud-init log