Skip to content

ci: rotate bazel cache keys on module bumps - #366

Merged
antonkri merged 1 commit into
mainfrom
ci/cache-key-known-good
Sep 24, 2026
Merged

antonkri merged 1 commit into
mainfrom
ci/cache-key-known-good

Conversation

@antonkri

Copy link
Copy Markdown
Contributor

setup-bazel derives the cache key hash for both the repository cache and the disk cache from the repository-cache file list. From config.js in 0.18.0:

let repositoryCacheFiles = [`${moduleRoot}/MODULE.bazel`, `${moduleRoot}/WORKSPACE.bazel`, ...]
if (repositoryCacheEnabled) {
  if (repositoryCacheConfig !== true) { repositoryCacheFiles = Array(repositoryCacheConfig).flat() }
}
// ...
diskCache: { files: [...repositoryCacheFiles, `${moduleRoot}/**/BUILD.bazel`, `${moduleRoot}/**/BUILD`], ... }

The README describes this input as the repository-cache key only; the coupling to the disk-cache key exists in code but is undocumented.

Why the caches went stale

Neither MODULE.bazel nor WORKSPACE* changes when we bump a module in known_good.json or bazel_common/*.MODULE.bazel, so the key does not rotate on a module bump. And post.js skips the save whenever the restored key was an exact match:

const cacheHit = core.getState(`${name}-cache-hit`)
if (cacheHit === 'true') { return }

Taken together: after the very first save, no run writes a cache again. The entry persists, keeps serving whatever module states happened to be current when it was created, and never fails - Bazel addresses its entries by action key, so stale content is simply never hit. The failure mode is a silent loss of reuse, not a wrong result.

Concrete evidence: the kyron bump in #364 produced disk-cache key ...4477fe5d..., byte-identical to the key in use before the bump. Over the last three months the repository cache rotated 3 times while module states changed 23 times.

Change

List the files that actually invalidate Bazel action keys or the set of fetched module sources:

  • known_good.json and bazel_common/*.MODULE.bazel - module states and toolchain deps
  • patches/** - patch contents alter module sources via bazel_patches without necessarily touching known_good.json
  • .bazelrc - build and test flags feed action keys

WORKSPACE* is dropped because this repository has no such file under Bzlmod. **/BUILD.bazel and **/BUILD are appended by the action itself and remain in effect.

No cold start on merge: the restore-key prefix is unchanged, so the first run still restores the existing content by prefix match and re-saves it under the new key.

Effect

Measured on run 35980109957, after the stale entry had been refreshed once by hand, against a re-run of an older PR on the same day and the same cache entry:

with fresh cache with stale cache
unit tests 6:36 18:13
job total 25:50 44:11
tests executed 1 of 628 628 of 628

baselibs, identical process totals in both runs:

stale: 5432 processes: 2416 disk cache hit, 573 processwrapper-sandbox
fresh: 5432 processes: 2988 disk cache hit,   1 processwrapper-sandbox

Cost

Rotations on main over the last three months: 29 instead of 3. Saving stays gated on push to main, so no PR run pays for it. The additional repository-cache upload measured roughly 90 seconds for 3.3 GB on the merge run.

setup-bazel derives the cache key hash for both the repository cache and
the disk cache from the `repository-cache` file list -- config.js spreads
repositoryCacheFiles into diskCache.files, which the README does not
mention. The default list covers only MODULE.bazel and WORKSPACE*, and
neither changes when a module is bumped in known_good.json or in
bazel_common/*.MODULE.bazel.

So the key never rotated. post.js returns early when the restored key
matches exactly, which means that after the very first save no run ever
wrote a cache again: the disk cache kept serving the results of whatever
module states happened to be current when it was created, and the
repository cache rotated three times in three months while module states
changed twenty times. Nothing ever failed, because Bazel addresses its
entries by action key -- stale entries are simply never hit.

List the files that actually invalidate action keys or the set of fetched
module sources. Saving stays gated on push-to-main, so only the merge run
pays the additional upload; measured against the existing repository
cache that is roughly 90 seconds for 3.3 GB.
@github-actions

Copy link
Copy Markdown

The created documentation from the pull request is available at: docu-html

@lurtz

lurtz commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Are you aware that @AlexanderLanin is currently reworking bazel and caching with https://github.com/eclipse-score/cicd-actions/tree/main/setup-bazel-cache ?

You might want to try and test that.

@antonkri
antonkri merged commit eed5160 into main Sep 24, 2026
16 of 17 checks passed
@antonkri
antonkri deleted the ci/cache-key-known-good branch September 24, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants