Cache keys

Cache keys for artifacts are generated from the inputs of the build process for the purpose of reusing artifacts in a well-defined, predictable way.

Structure

Cache keys are SHA256 hash values generated from a UTF-8 JSON document that includes:

  • Environment (e.g., project configuration and variables).

  • Element configuration (details depend on element kind, Element.get_unique_key()).

  • Sources (Source.get_unique_key()).

  • Dependencies (depending on cache key type, see below).

  • Public data.

Cache key types

There are two types of cache keys in BuildStream, strong and weak.

The purpose of a strong cache key is to capture the state of as many aspects as possible that can have an influence on the build output. The aim is that builds will be fully reproducible as long as the cache key doesn’t change, with suitable module build systems that don’t embed timestamps, for example.

A strong cache key includes the strong cache key of each build dependency (and their runtime dependencies) of the element as changes in build dependencies (or their runtime dependencies) can result in build differences in reverse dependencies. This means that whenever the strong cache key of a dependency changes, the strong cache key of its reverse dependencies will change as well.

A weak cache key has an almost identical structure, however, it includes only the names of build dependencies, not their cache keys or their runtime dependencies. A weak cache key will thus still change when the element itself or the environment changes but it will not change when a dependency is updated.

For elements without build dependencies the strong cache key is identical to the weak cache key.

Note that dependencies which are not required at build time do not affect either kind of key.

Strict build plan

This is the default build plan that exclusively uses strong cache keys for the core functionality. An element’s cache key can be calculated when the cache keys of the element’s build dependencies (and their runtime dependencies) have been calculated and either tracking is not enabled or it has already completed for this element, i.e., the ref is available. This means that with tracking disabled the cache keys of all elements could be calculated right at the start of a build session.

While BuildStream only uses strong cache keys with the strict build plan for the actual staging and build process, it will still calculate weak cache keys for each element. This allows BuildStream to store the artifact in the cache with both keys, reducing rebuilds when switching between strict and non-strict build plans. If the artifact cache already contains an artifact with the same weak cache key, it’s replaced. Thus, non-strict builds always use the latest artifact available for a given weak cache key.

Non-strict build plan

The non-strict build plan disables the time-consuming automatic rebuild of reverse dependencies at the cost of dropping the reproducibility benefits. It uses the weak cache keys for the core staging and build process. I.e., if an artifact is available with the calculated weak cache key, it will be reused for staging instead of being rebuilt. weak cache keys can be calculated early in the build session. After tracking, similar to when strong cache keys can be calculated with a strict build plan.

Similar to how strict build plans also calculate weak cache keys, non-strict build plans also calculate strong cache keys. However, this is slightly more complex. To calculate the strong cache key of an element, BuildStream requires the strong cache keys of the build dependencies (and their runtime dependencies).

The build dependencies of an element may have been updated since the artifact was built. With the non-strict build plan the artifact will still be reused. However, this means that we cannot use a strong cache key calculated purely based on the element definitions. We need a cache key that matches the environment at the time the artifact was built, not the current definitions.

The only way to get the correct strong cache key is by retrieving it from the metadata stored in the artifact. As artifacts may need to be pulled from a remote artifact cache, the strong cache key is not readily available early in the build session. However, it can always be retrieved when an element is about to be built, as the dependencies are guaranteed to be in the local artifact cache at that point.

Element._get_cache_key_from_artifact() extracts the strong cache key from an artifact in the local cache. Element._get_cache_key_for_build() calculates the strong cache key that is used for a particular build job. This is used for the embedded metadata and also as key to store the artifact in the cache.