Caches

BuildStream uses local caches to avoid repeating work, and can have remote caches configured to allow the results of work to be shared between multiple users. There are caches for both elements and sources that map keys to relevant metadata and point to data in CAS.

Content Addressable Storage (CAS)

The majority of data is stored in Content Addressable Storage or CAS, which indexes stored files by the SHA256 hash of their contents. This allows for a flat file structure as well as any repeated data to be shared across a CAS. In order to store directory structures BuildStream’s CAS uses protocol buffers for storing directory and file information as defined in Googles REAPI.

The data itself is stored in CAS which is defined by the remote execution protocol, and BuildStream also uses the remote asset protocol in order to address stored content using symbolic labesl, such as artifact names for artifacts.

Artifact caches

Artifacts store build results of an element which is then referred to by its cache key (described in Cache keys). The artifacts information is then stored in a protocol buffer, defined in artifact.proto, which includes metadata such as the digest of the files root; strong and weak keys; and log files digests. The digests point to locations in the CAS of relavant files and directories, allowing BuildStream to query remote CAS servers for this information.

Source caches

Sources are cached by running the Source.stage method and capturing the directory output of this into the CAS, which then use the sources key to refer to this. The source key will be calculated with the plugins defined Plugin.get_unique_key and, depending on whether the source requires previous sources to be staged (e.g. the patch plugin), the unique key of all sources listed before it in an element. Source caches are simpler than artifacts, as they just need to map a source key to a directory digest, with no additional metadata.

Note

Not all plugins use the same result as the staged output for workspaces. As a result when initialising a workspace, BuildStream may require fetching the original source if it only has the source in the source cache.