Caches

BuildStream uses local caches to avoid repeating work, and can have remote caches configured to allow the results of work to be shared between multiple users. There are caches for both elements and sources that map keys to relevant metadata and point to data in CAS.

Content Addressable Storage (CAS)

The majority of data is stored in Content Addressable Storage or CAS, which indexes stored files by the SHA256 hash of their contents. This allows for a flat file structure as well as any repeated data to be shared across a CAS. In order to store directory structures BuildStream’s CAS uses protocol buffers for storing directory and file information as defined in Googles REAPI.

bst-artifact-server runs a grpc CAS service (also defined in REAPI) that both artifact and source cache use, allowing them to download and upload files to a remote service.

Artifact caches

Artifacts store build results of an element which is then referred to by its cache key (described in Cache keys). The artifacts information is then stored in a protocol buffer, defined in artifact.proto, which includes metadata such as the digest of the files root; strong and weak keys; and log files digests. The digests point to locations in the CAS of relavant files and directories, allowing BuildStream to query remote CAS servers for this information.

bst-artifact-server uses grpc to implement a remote API for an artifact service, that BuildStream then uses to query, retrieve and update artifact files, before using this information to download the files and other data from the remote CAS.

Source caches

Sources are cached by running the Source.stage method and capturing the directory output of this into the CAS, which then use the sources key to refer to this. The source key will be calculated with the plugins defined Plugin.get_unique_key and, depending on whether the source requires previous sources to be staged (e.g. the patch plugin), the unique key of all sources listed before it in an element. Source caches are simpler than artifacts, as they just need to map a source key to a directory digest, with no additional metadata.

Similar to artifacts, bst-artifact-server uses grpc to implements a ‘reference service’ API that allows BuildStream to query for these source digests, which can then be used to retrieve sources from a CAS.

Note

Not all plugins use the same result as the staged output for workspaces. As a result when initialising a workspace, BuildStream may require fetching the original source if it only has the source in the source cache.