BuildStream uses local caches to avoid repeating work, and can have remote caches configured to allow the results of work to be shared between multiple users. There are caches for both elements and sources that map keys to relevant metadata and point to data in CAS.
Content Addressable Storage (CAS)¶
The majority of data is stored in Content Addressable Storage or CAS, which indexes stored files by the SHA256 hash of their contents. This allows for a flat file structure as well as any repeated data to be shared across a CAS. In order to store directory structures BuildStream’s CAS uses protocol buffers for storing directory and file information as defined in Googles REAPI.
Artifacts store build results of an element which is then referred to by its
cache key (described in Cache keys). The artifacts information is then
stored in a protocol buffer, defined in
artifact.proto, which includes
metadata such as the digest of the files root; strong and weak keys; and log
files digests. The digests point to locations in the CAS of relavant files and
directories, allowing BuildStream to query remote CAS servers for this
bst-artifact-server uses grpc to implement the Remote Asset API for an artifact service, that BuildStream then uses to query, retrieve and update artifact references, before using this information to download the files and other data from the remote CAS.
Sources are cached by running the
Source.stage method and capturing the directory output of
this into the CAS, which then use the sources key to refer to this. The source
key will be calculated with the plugins defined
Plugin.get_unique_key and, depending on whether the source
requires previous sources to be staged (e.g. the patch plugin), the unique key
of all sources listed before it in an element. Source caches are simpler than
artifacts, as they just need to map a source key to a directory digest, with no
Similar to artifacts, bst-artifact-server uses grpc to implement the Remote Asset API that allows BuildStream to query for these source digests, which can then be used to retrieve sources from a CAS.
Not all plugins use the same result as the staged output for workspaces. As a result when initialising a workspace, BuildStream may require fetching the original source if it only has the source in the source cache.