Configuring Cache Servers
BuildStream caches the results of builds in a local artifact cache, and will avoid building an element if there is a suitable build already present in the local artifact cache. Similarly it will cache sources and avoid pulling them if present in the local cache. See caches for more details.
In addition to the local caches, you can configure one or more remote caches and BuildStream will then try to pull a suitable object from one of the remotes, falling back to performing a local build or fetching a source if needed.
On the client side, cache servers are declared and configured in user configuration, and since it is typical for projects to maintain their own cache servers, it is also possible for projects to provide recommended artifact cache servers and source cache servers through project configuration, so that downstream users can download from services provided by upstream projects by default.
Setting up a remote cache
BuildStream relies on the ContentAddressableStorage protocol in order to exchange data with remote services, in concert with the remote asset protocol in order to assign symbolic labels (such as artifact names) to identify stored content. As such, BuildStream is able to function with any implementations of these two services.
Here are some details about known open source implementations of the required protocols
The Buildbarn project provides a remote execution service implementation for use in build tooling such as BuildStream, Bazel and recc, the bb-storage and bb-remote-asset services are tested to work as cache service for BuildStream’s artifact and source caches.
A simple configuration to spin up the service using docker compose follows:
## # Buildbarn Compose manifest for BuildStream. # # Spins-up a unnamed and unauthenticated cache server: # - STORAGE at http://localhost:7982 # - INDEX at: http://localhost:7981 # # BuildStream configuration snippet: # # artifacts: # - url: https://localhost:7981 # type: index # push: true # - url: https://localhost:7982 # type: storage # push: true # # Basic usage: # - docker-compose -f ci.buildbarn.yml up # - docker-compose -f ci.buildbarn.yml down version: '3.4' services: bb-asset: image: buildbarn/bb-remote-asset:20200903T103837Z-90136c4 command: /config/asset.jsonnet restart: unless-stopped ports: - "7981:7981" volumes: - type: volume source: assets target: /storage - type: bind source: ./buildbarn-config/ target: /config bb-storage: image: buildbarn/bb-storage:20200816T115912Z-d83e1f0 command: /config/storage.jsonnet restart: unless-stopped ports: - "7982:7982" volumes: - type: volume source: cas target: /cas - type: bind source: ./buildbarn-config/ target: /config volumes: assets: cas:
Visit the bb-storage and bb-remote-asset project pages to find more documentation about setting up services with authentication enabled.