Configuring Cache Servers

BuildStream caches the results of builds in a local artifact cache, and will avoid building an element if there is a suitable build already present in the local artifact cache. Similarly it will cache sources and avoid pulling them if present in the local cache. See caches for more details.

In addition to the local caches, you can configure one or more remote caches and BuildStream will then try to pull a suitable object from one of the remotes, falling back to performing a local build or fetching a source if needed.

On the client side, cache servers are declared and configured in user configuration, and since it is typical for projects to maintain their own cache servers, it is also possible for projects to provide recommended artifact cache servers and source cache servers through project configuration, so that downstream users can download from services provided by upstream projects by default.

Setting up a remote cache

BuildStream relies on the ContentAddressableStorage protocol in order to exchange data with remote services, in concert with the remote asset protocol in order to assign symbolic labels (such as artifact names) to identify stored content. As such, BuildStream is able to function with any implementations of these two services.

Known implementations

Here are some details about known open source implementations of the required protocols

Buildbarn

The Buildbarn project provides a remote execution service implementation for use in build tooling such as BuildStream, Bazel and recc, the bb-storage and bb-remote-asset services are tested to work as cache service for BuildStream’s artifact and source caches.

A simple configuration to spin up the service using docker compose follows:

##
# Buildbarn Compose manifest for BuildStream.
#
# Spins-up a unnamed and unauthenticated cache server:
#  - STORAGE at http://localhost:7982
#  - INDEX at: http://localhost:7981
#
# BuildStream configuration snippet:
#
# artifacts:
#   - url: https://localhost:7981
#     type: index
#     push: true
#   - url: https://localhost:7982
#     type: storage
#     push: true
#
# Basic usage:
#  - docker-compose -f ci.buildbarn.yml up
#  - docker-compose -f ci.buildbarn.yml down

version: '3.4'

services:
  bb-asset:
    image: buildbarn/bb-remote-asset:20200903T103837Z-90136c4
    command: /config/asset.jsonnet
    restart: unless-stopped
    ports:
    - "7981:7981"
    volumes:
    - type: volume
      source: assets
      target: /storage
    - type: bind
      source: ./buildbarn-config/
      target: /config

  bb-storage:
    image: buildbarn/bb-storage:20200816T115912Z-d83e1f0
    command: /config/storage.jsonnet
    restart: unless-stopped
    ports:
    - "7982:7982"
    volumes:
    - type: volume
      source: cas
      target: /cas
    - type: bind
      source: ./buildbarn-config/
      target: /config

volumes:
  assets:
  cas:

Visit the bb-storage and bb-remote-asset project pages to find more documentation about setting up services with authentication enabled.