User configuration

User configuration and preferences can be specified in a user provided configuration file, and in most cases these preferences can be overridden using command line options.

Values that are not specified in a user configuration file assume their default values.

Configuration file

Unless a configuration file is explicitly specified on the command line when invoking bst, an attempt is made to load user specific configuration from $XDG_CONFIG_HOME/buildstream.conf. On most Linux based systems, the location will be ~/.config/buildstream.conf

Note

If you have have multiple major versions of BuildStream installed, you can have separate configuration files in your ${XDG_CONFIG_HOME}.

You can do this by naming them according to the major versions of BuildStream you have installed. BuildStream 1 will load its configuration from $XDG_CONFIG_HOME/buildstream1.conf and BuildStream 2 will load its configuration from $XDG_CONFIG_HOME/buildstream2.conf, while any version will fallback to $XDG_CONFIG_HOME/buildstream.conf.

Working directories

The working directories control where BuildStream will store data. While these will have sane defaults, you may want to change these directories depending on your partitioning scheme and where you may have extra space.

Environment variables and ~/ home directory expressions are supported when specifying working directories.

Working directories are configured at the toplevel of your configuration file, like so:

#
# Configure working directories
#
sourcedir: ~/buildstream/sources

Attributes

  • sourcedir

    This is the location where BuildStream stores the source code it downloads for builds.

  • logdir

    This is the location where BuildStream stores log files of build command output and basically logs pertaining to any activity BuildStream orchestrates.

  • cachedir

    This is the location where BuildStream stores the local CAS (Content Addressable Storage).

    The CAS is used to cache anything and everything which BuildStream may reuse at a later time.

    Attention

    While it may be beneficial at times to delete the entire cache directory manually in order to free up disk space, one should keep in mind that the sourcedir and logdir are configured as subdirectories of this directory when default configuration is used.

    Take care not to accidentally remove all your cached downloaded sources when deleting your cache.

  • workspacedir

    A default location for opening workspaces.

    Note

    By default this is configured to ., which is to say workspaces are created as a subdirectory of the current working directory by default.

    Because of this, the workspacedir directory is the only directory which is allowed to be declared as a relative path.

Local cache control

Beyond deciding what directory you intend to place the cache, there are some controls on what is cached locally and how.

These are controlled by the attributes of a cache dictionary at the toplevel of your configuration file, like so:

#
# Control the cache
#
cache:

  # Allow using as much free space as possible
  quota: infinity

  # Avoid pulling large amounts of data we don't need locally
  pull-buildtrees: False

  #
  # Avoid caching build trees if we don't need them
  cache-buildtrees: auto

Attributes

  • quota

    This controls how much data you allow BuildStream to cache locally.

    An attempt will be made to error out instead of exceeding the maximum quota which the user has allowed here. Given that it is impossible for BuildStream to know how much data a given build will create, this quota is implemented on a best effort basis.

    The quota can be specified in multiple ways:

    • The special infinity value

      This default value states that BuildStream can use as much space as is available on the filesystem where the cache resides.

    • A number in bytes.

    • A human readable number, suffixed in K, M, G or T

      E.g. 250K being 250 kilobytes, 100M being 100 megabytes, etc.

    • A percentage value, e.g. 80%

      Percentage values are taken to represent a percentage of the partition size on the filesystem where the cache has been configured.

  • pull-buildtrees

    Whether to pull build trees when downloading remote artifacts.

    The build tree of an artifact is the directory where a build took place, this is useful for running a build shell in order to observe how an element was built or to debug how a build failed if the build failed remotely.

    Since build trees are rather expensive, the default is to not pull build trees for every artifact. If you need a build tree that exists remotely, it will be possible to download it as an option at the time you run a command which requires it.

  • cache-buildtrees

    Whether to cache build trees when creating artifacts, if build trees are cached locally and the client is configured to push to remote servers, then build trees will be pushed along with any uploaded artifacts.

    This configuration has three possible values:

    • never: Never cache build trees

    • auto: Only cache the build trees where necessary (e.g. for failed builds)

    • always: Always cache the build tree.

Scheduler controls

Controls related to how the scheduler works are exposed as attributes of the toplevel scheduler dictionary, like so:

#
# Control the scheduler
#
scheduler:

  # Allow building up to four seperate elements at a time
  builders: 4

  # Continue building as many elements as possible if anything fails
  on-error: continue

Attributes

  • fetchers

    The number of concurrent tasks which download sources or artifacts.

  • pushers

    The number of concurrent tasks which upload sources or artifacts.

  • builders

    The number of concurrent tasks which build elements.

    Note

    This does not control the number of processes in the scope of the build of a single element, but rather the number of elements which may be built in parallel.

  • network-retries

    The number of times to retry a task which failed due to network connectivity issues.

  • on-error

    What to do when a task fails and BuildStream is running in non-interactive mode. This can be set to the following values:

    • continue: Continue with other tasks, a summary of errors will be printed at the end

    • quit: Quit after all ongoing tasks have completed

    • terminate: Abort any ongoing tasks and exit immediately

    Note

    If BuildStream is running in interactive mode, then the ongoing build will be suspended and the user will be prompted and asked what to do when a task fails.

    Interactive mode is automatically enabled if BuildStream is connected to a terminal rather than being run automatically, or, it can be specified on the command line.

Build controls

Some aspects about how elements get built can be controlled by attributes of the build dictionary at the toplevel, like so:

#
# Build controls
#
build:

  #
  # Allow up to 4 parallel processes to execute within the scope of one build
  #
  max-jobs: 4

Attributes

  • max-jobs

    This is a best effort attempt to instruct build systems on how many parallel processes to use when building an element.

    It is supported by most popular build systems such as make, cmake, ninja, etc, via environment variables such as MAXJOBS and similar command line options.

    When using the special value 0, BuildStream will allocate the number of threads available on the host and limit this with a hard coded value of 8, which was found to be an optimial number when building even on hosts with many cores.

  • dependencies

    This instructs what dependencies of the target elements should be built, valid values for this attribute are:

    • plan: Only build elements required to generate the expected target artifacts

    • all: Build elements even if they are build dependencies of artifacts which are already cached

Logging controls

Various aspects of how BuildStream presents output and UI can be controlled with attributes of the toplevel logging dictionary, like so:

#
# Control logging output
#
logging:

  #
  # Whether to be verbose
  #
  verbose: True

Attributes

  • verbose

    Whether to use verbose logging.

  • debug

    Whether to print messages related to debugging BuildStream itself.

  • key-length

    When displaying abbreviated cache keys, this controls the number of characters of the cache key which should be printed.

  • throttle-ui-updates

    Whether the throttle updates to the status bar in interactive mode. If set to True, then the status bar will be updated once per second.

  • error-lines

    The maximum number of lines to print in the main logging output related to an error processing an element, these will be the last lines found in the relevant element’s stdout and stderr.

  • message-lines

    The maximum number of lines to print in a detailed message sent to the main logging output.

  • element-format

    The default format to use when printing out elements in bst show output, and when printing the pipeline summary at the beginning of sessions.

    The format is specified as a string containing variables which will be expanded in the resulting string, variables must be specified using a leading percent sign and enclosed in curly braces, a colon can be specified in the variable to perform python style string alignments, e.g.:

    logging:
    
      #
      # Set the element format
      #
      element-format: |
    
        %{state: >12} %{full-key} %{name} %{workspace-dirs}
    

    Variable names which can be used in the element format consist of:

    • name

      The element path, which is the name of the element including any leading junctions.

    • key

      The abbreviated cache key, the length of which is controlled by the key-length logging configuration.

    • full-key

      The full cache key.

    • state

      The element state, this will be formatted as one of the following:

      • no reference: If the element still needs to be tracked.

      • junction: If the element is a junction and as such does not have any relevant state.

      • failed: If the element has been built and the build has failed.

      • cached: If the element has been successfully built and is present in the local cache.

      • fetch needed: If the element cannot be built yet because the sources need to be fetched.

      • buildable: If the element has all of its sources and build dependency artifacts cached locally.

      • waiting: If the element has all of its sources cached but its build dependencies are not yet locally cached.

    • config

      The element configuration, formatted in YAML.

    • vars

      The resolved element variables, formatted as a simple YAML dictionary.

    • env

      The resolved environment variables, formatted as a simple YAML dictionary.

    • public

      The resolved public data, formatted in YAML.

    • workspaced

      If the element has an open workspace, this will expand to the string “(workspaced)”, otherwise it will expand to an empty string.

    • workspace-dirs

      If the element has an open workspace, this will expand to the workspace directory, prefixed with the text “Workspace: “, otherwise it will expand to an empty string.

    • deps

      A list of the element paths of all dependency elements.

    • build-deps

      A list of the element paths of all build dependency elements.

    • runtime-deps

      A list of the element paths of all runtime dependency elements.

  • message-format

    The format to use for messages being logged in the aggregated main logging output.

    Similarly to the element-format, The format is specified as a string containing variables which will be expanded in the resulting string, and variables must be specified using a leading percent sign and enclosed in curly braces, e.g.:

    logging:
    
      #
      # Set the message format
      #
      message-format: |
    
        [%{elapsed}][%{key}][%{element}] %{action} %{message}
    

    Variable names which can be used in the element format consist of:

    • elapsed

      If this message announces the completion of (successful or otherwise) of an activity, then this will expand to a time code showing how much time elapsed for the given activity, in the format: HH:MM:SS, otherwise an empty time code will be displayed in the format: --:--:--.

    • elapsed-us

      Similar to the elapsed variable, however the timecode will include microsecond precision.

    • wallclock

      This will display a timecode for each message displaying the local wallclock time, in the format HH:MM:SS.

    • wallclock-us

      Similar to the wallclock variable, however the timecode will include microsecond precision.

    • key

      The abbreviated cache key of the element the message is related to, the length of which is controlled by the key-length logging configuration.

      If the message in question is not related to any element, then this will expand to whitespace of equal length.

    • element

      This will be formatted to an indicator consisting of the type of activity which is being performed on the element (e.g. “build” or “fetch” etc), and the element path of the element this message is related to.

      If the message in question is not related to any element, then a string will be formatted to indicate that this message is related to a core activity instead.

    • action

      A classifier of the type of message this is, the possible values this will expand to are:

      • DEBUG

        This is a message related to debugging BuildStream itself

      • STATUS

        A message showing some detail of what is currently happening, this message will not be displayed unless verbose output is enabled.

      • INFO

        An informative message, this may be printed for instance when discovering a new ref for source code when running bst source track.

      • WARN

        A warning message.

      • ERROR

        An error message.

      • BUG

        A bug happened in BuildStream, this will usually be accompanied by a python stack trace.

      • START

        An activity related to an element started.

        Any START message will always be accompanied by a later SUCCESS, FAILURE or SKIPPED message.

      • SUCCESS

        An activity related to an element completed successfully.

      • FAILURE

        An activity related to an element failed.

      • SKIPPED

        After strating this activity, it was discovered that no work was needed and the activity was skipped.

    • message

      The brief message, or the path to the corresponding log file, will be printed here.

      When this is a scheduler related message about the commencement or completion of an element related activity, then the path to the corresponding log for that activity will be printed here.

      If it is a message issued for any other reason, then the message text will be formatted here.

    Note

    Messages issued by the core or by plugins are allowed to provide detailed accounts, these are the indented multiline messages which sometimes get displayed in the main aggregated logging output, and will be printed regardless of the logging message-format value.

Remote services

BuildStream can be configured to cooperate with remote caches and execution services.

Authentication

BuildStream supports end to end encryption when communicating with remote services.

All remote service configuration blocks come with an optional auth configuration block which allows one to specify the certificates and keys required for encrypted traffic.

See the server configuration documentation for details on how the keys can be generated and managed on the server side.

The auth configuration block looks like this:

auth:
  server-cert: server.crt
  client-cert: client.crt
  client-key: client.key

Attributes

  • server-cert

    The server certificate is used to decrypt traffic coming from the server.

  • client-cert

    The client certificate is used by the remote server to decrypt traffic being uploaded to the server.

    The remote server will have its own copy of this certificate, but the client needs to send this certificate’s identity to the server so that the server knows which certificate to use.

  • client-key

    The client key is used to encrypt traffic when uploading traffic to the server.

Normally, only the server-cert is required to securely download data from remote cache services, while both the client-key and client-cert is required to securely upload data to the server.

Cache servers

BuildStream supports two types of cache servers, source cache servers and artifact cache servers. These services allow you to store sources and build artifacts for later reuse, and share them among your peers.

Important

Storing and indexing

Cache servers are split into two separate services, the index and the storage. Sometimes these services are provided by the same server, and sometimes it is desirable to use different cache servers for indexing and storing data.

In simple setups, it is possible to use the same cache server for indexing and storing of both sources and artifacts. However, when using remote execution it is recommended to use the remote execution build cluster’s storage-service as the storage service of your cache servers, which may require setting up your index service separately.

When configuring cache servers, BuildStream will require both storage and indexing capabilities, otherwise no attempt will be made to fetch or push data to and from cache servers.

Cache server configuration is declared in the following way:

override-project-caches: false
servers:
- url: https://cache-server.com/cache:11001
  instance-name: main
  type: all
  push: true
  auth:
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key

Attributes

  • override-project-caches

    Whether this user configuration overrides the project recommendations for artifact caches or source caches.

    If this is false (which is the default), then project recommended cache servers will be observed after user specified caches.

  • servers

    This is the list of cache servers in the configuration block, every entry in the block represents a server which will be accessed in the specified order.

    • url

      Indicates the http or https url and optionally the port number of where the cache server is located.

    • instance-name

      Instance names separate different shards on the same endpoint (url).

      The instance name is optional, and not all cache server implementations support instance names. The instance name should be given to you by the service provider of each service.

    • type

      The type of service you intend to use this cache server for. If unspecified, the default value for this field is all.

      • storage

        Use this cache service for storage.

      • index

        Use this cache service for index content expected to be present in one or more storage services.

      • all

        Use this cache service for both indexing and storing data.

    • push

      Set this to true if you intend to upload data to this cache server.

      Normally this requires additional credentials in the auth field.

    • auth

      The authentication attributes to connect to this server.

Cache server lists

Cache servers are always specified as lists in the configuration, this allows index and storage services to be declared separately, and also allows for some redundancy.

Example:

- url: https://cache-server-1.com/index
  type: index
- url: https://cache-server-1.com/storage
  type: storage
- url: https://cache-server-2.com
  type: all

When downloading data from a cache server, BuildStream will iterate over each index service one by one until it finds the reference to the data it is looking for, and then it will iterate over each storage service one by one, downloading the referenced data until all data is downloaded.

When uploading data to a cache server, BuildStream will first upload the data to each storage service which was configured with the push attribute, and upon successful upload, it will proceed to upload the references to the uploaded data to each index service in the list.

Artifact cache servers

Using artifact cache servers is an essential means of build avoidance, as it will allow you to avoid building an element which has already been built and uploaded to a common artifact server.

Artifact cache servers can be declared in different ways, with differing priorities.

Command line

Various commands which involve connecting to artifact servers allow specifying remotes, remotes specified on the command line replace all user configuration.

Global caches

To declare the global artifact server list, use the artifacts key at the toplevel of the user configuration.

#
# Configure a global artifact server for pushing and pulling artifacts
#
artifacts:
  override-project-caches: false
  servers:
  - url: https://artifacts.com/artifacts:11001
    push: true
    auth:
      server-cert: server.crt
      client-cert: client.crt
      client-key: client.key

Project overrides

To declare artifact servers lists for individual projects, declare them in the project specific section of the user configuration.

Artifact server lists declared in this section will only be used for elements belonging to the specified project, and will be used instead of artifact cache servers declared in the global caches.

#
# Configure an artifact server for pushing and pulling artifacts from project "foo"
#
projects:
  foo:
    artifacts:
      override-project-caches: false
      servers:
      - url: https://artifacts.com/artifacts:11001
        push: true
        auth:
          server-cert: server.crt
          client-cert: client.crt
          client-key: client.key

Project recommendations

Projects can recommend artifact cache servers in their individual project configuration files.

These will only be used for elements belonging to their respective projects, and are the lowest priority configuration.

Source cache servers

Using source cache servers enables BuildStream to cache source code referred to by your project and share those sources with peers who have access to the same source cache server.

This can optimize your build times in the case that it is determined that an element needs to be rebuilt because of changes in the dependency graph, as BuildStream will first attempt to download the source code from the cache server before attempting to obtain it from an external source, which may suffer higher latencies.

Source cache servers can be declared in different ways, with differing priorities.

Command line

Various commands which involve connecting to source cache servers allow specifying remotes, remotes specified on the command line replace all user configuration.

Global caches

To declare the global source cache server list, use the source-caches key at the toplevel of the user configuration.

#
# Configure a global source cache server for pushing and pulling sources
#
source-caches:
  override-project-caches: false
  servers:
  - url: https://sources.com/sources:11001
    push: true
    auth:
      server-cert: server.crt
      client-cert: client.crt
      client-key: client.key

Project overrides

To declare source cache servers lists for individual projects, declare them in the project specific section of the user configuration.

Source cache server lists declared in this section will only be used for elements belonging to the specified project, and will be used instead of source cache servers declared in the global caches.

#
# Configure a source cache server for pushing and pulling sources from project "foo"
#
projects:
  foo:
    source-caches:
      override-project-caches: false
      servers:
      - url: https://sources.com/sources:11001
        push: true
        auth:
          server-cert: server.crt
          client-cert: client.crt
          client-key: client.key

Project recommendations

Projects can recommend source cache servers in their individual project configuration files.

These will only be used for elements belonging to their respective projects, and are the lowest priority configuration.

Remote execution

BuildStream supports building remotely using the Google Remote Execution API (REAPI)..

You can configure the remote execution services globally in your user configuration using the remote-execution key, like so:

remote-execution:
  pull-artifact-files: True
  execution-service:
    url: http://execution.fallback.example.com:50051
    instance-name: main
  storage-service:
    url: https://storage.fallback.example.com:11002
    instance-name: main
    auth:
      server-cert: /keys/server.crt
      client-cert: /keys/client.crt
      client-key: /keys/client.key
  action-cache-service:
    url: http://cache.flalback.example.com:50052
    instance-name: main

Attributes

  • pull-artifact-files

    This determines whether you want the artifacts which were built remotely to be downloaded into the local CAS, so that it is ready for checkout directly after a built completes.

    If this is set to false, then you will need to download the artifacts you intend to use with bst artifact checkout after your build completes.

  • execution-service

    A service configuration specifying how to connect with the main execution service, this service is the main controlling entity in a remote execution build cluster.

  • storage-service

    A service configuration specifying how to connect with the Content Addressable Storage service, this is where build input and output is stored on the remote execution build cluster.

    This service is compatible with the storage service offered by cache servers.

  • action-cache-service

    A service configuration specifying how to connect with the action cache, this service stores information about activities which clients request be performed by workers on the remote execution build cluster, and results of completed operations.

    This service is optional in a remote execution build cluster, if your remote execution service provides an action cache, then you should configure it here.

Remote execution service configuration

Each of the distinct services are described by the same configuration block, which looks like this:

url: https://storage.fallback.example.com:11002
instance-name: main
auth:
  server-cert: /keys/server.crt
  client-cert: /keys/client.crt
  client-key: /keys/client.key

Attributes:

  • url

    Indicates the http or https url and optionally the port number of where the service is located.

  • instance-name

    The instance name is optional. Instance names separate different shards on the same endpoint (url). The instance name should be given to you by the service provider of each service.

    Not all service providers support instance names.

  • auth

    The authentication attributes to connect to this server.

Project specific values

The projects key can be used to specify project specific configurations, the supported configurations on a project wide basis are listed here.

Strict build plan

The strict build plan option decides whether you want elements to rebuild when their dependencies have changed. This is enabled by default, but recommended to turn off in developer scenarios where you might want to build a large system and test it quickly after modifying some low level component.

Example

projects:
  project-name:
    strict: False

Note

It is always possible to override this at invocation time using the --strict and --no-strict command line options.

Default mirror

When using mirrors, a default mirror can be defined to be fetched first. The default mirror is defined by its name, e.g.

projects:
  project-name:
    default-mirror: oz

Note

It is possible to override this at invocation time using the --default-mirror command-line option.

Project options

One can specify values to use for project options for the projects you use here, this avoids needing to specify the options on the command line every time.

Example

projects:

  #
  # Configure the debug flag offered by `project-name`
  #
  project-name:
    options:
      debug-build: True

Source cache servers

As already described in the section concerning configuration of source cache servers, these can be specified on a per project basis.

Artifact cache servers

As already described in the section concerning configuration of artifact cache servers, these can be specified on a per project basis.

Remote execution configuration

Following the same format as the toplevel remote execution configuration, the global configuration can be overridden on a per project basis in this project override section.

Example

projects:

  project-name:

    #
    # If `project-name` is built as the toplevel project in this BuildStream session,
    # then use this remote execution configuration instead of any globally defined
    # remote execution configuration.
    #
    remote-execution:
      pull-artifact-files: True
      execution-service:
        url: http://execution.example.com:50051
        instance-name: main

Note

Only one remote execution service will be considered for any invocation of BuildStream.

If you are building a project which has a junction into another subproject for which you have specified a project specific remote execution service for in your user configuration, then it will be ignored in the context of building that toplevel project.

Default configuration

The default BuildStream configuration is specified here for reference:

# Default BuildStream user configuration.

#
#    Work Directories
#

# Location to store sources
sourcedir: ${XDG_CACHE_HOME}/buildstream/sources

# Root location for other directories in the cache
cachedir: ${XDG_CACHE_HOME}/buildstream

# Location to store build logs
logdir: ${XDG_CACHE_HOME}/buildstream/logs

# Default root location for workspaces, blank for no default set.
workspacedir: .

#
#    Cache
#
cache:

  # Use as much space as possible
  quota: infinity

  # Whether to pull build trees when downloading element artifacts
  pull-buildtrees: False

  # Whether to cache build trees on artifact creation
  #
  cache-buildtrees: auto


#
#    Scheduler
#
scheduler:

  # Maximum number of simultaneous downloading tasks.
  fetchers: 10

  # Maximum number of simultaneous build tasks.
  builders: 4

  # Maximum number of simultaneous uploading tasks.
  pushers: 4

  # Maximum number of retries for network tasks.
  network-retries: 2

  # Control what to do when a task fails, if not running in
  # interactive mode
  #
  on-error: quit


#
# Build related configuration
#
build:

  #
  # Maximum number of jobs to run per build task.
  #
  max-jobs: 0

  #
  # Control which dependencies to build
  #
  dependencies: plan


#
#    Logging
#
logging:

  # The abbreviated cache key length to display in the UI
  key-length: 8

  # Whether to show extra detailed messages
  verbose: True

  # Maximum number of lines to print from the
  # end of a failing build log
  error-lines: 20

  # Maximum number of lines to print in a detailed
  # message on the console or in the master log (the full
  # messages are always recorded in the individual build
  # logs)
  message-lines: 20

  # Whether to enable debugging messages
  debug: False

  # Format string for printing the pipeline at startup, this
  # also determines the default display format for `bst show`
  #
  element-format: |

    %{state: >12} %{full-key} %{name} %{workspace-dirs}

  # Format string for log messages.
  #
  message-format: |

    [%{elapsed}][%{key}][%{element}] %{action} %{message}

  # Limit bst console output update rate to 1Hz where applicable
  throttle-ui-updates: True