Project configuration

The project configuration file should be named project.conf and be located at the project root. It holds information such as Source aliases relevant for the sources used in the given project as well as overrides for the configuration of element types used in the project.

Values specified in the project configuration override any of the default BuildStream project configuration, which is included here for reference.

Essentials

Project name

The project name is a unique symbol for your project and will be used to distinguish your project from others in user preferences, namspaceing of your project’s artifacts in shared artifact caches, and in any case where BuildStream needs to distinguish between multiple projects.

The first thing to setup in your project.conf should be the name of your project.

name: my-project-name

Note

The project name may contain alphanumeric characters, dashes and underscores, and may not start with a leading digit.

Format version

The BuildStream format is guaranteed to be backwards compatible with any earlier releases. The project’s minimum required format version of BuildStream can be specified in project.conf with the format-version field, e.g.:

# The minimum base BuildStream format
format-version: 0

BuildStream will increment its core YAML format version at least once in any given minor point release where the format has been extended to support a new feature.

Note

External Element and Source plugins also implement their own YAML configuration fragments and as such are revisioned separately from the core format. See External plugins for details on specifying a minimum version of a specific plugin.

Core Elements and Sources which are maintained and distributed as a part of BuildStream are revisioned under the same global format-version described here.

Element path

To allow the user to structure their project nicely, BuildStream allows the user to specify a project subdirectory where element .bst files are stored.

element-path: elements

Note that elements are referred to by their relative paths, whenever elements are referred to in a .bst file or on the command line.

Ref storage

By default, BuildStream expects to read and write source references directly in the source declaration, but this can be inconvenient and prohibitive in some workflows.

Alternatively, BuildStream allows source references to be stored centrally in a project.refs file in the toplevel project directory.

This can be controlled with the ref-storage option, which is allowed to be configured with the following values:

  • inline

    Source references are stored directly in the source declaration

  • project.refs

    Source references are stored in the project.refs file, and junction source references are stored in the junction.refs file.

To enable storing of source references in project.refs, add the following to your project.conf:

ref-storage: project.refs

Note

The ref-storage configuration is available since format version 8

Configurable Warnings

Warnings can be configured as fatal using the fatal-warnings configuration item. When a warning is configured as fatal, where a warning would usually be thrown instead an error will be thrown causing the build to fail.

Individual warnings can be configured as fatal by setting fatal-warnings to a list of warnings.

fatal-warnings:
- overlaps
- ref-not-in-track
- <plugin>:<warning>

BuildStream provides a collection of Core Warnings which may be raised by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation.

Note

The fatal-warnings configuration is available since format version 14

Source aliases

In order to abstract the download location of source code and any assets which need to be downloaded, and also as a matter of convenience, BuildStream allows one to create named aliases for URLs which are to be used in the individual .bst files.

aliases:
  foo: git://git.foo.org/
  bar: http://bar.com/downloads/

Sandbox options

Sandbox options for the whole project can be supplied in project.conf in the same way as in an element. See element configuration for more detail.

# Specify a user id and group id to use in the build sandbox.
sandbox:
  build-uid: 1003
  build-gid: 1001

Note

The sandbox configuration is available since format version 6

Artifact server

If you have setup an artifact server for your project then it is convenient to configure the following in your project.conf so that users need not have any additional configuration to communicate with an artifact share.

#
# Artifacts
#
artifacts:
  # A remote cache from which to download prebuilt artifacts
  - url: https://foo.com:11001
    server.cert: server.crt
  # A remote cache from which to upload/download built/prebuilt artifacts
  - url: https://foo.com:11002
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key

Note

You can also specify a list of different caches here; earlier entries in the list will have higher priority than later ones.

The use of ports are required to distinguish between pull only access and push/pull access. For information regarding the server/client certificates and keys, please see: Key pair for the server.

Source cache server

Exactly the same as artifact servers, source cache servers can be specified.

#
# Source caches
#
source-caches:
  # A remote cache from which to download prestaged sources
  - url: https://foo.com:11001
    server.cert: server.crt
  # A remote cache from which to upload/download prestaged sources
  - url: https://foo.com:11002
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key

Note

As artifact caches work in exactly the same way, a configured artifact server can also be used as a source cache server. If you want to use a server as both you can put it under both artifacts and source caches configs.

Remote execution

BuildStream supports remote execution using the Google Remote Execution API (REAPI). A description of how remote execution works is beyond the scope of this document, but you can specify a remote server complying with the REAPI using the remote-execution option:

remote-execution:

  # A url defining a remote execution server
  execution-service:
    url: http://buildserver.example.com:50051
    instance-name: development-emea-1
  storage-service:
    url: https://foo.com:11002/
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key
    instance-name: development-emea-1
  action-cache-service:
    url: http://bar.action.com:50052
    instance-name: development-emea-1

storage-service specifies a remote CAS store and the parameters are the same as those used to specify an artifact server.

The action-cache-service specifies where built actions are cached, allowing buildstream to check whether an action has already been executed and download it if so. This is similar to the artifact cache but REAPI specified, and is optional for remote execution to work.

The storage service may be the same endpoint used for artifact caching. Remote execution cannot work without push access to the storage endpoint though.

Instance name is optional. Instance names separate different shards on the same endpoint (url). You can supply a different instance name for execution-service and storage-service, if needed. The instance name should be given to you by the service provider of each service. Not all remote execution and storage services support instance names.

The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.

Remote execution configuration can be also provided in the user configuration <user_config_remote_execution>.

Mirrors

A list of mirrors can be defined that couple a location to a mapping of aliases to a list of URIs, e.g.

mirrors:
- name: middle-earth
  aliases:
    foo:
    - http://www.middle-earth.com/foo/1
    - http://www.middle-earth.com/foo/2
    bar:
    - http://www.middle-earth.com/bar/1
    - http://www.middle-earth.com/bar/2
- name: oz
  aliases:
    foo:
    - http://www.oz.com/foo
    bar:
    - http://www.oz.com/bar

The order that the mirrors (and the URIs therein) are consulted is in the order they are defined when fetching, and in reverse-order when tracking.

A default mirror to consult first can be defined via user config, or the command-line argument –default-mirror.

Note

The mirrors field is available since format version 11

External plugins

If your project makes use of any custom Element or Source plugins, then the project must inform BuildStream of the plugins it means to make use of and the origin from which they can be loaded.

Note that plugins with the same name from different origins are not permitted.

Local plugins

Local plugins are expected to be found in a subdirectory of the actual BuildStream project. Element and Source plugins should be stored in separate directories to avoid namespace collisions.

The versions of local plugins are largely immaterial since they are revisioned along with the project by the user, usually in a VCS like git. However, for the sake of consistency with other plugin loading origins we require that you specify a version, this can always be 0 for a local plugin.

plugins:

- origin: local
  path: plugins/sources

  # We want to use the `mysource` source plugin located in our
  # project's `plugins/sources` subdirectory.
  sources:
    mysource: 0

Pip plugins

Plugins loaded from the pip origin are expected to be installed separately on the host operating system using python’s package management system.

plugins:

- origin: pip

  # Specify the name of the python package containing
  # the plugins we want to load. The name one would use
  # on the `pip install` command line.
  #
  package-name: potato

  # We again must specify a minimal format version for the
  # external plugin, it is allowed to be `0`.
  #
  elements:
    potato: 0

Options

Options are how BuildStream projects can define parameters which can be configured by users invoking BuildStream to build your project.

Options are declared in the project.conf in the main options dictionary.

options:
  debug:
    type: bool
    description: Whether to enable debugging
    default: False

Users can configure those options when invoking BuildStream with the --option argument:

$ bst --option debug True ...

Note

The name of the option may contain alphanumeric characters underscores, and may not start with a leading digit.

Common properties

All option types accept the following common attributes

  • type

    Indicates the type of option to declare

  • description

    A description of the meaning of the option

  • variable

    Optionally indicate a variable name to export the option to. A string form of the selected option will be used to set the exported value.

    If used, this value will override any existing value for the variable declared in project.conf, and will be overridden in the regular composition order.

    Note

    The name of the variable to export may contain alphanumeric characters, dashes, underscores, and may not start with a leading digit.

Boolean

The bool option type allows specifying boolean values which can be cased in conditional expressions.

Declaring

options:
  debug:
    type: bool
    description: Whether to enable debugging
    default: False

Evaluating

Boolean options can be tested in expressions with equality tests:

variables:
  enable-debug: False
  (?):
  - debug == True:
      enable-debug: True

Or simply treated as truthy values:

variables:
  enable-debug: False
  (?):
  - debug:
      enable-debug: True

Exporting

When exporting boolean options as variables, a True option value will be exported as 1 and a False option as 0

Enumeration

The enum option type allows specifying a string value with a restricted set of possible values.

Declaring

options:
  loglevel:
    type: enum
    description: The logging level
    values:
    - debug
    - info
    - warning
    default: info

Evaluating

Enumeration options must be tested as strings in conditional expressions:

variables:
  enable-debug: False
  (?):
  - loglevel == "debug":
      enable-debug: True

Exporting

When exporting enumeration options as variables, the value is exported as a variable directly, as it is a simple string.

Flags

The flags option type allows specifying a list of string values with a restricted set of possible values.

In contrast with the enum option type, the default value need not be specified and will default to an empty set.

Declaring

options:
  logmask:
    type: flags
    description: The logging mask
    values:
    - debug
    - info
    - warning
    default:
    - info

Evaluating

Options of type flags can be tested in conditional expressions using a pythonic in syntax to test if an element is present in a set:

variables:
  enable-debug: False
  (?):
  - ("debug" in logmask):
      enable-debug: True

Exporting

When exporting flags options as variables, the value is exported as a comma separated list of selected value strings.

Architecture

The arch option type is a special enumeration option which defaults via uname -m results to the following list.

  • aarch32
  • aarch64
  • aarch64-be
  • power-isa-be
  • power-isa-le
  • sparc-v9
  • x86-32
  • x86-64

The reason for this, opposed to using just uname -m, is that we want an OS-independent list, as well as several results mapping to the same architecture (e.g. i386, i486 etc. are all x86-32). It does not support assigning any default in the project configuration.

options:
  machine_arch:
    type: arch
    description: The machine architecture
    values:
    - aarch32
    - aarch64
    - x86-32
    - x86-64

Architecture options can be tested with the same expressions as other Enumeration options.

OS

The os option type is a special enumeration option, which defaults to the results of uname -s. It does not support assigning any default in the project configuration.

options:
  machine_os:
    type: os
    description: The machine OS
    values:
    - Linux
    - SunOS
    - Darwin
    - FreeBSD

Os options can be tested with the same expressions as other Enumeration options.

Element mask

The element-mask option type is a special Flags option which automatically allows only element names as values.

options:
  debug_elements:
    type: element-mask
    description: The elements to build in debug mode

This can be convenient for automatically declaring an option which might apply to any element, and can be tested with the same syntax as other Flag options.

variables:
  enable-debug: False
  (?):
  - ("element.bst" in debug_elements):
      enable-debug: True

Element default configuration

The project.conf plays a role in defining elements by providing default values and also by overriding values declared by plugins on a plugin wide basis.

See the composition documentation for more detail on how elements are composed.

Variables

The defaults for Variables used in your project is defined here.

variables:
  prefix: "/usr"

Environment

The defaults environment for the build sandbox is defined here.

environment:
  PATH: /usr/bin:/bin:/usr/sbin:/sbin

Additionally, the special environment-nocache list which specifies which environment variables do not affect build output, and are thus not considered in the calculation of artifact keys can be defined here.

environment-nocache:
- MAXJOBS

Note that the environment-nocache list only exists so that we can control parameters such as make -j ${MAXJOBS}, allowing us to control the number of jobs for a given build without affecting the resulting cache key.

Split rules

The project wide split rules defaults can be specified here.

split-rules:
  devel:
  - |
    %{includedir}
  - |
    %{includedir}/**
  - |
    %{libdir}/lib*.a
  - |
    %{libdir}/lib*.la

Overriding plugin defaults

Base attributes declared by element and source plugins can be overridden on a project wide basis. This section explains how to make project wide statements which augment the configuration of an element or source plugin.

Element overrides

The elements dictionary can be used to override variables, environments or plugin specific configuration data as shown below.

elements:

  # Override default values for all autotools elements
  autotools:

    variables:
      bindir: "%{prefix}/bin"

    config:
      configure-commands: ...

    environment:
      PKG_CONFIG_PATH=%{libdir}/pkgconfig

Source overrides

The sources dictionary can be used to override source plugin specific configuration data as shown below.

sources:

  # Override default values for all git sources
  git:

    config:
      checkout-submodules: False

Note

The sources override is available since format version 1

Customizing the shell

Since BuildStream cannot know intimate details about your host or about the nature of the runtime and software that you are building, the shell environment for debugging and testing applications may need some help.

The shell section allows some customization of the shell environment.

Note

The shell section is available since format version 1

Interactive shell command

By default, BuildStream will use sh -i when running an interactive shell, unless a specific command is given to the bst shell command.

BuildStream will automatically set a convenient prompt via the PS1 environment variable for interactive shells; which might be overwritten depending on the shell you use in your runtime.

If you are using bash, we recommend the following configuration to ensure that the customized prompt is not overwritten:

shell:

  # Specify the command to run by default for interactive shells
  command: [ 'bash', '--noprofile', '--norc', '-i' ]

Environment assignments

In order to cooperate with your host environment, a debugging shell sometimes needs to be configured with some extra knowledge inheriting from your host environment.

This can be achieved by setting up the shell environment configuration, which is expressed as a dictionary very similar to the default environment, except that it supports host side environment variable expansion in values.

Note

The environment configuration is available since format version 4

For example, to share your host DISPLAY and DBUS_SESSION_BUS_ADDRESS environments with debugging shells for your project, specify the following:

shell:

  # Share some environment variables from the host environment
  environment:
    DISPLAY: '$DISPLAY'
    DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'

Or, a more complex example is how one might share the host pulseaudio server with a bst shell environment:

shell:

  # Set some environment variables explicitly
  environment:
    PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'

Host files

It can be useful to share some files on the host with a shell so that it can integrate better with the host environment.

The host-files configuration allows one to specify files and directories on the host to be bind mounted into the sandbox.

Note

The host-files configuration is available since format version 4

Warning

One should never mount directories where one expects to find data and files which belong to the user, such as /home on POSIX platforms.

This is because the unsuspecting user may corrupt their own files accidentally as a result. Instead users can use the --mount option of bst shell to mount data into the shell.

The host-files configuration is an ordered list of mount specifications.

Members of the list can be fully specified as a dictionary, or a simple string can be used if only the defaults are required.

The fully specified dictionary has the following members:

  • path

    The path inside the sandbox. This is the only mandatory member of the mount specification.

  • host_path

    The host path to mount at path in the sandbox. This will default to path if left unspecified.

  • optional

    Whether the mount should be considered optional. This is False by default.

Here is an example of a fully specified mount specification:

shell:

  # Mount an arbitrary resolv.conf from the host to
  # /etc/resolv.conf in the sandbox, and avoid any
  # warnings if the host resolv.conf doesnt exist.
  host-files:
  - host_path: '/usr/local/work/etc/resolv.conf'
    path: '/etc/resolv.conf'
    optional: True

Here is an example of using shorthand mount specifications:

shell:

  # Specify a list of files to mount in the sandbox
  # directory from the host.
  #
  # If these do not exist on the host, a warning will
  # be issued but the shell will still be launched.
  host-files:
  - '/etc/passwd'
  - '/etc/group'
  - '/etc/resolv.conf'

Host side environment variable expansion is also supported:

shell:

  # Mount a host side pulseaudio server socket into
  # the shell environment at the same location.
  host-files:
  - '${XDG_RUNTIME_DIR}/pulse/native'

Default targets

When running BuildStream commands from a project directory or subdirectory without specifying any target elements on the command line, the default targets of the project will be used. The default targets can be configured in the defaults section as follows:

defaults:

  # List of default target elements
  targets:
  - app.bst

If no default targets are configured in project.conf, BuildStream commands will default to all .bst files in the configured element path.

Commands that cannot support junctions as target elements (bst build, bst artifact push, and bst artifact pull) ignore junctions in the list of default targets.

When running BuildStream commands from a workspace directory (that is not a BuildStream project directory), project default targets are not used and the workspace element will be used as the default target instead.

bst artifact checkout, bst source checkout, and bst shell are currently limited to a single target element and due to this, they currently do not use project default targets. However, they still use the workspace element as default target when run from a workspace directory.

Note

The targets configuration is available since format version 21

Builtin defaults

BuildStream defines some default values for convenience, the default values overridden by your project’s project.conf are presented here:

# Default BuildStream project configuration.


# General configuration defaults
#

# Require format version 0
format-version: 0

# Elements are found at the project root
element-path: .

# Store source references in element files
ref-storage: inline

# Variable Configuration
#
variables:
  # Path configuration, to be used in build instructions.
  prefix: "/usr"
  exec_prefix: "%{prefix}"
  bindir: "%{exec_prefix}/bin"
  sbindir: "%{exec_prefix}/sbin"
  libexecdir: "%{exec_prefix}/libexec"
  datadir: "%{prefix}/share"
  sysconfdir: "/etc"
  sharedstatedir: "%{prefix}/com"
  localstatedir: "/var"
  lib: "lib"
  libdir: "%{prefix}/%{lib}"
  debugdir: "%{libdir}/debug"
  includedir: "%{prefix}/include"
  docdir: "%{datadir}/doc"
  infodir: "%{datadir}/info"
  mandir: "%{datadir}/man"

  # Indicates the default build directory where input is
  # normally staged
  build-root: /buildstream/%{project-name}/%{element-name}

  # Indicates where the build system should look for configuration files  
  conf-root: .
  
  # Indicates the build installation directory in the sandbox
  install-root: /buildstream-install

  # You need to override this with the commands specific for your system
  strip-binaries: ""

  # Generic implementation for reproducible python builds
  fix-pyc-timestamps: |

    find "%{install-root}" -name '*.pyc' -exec \
      dd if=/dev/zero of={} bs=1 count=4 seek=4 conv=notrunc ';'

# Base sandbox environment, can be overridden by plugins
environment:
  PATH: /usr/bin:/bin:/usr/sbin:/sbin
  SHELL: /bin/sh
  TERM: dumb
  USER: tomjon
  USERNAME: tomjon
  LOGNAME: tomjon
  LC_ALL: C
  HOME: /tmp
  TZ: UTC

  # For reproducible builds we use 2011-11-11 as a constant
  SOURCE_DATE_EPOCH: 1320937200

# List of environment variables which should not be taken into
# account when calculating a cache key for a given element.
#
environment-nocache: []

# Configuration for the sandbox other than environment variables
# should go in 'sandbox'. This just contains the UID and GID that
# the user in the sandbox will have. Not all sandboxes will support
# changing the values.
sandbox:
  build-uid: 0
  build-gid: 0

# Defaults for the 'split-rules' public data found on elements
# in the 'bst' domain.
#
split-rules:

  # The runtime domain includes whatever is needed for the
  # built element to run, this includes stripped executables
  # and shared libraries by default.
  runtime:
  - |
    %{bindir}
  - |
    %{bindir}/*
  - |
    %{sbindir}
  - |
    %{sbindir}/*
  - |
    %{libexecdir}
  - |
    %{libexecdir}/*
  - |
    %{libdir}/lib*.so*

  # The devel domain includes additional things which
  # you may need for development.
  #
  # By default this includes header files, static libraries
  # and other metadata such as pkgconfig files, m4 macros and
  # libtool archives.
  devel:
  - |
    %{includedir}
  - |
    %{includedir}/**
  - |
    %{libdir}/lib*.a
  - |
    %{libdir}/lib*.la
  - |
    %{libdir}/pkgconfig/*.pc
  - |
    %{datadir}/pkgconfig/*.pc
  - |
    %{datadir}/aclocal/*.m4

  # The debug domain includes debugging information stripped
  # away from libraries and executables
  debug:
  - |
    %{debugdir}
  - |
    %{debugdir}/**

  # The doc domain includes documentation
  doc:
  - |
    %{docdir}
  - |
    %{docdir}/**
  - |
    %{infodir}
  - |
    %{infodir}/**
  - |
    %{mandir}
  - |
    %{mandir}/**

  # The locale domain includes translations etc
  locale:
  - |
    %{datadir}/locale
  - |
    %{datadir}/locale/**
  - |
    %{datadir}/i18n
  - |
    %{datadir}/i18n/**
  - |
    %{datadir}/zoneinfo
  - |
    %{datadir}/zoneinfo/**


# Default behavior for `bst shell`
#
shell:

  # Command to run when `bst shell` does not provide a command
  #
  command: [ 'sh', '-i' ]

# Defaults for bst commands
#
defaults:

  # Set default target elements to use when none are passed on the command line.
  # If none are configured in the project, default to all project elements.
  targets: []