Project configuration

The project configuration file should be named project.conf and be located at the project root. It holds information such as Source aliases relevant for the sources used in the given project as well as overrides for the configuration of element types used in the project.

Values specified in the project configuration override any of the default BuildStream project configuration, which is included here for reference.

Essentials

Project name

The project name is a unique symbol for your project and will be used to distinguish your project from others in user preferences, namspaceing of your project’s artifacts in shared artifact caches, and in any case where BuildStream needs to distinguish between multiple projects.

The first thing to setup in your project.conf should be the name of your project.

name: my-project-name

Note

The project name may contain alphanumeric characters, dashes and underscores, and may not start with a leading digit.

Format version

The BuildStream format is guaranteed to be backwards compatible with any earlier releases. The project’s minimum required format version of BuildStream can be specified in project.conf with the format-version field, e.g.:

# The minimum base BuildStream format
format-version: 0

BuildStream will increment it’s core YAML format version at least once in any given minor point release where the format has been extended to support a new feature.

Note

External Element and Source plugins also implement their own YAML configuration fragments and as such are revisioned separately from the core format. See External plugins for details on specifying a minimum version of a specific plugin.

Core Elements and Sources which are maintained and distributed as a part of BuildStream are revisioned under the same global format-version described here.

Element path

To allow the user to structure their project nicely, BuildStream allows the user to specify a project subdirectory where element .bst files are stored.

element-path: elements

Note that elements are referred to by their relative paths, whenever elements are referred to in a .bst file or on the command line.

Ref storage

By default, BuildStream expects to read and write source references directly in the source declaration, but this can be inconvenient and prohibitive in some workflows.

Alternatively, BuildStream allows source references to be stored centrally in a project.refs file in the toplevel project directory.

This can be controlled with the ref-storage option, which is allowed to be configured with the following values:

• inline

  Source references are stored directly in the source declaration

• project.refs

  Source references are stored in the project.refs file, and junction source references are stored in the junction.refs file.

To enable storing of source references in project.refs, add the following to your project.conf:

ref-storage: project.refs

Note

The ref-storage configuration is available since format version 8

Fail on overlaps

When multiple elements are staged, there’s a possibility that different elements will try and stage different versions of the same file.

When fail-on-overlap is true, if an overlap is detected that hasn’t been allowed by the element’s overlap whitelist, then an error will be raised and the build will fail.

otherwise, a warning will be raised indicating which files had overlaps, and the order that the elements were overlapped.

fail-on-overlap: true

Source aliases

In order to abstract the download location of source code and any assets which need to be downloaded, and also as a matter of convenience, BuildStream allows one to create named aliases for URLs which are to be used in the individual .bst files.

aliases:
  foo: git://git.foo.org/
  bar: http://bar.com/downloads/

Sandbox options

Sandbox options for the whole project can be supplied in project.conf in the same way as in an element. See element configuration for more detail.

# Specify a user id and group id to use in the build sandbox.
sandbox:
  build-uid: 1003
  build-gid: 1001

Note

The sandbox configuration is available since format version 6

Artifact server

If you have setup an artifact server for your project then it is convenient to configure this in your project.conf so that users need not have any additional configuration to communicate with an artifact share.

artifacts:

  # A url from which to download prebuilt artifacts
  url: https://foo.com/artifacts

You can also specify a list of caches here; earlier entries in the list will have higher priority than later ones.

Mirrors

A list of mirrors can be defined that couple a location to a mapping of aliases to a list of URIs, e.g.

mirrors:
- name: middle-earth
  aliases:
    foo:
    - http://www.middle-earth.com/foo/1
    - http://www.middle-earth.com/foo/2
    bar:
    - http://www.middle-earth.com/bar/1
    - http://www.middle-earth.com/bar/2
- name: oz
  aliases:
    foo:
    - http://www.oz.com/foo
    bar:
    - http://www.oz.com/bar

The order that the mirrors (and the URIs therein) are consulted is in the order they are defined when fetching, and in reverse-order when tracking.

A default mirror to consult first can be defined via user config, or the command-line argument –default-mirror.

Note

The mirrors field is available since format version 11

External plugins

If your project makes use of any custom Element or Source plugins, then the project must inform BuildStream of the plugins it means to make use of and the origin from which they can be loaded.

Note that plugins with the same name from different origins are not permitted.

Local plugins

Local plugins are expected to be found in a subdirectory of the actual BuildStream project. Element and Source plugins should be stored in separate directories to avoid namespace collisions.

The versions of local plugins are largely immaterial since they are revisioned along with the project by the user, usually in a VCS like git. However, for the sake of consistency with other plugin loading origins we require that you specify a version, this can always be 0 for a local plugin.

plugins:

- origin: local
  path: plugins/sources

  # We want to use the `mysource` source plugin located in our
  # project's `plugins/sources` subdirectory.
  sources:
    mysource: 0

Pip plugins

Plugins loaded from the pip origin are expected to be installed separately on the host operating system using python’s package management system.

plugins:

- origin: pip

  # Specify the name of the python package containing
  # the plugins we want to load. The name one would use
  # on the `pip install` command line.
  #
  package-name: potato

  # We again must specify a minimal format version for the
  # external plugin, it is allowed to be `0`.
  #
  elements:
    potato: 0

Options

Options are how BuildStream projects can define parameters which can be configured by users invoking BuildStream to build your project.

Options are declared in the project.conf in the main options dictionary.

options:
  debug:
    type: bool
    description: Whether to enable debugging
    default: False

Users can configure those options when invoking BuildStream with the --option argument:

$ bst --option debug True ...

Note

The name of the option may contain alphanumeric characters underscores, and may not start with a leading digit.

Common properties

All option types accept the following common attributes

• type

  Indicates the type of option to declare

• description

  A description of the meaning of the option

• variable

  Optionally indicate a variable name to export the option to. A string form of the selected option will be used to set the exported value.

  If used, this value will override any existing value for the variable declared in project.conf, and will be overridden in the regular composition order.

  Note

  The name of the variable to export may contain alphanumeric characters, dashes, underscores, and may not start with a leading digit.

Boolean

The bool option type allows specifying boolean values which can be cased in conditional expressions.

Declaring

options:
  debug:
    type: bool
    description: Whether to enable debugging
    default: False

Evaluating

Boolean options can be tested in expressions with equality tests:

variables:
  enable-debug: False
  (?):
  - debug == True:
      enable-debug: True

Or simply treated as truthy values:

variables:
  enable-debug: False
  (?):
  - debug:
      enable-debug: True

Exporting

When exporting boolean options as variables, a True option value will be exported as 1 and a False option as 0

Enumeration

The enum option type allows specifying a string value with a restricted set of possible values.

Declaring

options:
  loglevel:
    type: enum
    description: The logging level
    values:
    - debug
    - info
    - warning
    default: info

Evaluating

Enumeration options must be tested as strings in conditional expressions:

variables:
  enable-debug: False
  (?):
  - loglevel == "debug":
      enable-debug: True

Exporting

When exporting enumeration options as variables, the value is exported as a variable directly, as it is a simple string.

Flags

The flags option type allows specifying a list of string values with a restricted set of possible values.

In contrast with the enum option type, the default value need not be specified and will default to an empty set.

Declaring

options:
  logmask:
    type: flags
    description: The logging mask
    values:
    - debug
    - info
    - warning
    default:
    - info

Evaluating

Options of type flags can be tested in conditional expressions using a pythonic in syntax to test if an element is present in a set:

variables:
  enable-debug: False
  (?):
  - ("debug" in logmask):
      enable-debug: True

Exporting

When exporting flags options as variables, the value is exported as a comma separated list of selected value strings.

Architecture

The arch option type is special enumeration option which defaults to the result of uname -m, and does not support assigning any default in the project configuration.

options:
  machine_arch:
    type: arch
    description: The machine architecture
    values:
    - arm
    - aarch64
    - i386
    - x86_64

Architecture options can be tested with the same expressions as other Enumeration options.

Element mask

The element-mask option type is a special Flags option which automatically allows only element names as values.

options:
  debug_elements:
    type: element-mask
    description: The elements to build in debug mode

This can be convenient for automatically declaring an option which might apply to any element, and can be tested with the same syntax as other Flag options.

variables:
  enable-debug: False
  (?):
  - ("element.bst" in debug_elements):
      enable-debug: True

Element default configuration

The project.conf plays a role in defining elements by providing default values and also by overriding values declared by plugins on a plugin wide basis.

See the composition documentation for more detail on how elements are composed.

Variables

The defaults for Variables used in your project is defined here.

variables:
  prefix: "/usr"

Environment

The defaults environment for the build sandbox is defined here.

environment:
  PATH: /usr/bin:/bin:/usr/sbin:/sbin

Additionally, the special environment-nocache list which specifies which environment variables do not effect build output, and are thus not considered in the calculation of artifact keys can be defined here.

environment-nocache:
- MAXJOBS

Note that the environment-nocache list only exists so that we can control parameters such as make -j ${MAXJOBS}, allowing us to control the number of jobs for a given build without effecting the resulting cache key.

Split rules

The project wide split rules defaults can be specified here.

split-rules:
  devel:
  - |
    %{includedir}
  - |
    %{includedir}/**
  - |
    %{libdir}/lib*.a
  - |
    %{libdir}/lib*.la

Overriding plugin defaults

Base attributes declared by element and source plugins can be overridden on a project wide basis. This section explains how to make project wide statements which augment the configuration of an element or source plugin.

Element overrides

The elements dictionary can be used to override variables, environments or plugin specific configuration data as shown below.

elements:

  # Override default values for all autotools elements
  autotools:

    variables:
      bindir: "%{prefix}/bin"

    config:
      configure-commands: ...

    environment:
      PKG_CONFIG_PATH=%{libdir}/pkgconfig

Source overrides

The sources dictionary can be used to override source plugin specific configuration data as shown below.

sources:

  # Override default values for all git sources
  git:

    config:
      checkout-submodules: False

Note

The sources override is available since format version 1

Customizing the shell

Since BuildStream cannot know intimate details about your host or about the nature of the runtime and software that you are building, the shell environment for debugging and testing applications may need some help.

The shell section allows some customization of the shell environment.

Note

The shell section is available since format version 1

Interactive shell command

By default, BuildStream will use sh -i when running an interactive shell, unless a specific command is given to the bst shell command.

BuildStream will automatically set a convenient prompt via the PS1 environment variable for interactive shells; which might be overwritten depending on the shell you use in your runtime.

If you are using bash, we recommend the following configuration to ensure that the customized prompt is not overwritten:

shell:

  # Specify the command to run by default for interactive shells
  command: [ 'bash', '--noprofile', '--norc', '-i' ]

Environment assignments

In order to cooperate with your host environment, a debugging shell sometimes needs to be configured with some extra knowledge inheriting from your host environment.

This can be achieved by setting up the shell environment configuration, which is expressed as a dictionary very similar to the default environment, except that it supports host side environment variable expansion in values.

Note

The environment configuration is available since format version 4

For example, to share your host DISPLAY and DBUS_SESSION_BUS_ADDRESS environments with debugging shells for your project, specify the following:

shell:

  # Share some environment variables from the host environment
  environment:
    DISPLAY: '$DISPLAY'
    DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'

Or, a more complex example is how one might share the host pulseaudio server with a bst shell environment:

shell:

  # Set some environment variables explicitly
  environment:
    PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'

Host files

It can be useful to share some files on the host with a shell so that it can integrate better with the host environment.

The host-files configuration allows one to specify files and directories on the host to be bind mounted into the sandbox.

Note

The host-files configuration is available since format version 4

Warning

One should never mount directories where one expects to find data and files which belong to the user, such as /home on POSIX platforms.

This is because the unsuspecting user may corrupt their own files accidentally as a result. Instead users can use the --mount option of bst shell to mount data into the shell.

The host-files configuration is an ordered list of mount specifications.

Members of the list can be fully specified as a dictionary, or a simple string can be used if only the defaults are required.

The fully specified dictionary has the following members:

• path

  The path inside the sandbox. This is the only mandatory member of the mount specification.

• host_path

  The host path to mount at path in the sandbox. This will default to path if left unspecified.

• optional

  Whether the mount should be considered optional. This is False by default.

Here is an example of a fully specified mount specification:

shell:

  # Mount an arbitrary resolv.conf from the host to
  # /etc/resolv.conf in the sandbox, and avoid any
  # warnings if the host resolv.conf doesnt exist.
  host-files:
  - host_path: '/usr/local/work/etc/resolv.conf'
    path: '/etc/resolv.conf'
    optional: True

Here is an example of using shorthand mount specifications:

shell:

  # Specify a list of files to mount in the sandbox
  # directory from the host.
  #
  # If these do not exist on the host, a warning will
  # be issued but the shell will still be launched.
  host-files:
  - '/etc/passwd'
  - '/etc/group'
  - '/etc/resolv.conf'

Host side environment variable expansion is also supported:

shell:

  # Mount a host side pulseaudio server socket into
  # the shell environment at the same location.
  host-files:
  - '${XDG_RUNTIME_DIR}/pulse/native'

Builtin defaults

BuildStream defines some default values for convenience, the default values overridden by your project’s project.conf are presented here:

# Default BuildStream project configuration.


# General configuration defaults
#

# Require format version 0
format-version: 0

# Elements are found at the project root
element-path: .

# Store source references in element files
ref-storage: inline

# Overlaps are just warnings
fail-on-overlap: False


# Variable Configuration
#
variables:
  # Path configuration, to be used in build instructions.
  prefix: "/usr"
  exec_prefix: "%{prefix}"
  bindir: "%{exec_prefix}/bin"
  sbindir: "%{exec_prefix}/sbin"
  libexecdir: "%{exec_prefix}/libexec"
  datadir: "%{prefix}/share"
  sysconfdir: "/etc"
  sharedstatedir: "%{prefix}/com"
  localstatedir: "/var"
  lib: "lib"
  libdir: "%{prefix}/%{lib}"
  debugdir: "%{libdir}/debug"
  includedir: "%{prefix}/include"
  docdir: "%{datadir}/doc"
  infodir: "%{datadir}/info"
  mandir: "%{datadir}/man"

  # Indicates the default build directory where input is
  # normally staged
  build-root: /buildstream/%{project-name}/%{element-name}

  # Indicates the build installation directory in the sandbox
  install-root: /buildstream-install

  # Arguments for tooling used when stripping debug symbols
  objcopy-link-args: --add-gnu-debuglink
  objcopy-extract-args: |

    --only-keep-debug --compress-debug-sections

  strip-args: |

    --remove-section=.comment --remove-section=.note --strip-unneeded

  # Generic implementation for stripping debugging symbols
  strip-binaries: |

    cd "%{install-root}" && find -type f \
      '(' -perm -111 -o -name '*.so*' \
          -o -name '*.cmxs' -o -name '*.node' ')' \
      -exec sh -ec \
      'read -n4 hdr <"$1" # check for elf header
       if [ "$hdr" != "$(printf \\x7fELF)" ]; then
           exit 0
       fi
       debugfile="%{install-root}%{debugdir}/$1"
       mkdir -p "$(dirname "$debugfile")"
       objcopy %{objcopy-extract-args} "$1" "$debugfile"
       chmod 644 "$debugfile"
       strip %{strip-args} "$1"
       objcopy %{objcopy-link-args} "$debugfile" "$1"' - {} ';'

  # Generic implementation for reproducible python builds
  fix-pyc-timestamps: |

    find "%{install-root}" -name '*.pyc' -exec \
      dd if=/dev/zero of={} bs=1 count=4 seek=4 conv=notrunc ';'

# Base sandbox environment, can be overridden by plugins
environment:
  PATH: /usr/bin:/bin:/usr/sbin:/sbin
  SHELL: /bin/sh
  TERM: dumb
  USER: tomjon
  USERNAME: tomjon
  LOGNAME: tomjon
  LC_ALL: C
  HOME: /tmp
  TZ: UTC

  # For reproducible builds we use 2011-11-11 as a constant
  SOURCE_DATE_EPOCH: 1320937200

# List of environment variables which should not be taken into
# account when calculating a cache key for a given element.
#
environment-nocache: []

# Configuration for the sandbox other than environment variables
# should go in 'sandbox'. This just contains the UID and GID that
# the user in the sandbox will have. Not all sandboxes will support
# changing the values.
sandbox:
  build-uid: 0
  build-gid: 0

# Defaults for the 'split-rules' public data found on elements
# in the 'bst' domain.
#
split-rules:

  # The runtime domain includes whatever is needed for the
  # built element to run, this includes stripped executables
  # and shared libraries by default.
  runtime:
  - |
    %{bindir}/*
  - |
    %{sbindir}/*
  - |
    %{libexecdir}/*
  - |
    %{libdir}/lib*.so*

  # The devel domain includes additional things which
  # you may need for development.
  #
  # By default this includes header files, static libraries
  # and other metadata such as pkgconfig files, m4 macros and
  # libtool archives.
  devel:
  - |
    %{includedir}
  - |
    %{includedir}/**
  - |
    %{libdir}/lib*.a
  - |
    %{libdir}/lib*.la
  - |
    %{libdir}/pkgconfig/*.pc
  - |
    %{datadir}/pkgconfig/*.pc
  - |
    %{datadir}/aclocal/*.m4

  # The debug domain includes debugging information stripped
  # away from libraries and executables
  debug:
  - |
    %{debugdir}
  - |
    %{debugdir}/**

  # The doc domain includes documentation
  doc:
  - |
    %{docdir}
  - |
    %{docdir}/**
  - |
    %{infodir}
  - |
    %{infodir}/**
  - |
    %{mandir}
  - |
    %{mandir}/**

  # The locale domain includes translations etc
  locale:
  - |
    %{datadir}/locale
  - |
    %{datadir}/locale/**
  - |
    %{datadir}/i18n
  - |
    %{datadir}/i18n/**
  - |
    %{datadir}/zoneinfo
  - |
    %{datadir}/zoneinfo/**


# Default behavior for `bst shell`
#
shell:

  # Command to run when `bst shell` does not provide a command
  #
  command: [ 'sh', '-i' ]