5. Optionality and directives

In this chapter we’re going to go over some of the more flexible constructs which BuildStream offers for optionality, and show how we can use directives in the BuildStream YAML format.

Note

This example is distributed with BuildStream in the doc/examples/directives subdirectory.

5.1. Overview

This chapter’s example will build another hello.c program which much resembles the program in the running commands example, but here we’re going to make the greeting string configurable using the C preprocessor.

We’ll be compiling the following C file:

5.1.1. files/src/hello.c

/*
 * hello.c - Simple hello world program
 */
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf(GREETING_MESSAGE);
  return 0;
}

And we’re going to build it using make, using the following Makefile:

5.1.2. files/src/Makefile

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -m 755 hello ${DESTDIR}${PREFIX}/bin

hello: hello.c
	$(CC) -DGREETING_MESSAGE="\"${GREETING}\"" -Wall -o $@ $<

Notice the addition of -DGREETING_MESSAGE="\"${GREETING}\"" in the above Makefile, this will allow us to configure the greeting message from the hello.bst element declaration.

We will need to add support to our project for optionality, and we’ll have to make conditional statements to resolve what kind of greeting we want from the hello world program.

5.2. Project structure

Since this project has much the same structure as the running commands chapter did, we won’t go over all of these elements in detail. Instead let’s focus on the addition of the new project options in project.conf, the added file in the include/ project subdirectory, and how these come together in the the hello.bst element.

5.2.1. project.conf

# Unique project name
name: directives

# Minimum required BuildStream version
min-version: 2.0

# Subdirectory where elements are stored
element-path: elements

# Define an alias for our alpine tarball
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

# Define an option for this project
#
options:
  flavor:
    type: enum
    description: Flavor of the greeting in the hello world program
    values:
    - normal
    - somber
    - excited
    default: normal

Here, our project.conf declares a project option called flavor, and this will inform what kind of greeting message we want to use when building the project.

5.2.2. elements/hello.bst

kind: manual
description: |

  A hello world program with a custom greeting message

# Depend on the base system
depends:
- base.bst

# Stage the files/src directory for building
sources:
  - kind: local
    path: files/src

# This include file defines the %{greeting} variable used below
variables:
  (@): include/greeting.bst

# Now configure the commands to run
config:

  # This time we inform the Makefile of which greeting we want
  build-commands:
  - make PREFIX="%{prefix}" GREETING="%{greeting}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

Notice the (@) symbol we’ve added in the variables: section, this symbol is used to invoke the include directive, which can be useful for code sharing between elements or simply to improve readability.

In this case, we are compositing the content of include/greeting.bst into the variables section of the element declaration, directives can however be used virtually anywhere in the BuildStream YAML format.

5.2.3. include/greeting.bst

# We define the greeting message here conditionally
(?):
- flavor == "normal":
    greeting: |

      Hello world !

- flavor == "somber":
    greeting: |

      Hey world.

- flavor == "excited":
    greeting: |

      Howdy there, and what a world it is !

Here we can see the dictionary which will be composited into the variables: section of the hello.bst element described above.

Note the usage of the (?) symbol at the toplevel of the YAML dictionary, this is how we perform conditional statements in the BuildStream YAML format.

This include file uses the flavor project option we declared in project.conf to decide what value will end up being assigned to the %{greeting} variable, which will ultimately be used in the hello.bst element.

5.3. Using the project

Now that we have a project which uses options and conditional statements, lets build the project with a few different options and observe the outputs.

5.3.1. Building hello.bst element with options

Since the flavor option we’ve declared above has a default, we can build it the first time using bst build without any special command line options:

user@host:~/directives$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.0.1
    Session Start: Friday, 17-02-2023 at 10:33:55
    Project:       directives (/home/user/buildstream/doc/examples/directives)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-x3iqf6z1/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-x3iqf6z1
    Log Files:               /home/user/buildstream/doc/run-bst-x3iqf6z1/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-x3iqf6z1/sources
    Build Area:              /home/user/buildstream/doc/run-bst-x3iqf6z1/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: directives

    Project Options
        flavor: normal

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
fetch needed 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
     waiting 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
     waiting 49a52215c02358a4bbbb632a74986fa23dc31a36ee8055feffbddc9e3c735d1a hello.bst 
===============================================================================
[--:--:--][4275015f][   fetch:base/alpine.bst               ] START   directives/base-alpine/4275015f-fetch.1889.log
[--:--:--][4275015f][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][22843536][   fetch:base.bst                      ] START   directives/base/22843536-fetch.1889.log
[00:00:00][22843536][   fetch:base.bst                      ] SUCCESS directives/base/22843536-fetch.1889.log
[00:00:00][4275015f][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:10][4275015f][   fetch:base/alpine.bst               ] SUCCESS directives/base-alpine/4275015f-fetch.1889.log
[--:--:--][4275015f][   build:base/alpine.bst               ] START   directives/base-alpine/4275015f-build.1889.log
[--:--:--][4275015f][   build:base/alpine.bst               ] START   Staging sources
[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][4275015f][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][4275015f][   build:base/alpine.bst               ] SUCCESS directives/base-alpine/4275015f-build.1889.log
[--:--:--][22843536][   build:base.bst                      ] START   directives/base/22843536-build.1889.log
[--:--:--][22843536][   build:base.bst                      ] START   Caching artifact
[00:00:00][22843536][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][22843536][   build:base.bst                      ] SUCCESS directives/base/22843536-build.1889.log
[--:--:--][49a52215][   build:hello.bst                     ] START   directives/hello/49a52215-build.1889.log
[--:--:--][49a52215][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][49a52215][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][49a52215][   build:hello.bst                     ] START   Staging sources
[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][49a52215][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr" GREETING="Hello world !"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][49a52215][   build:hello.bst                     ] START   Caching artifact
[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][49a52215][   build:hello.bst                     ] SUCCESS directives/hello/49a52215-build.1889.log
[00:00:11][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 2, skipped 1, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

If we want to build the somber flavor, we just need to specify the additional --option command line option to bst in order to inform BuildStream of the options we want.

user@host:~/directives$ bst --option flavor somber build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.0.1
    Session Start: Friday, 17-02-2023 at 10:34:06
    Project:       directives (/home/user/buildstream/doc/examples/directives)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-x3iqf6z1/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-x3iqf6z1
    Log Files:               /home/user/buildstream/doc/run-bst-x3iqf6z1/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-x3iqf6z1/sources
    Build Area:              /home/user/buildstream/doc/run-bst-x3iqf6z1/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: directives

    Project Options
        flavor: somber

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
      cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
      cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
   buildable 14183452db56642e12bc721cd3501237d397ac87b1f24988665d69177fe488cb hello.bst 
===============================================================================
[--:--:--][14183452][   build:hello.bst                     ] START   directives/hello/14183452-build.1940.log
[--:--:--][14183452][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][14183452][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][14183452][   build:hello.bst                     ] START   Staging sources
[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][14183452][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr" GREETING="Hey world."
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][14183452][   build:hello.bst                     ] START   Caching artifact
[00:00:00][14183452][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][14183452][   build:hello.bst                     ] SUCCESS directives/hello/14183452-build.1940.log
[00:00:00][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 0, skipped 3, failed 0 
    Build Queue: processed 1, skipped 2, failed 0

Note that the --option option can be specified many times on the bst command line, so as to support projects which have multiple options.

Finally lets get the excited flavor built as well:

user@host:~/directives$ bst --option flavor excited build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.0.1
    Session Start: Friday, 17-02-2023 at 10:34:07
    Project:       directives (/home/user/buildstream/doc/examples/directives)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-x3iqf6z1/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-x3iqf6z1
    Log Files:               /home/user/buildstream/doc/run-bst-x3iqf6z1/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-x3iqf6z1/sources
    Build Area:              /home/user/buildstream/doc/run-bst-x3iqf6z1/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: directives

    Project Options
        flavor: excited

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
      cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst 
      cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst 
   buildable 72169f19db218d37d9f9b2a8603c0b75dcd391d3a07e5ef900f95e6fdaaca1ba hello.bst 
===============================================================================
[--:--:--][72169f19][   build:hello.bst                     ] START   directives/hello/72169f19-build.1986.log
[--:--:--][72169f19][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][72169f19][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][72169f19][   build:hello.bst                     ] START   Staging sources
[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][72169f19][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr" GREETING="Howdy there, and what a world it is !"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][72169f19][   build:hello.bst                     ] START   Caching artifact
[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][72169f19][   build:hello.bst                     ] SUCCESS directives/hello/72169f19-build.1986.log
[00:00:00][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 0, skipped 3, failed 0 
    Build Queue: processed 1, skipped 2, failed 0

If you observe the cache keys above, you will notice that while we have only three elements in the pipeline, counting base/alpine.bst, base.bst and hello.bst, we have actually built five artifacts, because the hello.bst is built differently each time, it has a different cache key and is stored separately in the artifact cache.

5.3.2. Run the hello world program with options

Since the --option command line option to bst is a main option, it can be used in any command.

Let’s run the hello program using bst shell three times in a row, each time using a different option so we can observe the results.

user@host:~/directives$ bst shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][49a52215][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][49a52215][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][49a52215][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][49a52215][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][49a52215][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:34:08.734+0000 [2055:139788074340352] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello world !
user@host:~/directives$ bst --option flavor somber shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][14183452][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][14183452][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][14183452][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][14183452][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][14183452][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:34:09.532+0000 [2089:140002973306880] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hey world.
user@host:~/directives$ bst --option flavor excited shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][72169f19][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][72169f19][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][72169f19][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][72169f19][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][72169f19][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:34:10.301+0000 [2123:140158674456576] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Howdy there, and what a world it is !

5.4. Summary

In this chapter we’ve demonstrated how to declare project options, how to use conditional directives, and also how to use include directives.

To get more familliar with these concepts, you may want to explore the remaining directives in the BuildStream YAML format, and also take a look at the various types of project options that are also supported.