5. Optionality and directives
In this chapter we’re going to go over some of the more flexible constructs which BuildStream offers for optionality, and show how we can use directives in the BuildStream YAML format.
Note
This example is distributed with BuildStream in the doc/examples/directives subdirectory.
5.1. Overview
This chapter’s example will build another hello.c program which much
resembles the program in the running commands example,
but here we’re going to make the greeting string configurable using the C
preprocessor.
We’ll be compiling the following C file:
5.1.1. files/src/hello.c
/*
* hello.c - Simple hello world program
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf(GREETING_MESSAGE);
return 0;
}
And we’re going to build it using make, using the following Makefile:
5.1.2. files/src/Makefile
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -DGREETING_MESSAGE="\"${GREETING}\"" -Wall -o $@ $<
Notice the addition of -DGREETING_MESSAGE="\"${GREETING}\"" in the above
Makefile, this will allow us to configure the greeting message from the
hello.bst element declaration.
We will need to add support to our project for optionality, and we’ll have to make conditional statements to resolve what kind of greeting we want from the hello world program.
5.2. Project structure
Since this project has much the same structure as the
running commands chapter did, we won’t go over all of
these elements in detail. Instead let’s focus on the addition of the new
project options in project.conf, the added
file in the include/ project subdirectory, and how these come together
in the the hello.bst element.
5.2.1. project.conf
# Unique project name
name: directives
# Minimum required BuildStream version
min-version: 2.0
# Subdirectory where elements are stored
element-path: elements
# Define an alias for our alpine tarball
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
# Define an option for this project
#
options:
flavor:
type: enum
description: Flavor of the greeting in the hello world program
values:
- normal
- somber
- excited
default: normal
Here, our project.conf declares a project option called flavor, and this
will inform what kind of greeting message we want to use when building the project.
5.2.2. elements/hello.bst
kind: manual
description: |
A hello world program with a custom greeting message
# Depend on the base system
depends:
- base.bst
# Stage the files/src directory for building
sources:
- kind: local
path: files/src
# This include file defines the %{greeting} variable used below
variables:
(@): include/greeting.bst
# Now configure the commands to run
config:
# This time we inform the Makefile of which greeting we want
build-commands:
- make PREFIX="%{prefix}" GREETING="%{greeting}"
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Notice the (@) symbol we’ve added in the variables: section, this
symbol is used to invoke the include directive,
which can be useful for code sharing between elements or simply to improve readability.
In this case, we are compositing the content of include/greeting.bst into the
variables section of the element declaration, directives
can however be used virtually anywhere in the BuildStream YAML format.
5.2.3. include/greeting.bst
# We define the greeting message here conditionally
(?):
- flavor == "normal":
greeting: |
Hello world !
- flavor == "somber":
greeting: |
Hey world.
- flavor == "excited":
greeting: |
Howdy there, and what a world it is !
Here we can see the dictionary which will be composited into the variables:
section of the hello.bst element described above.
Note the usage of the (?) symbol at the toplevel of the YAML dictionary,
this is how we perform conditional statements
in the BuildStream YAML format.
This include file uses the flavor project option we declared in project.conf to
decide what value will end up being assigned to the %{greeting} variable, which
will ultimately be used in the hello.bst element.
5.3. Using the project
Now that we have a project which uses options and conditional statements, lets build the project with a few different options and observe the outputs.
5.3.1. Building hello.bst element with options
Since the flavor option we’ve declared above has a default, we can build it the first time using bst build without any special command line options:
user@host:~/directives$ bst build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732063.899610 2096 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:43
Project: directives (/home/user/buildstream/doc/examples/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-kki3p4d4/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-kki3p4d4
Log Files: /home/user/buildstream/doc/run-bst-kki3p4d4/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-kki3p4d4/sources
Build Area: /home/user/buildstream/doc/run-bst-kki3p4d4/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: normal
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
fetch needed 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst
waiting 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst
waiting 49a52215c02358a4bbbb632a74986fa23dc31a36ee8055feffbddc9e3c735d1a hello.bst
===============================================================================
[--:--:--][4275015f][ fetch:base/alpine.bst ] START directives/base-alpine/4275015f-fetch.20240815-142743.log
[--:--:--][22843536][ fetch:base.bst ] START directives/base/22843536-fetch.20240815-142743.log
[--:--:--][4275015f][ fetch:base/alpine.bst ] START Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:00][22843536][ fetch:base.bst ] SUCCESS directives/base/22843536-fetch.20240815-142743.log
[00:00:00][4275015f][ fetch:base/alpine.bst ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][4275015f][ fetch:base/alpine.bst ] SUCCESS directives/base-alpine/4275015f-fetch.20240815-142743.log
[--:--:--][4275015f][ build:base/alpine.bst ] START directives/base-alpine/4275015f-build.20240815-142750.log
[--:--:--][4275015f][ build:base/alpine.bst ] START Staging sources
[00:00:00][4275015f][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][4275015f][ build:base/alpine.bst ] START Caching artifact
[00:00:00][4275015f][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][4275015f][ build:base/alpine.bst ] SUCCESS directives/base-alpine/4275015f-build.20240815-142750.log
[--:--:--][22843536][ build:base.bst ] START directives/base/22843536-build.20240815-142750.log
[--:--:--][22843536][ build:base.bst ] START Caching artifact
[00:00:00][22843536][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][22843536][ build:base.bst ] SUCCESS directives/base/22843536-build.20240815-142750.log
[--:--:--][49a52215][ build:hello.bst ] START directives/hello/49a52215-build.20240815-142750.log
[--:--:--][49a52215][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][49a52215][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][49a52215][ build:hello.bst ] START Integrating sandbox
[00:00:00][49a52215][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][49a52215][ build:hello.bst ] START Staging sources
[00:00:00][49a52215][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][49a52215][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Hello world !"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][49a52215][ build:hello.bst ] SUCCESS Running commands
[--:--:--][49a52215][ build:hello.bst ] START Caching artifact
[00:00:00][49a52215][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][49a52215][ build:hello.bst ] SUCCESS directives/hello/49a52215-build.20240815-142750.log
[00:00:06][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 2, skipped 1, failed 0
Build Queue: processed 3, skipped 0, failed 0
If we want to build the somber flavor, we just need to specify the
additional --option command line option to bst
in order to inform BuildStream of the options we want.
user@host:~/directives$ bst --option flavor somber build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732071.142859 2158 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:51
Project: directives (/home/user/buildstream/doc/examples/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-kki3p4d4/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-kki3p4d4
Log Files: /home/user/buildstream/doc/run-bst-kki3p4d4/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-kki3p4d4/sources
Build Area: /home/user/buildstream/doc/run-bst-kki3p4d4/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: somber
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst
cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst
buildable 14183452db56642e12bc721cd3501237d397ac87b1f24988665d69177fe488cb hello.bst
===============================================================================
[--:--:--][14183452][ build:hello.bst ] START directives/hello/14183452-build.20240815-142751.log
[--:--:--][14183452][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][14183452][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][14183452][ build:hello.bst ] START Integrating sandbox
[00:00:00][14183452][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][14183452][ build:hello.bst ] START Staging sources
[00:00:00][14183452][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][14183452][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Hey world."
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][14183452][ build:hello.bst ] SUCCESS Running commands
[--:--:--][14183452][ build:hello.bst ] START Caching artifact
[00:00:00][14183452][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][14183452][ build:hello.bst ] SUCCESS directives/hello/14183452-build.20240815-142751.log
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 1, skipped 2, failed 0
Note that the --option option can be specified many times on the
bst command line, so as to support projects which have multiple
options.
Finally lets get the excited flavor built as well:
user@host:~/directives$ bst --option flavor excited build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732071.747530 2214 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:51
Project: directives (/home/user/buildstream/doc/examples/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-kki3p4d4/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-kki3p4d4
Log Files: /home/user/buildstream/doc/run-bst-kki3p4d4/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-kki3p4d4/sources
Build Area: /home/user/buildstream/doc/run-bst-kki3p4d4/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: excited
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 4275015f95b90fd68b24421a9586814f97cc03630ce28d0b9fe35435bfe08ca5 base/alpine.bst
cached 2284353672b080d3b1f19eacc625c0b6449109676abd0144328ca2521dabc977 base.bst
buildable 72169f19db218d37d9f9b2a8603c0b75dcd391d3a07e5ef900f95e6fdaaca1ba hello.bst
===============================================================================
[--:--:--][72169f19][ build:hello.bst ] START directives/hello/72169f19-build.20240815-142751.log
[--:--:--][72169f19][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][72169f19][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][72169f19][ build:hello.bst ] START Integrating sandbox
[00:00:00][72169f19][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][72169f19][ build:hello.bst ] START Staging sources
[00:00:00][72169f19][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][72169f19][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Howdy there, and what a world it is !"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][72169f19][ build:hello.bst ] SUCCESS Running commands
[--:--:--][72169f19][ build:hello.bst ] START Caching artifact
[00:00:00][72169f19][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][72169f19][ build:hello.bst ] SUCCESS directives/hello/72169f19-build.20240815-142751.log
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 1, skipped 2, failed 0
If you observe the cache keys above, you will notice that while
we have only three elements in the pipeline, counting base/alpine.bst,
base.bst and hello.bst, we have actually built five artifacts,
because the hello.bst is built differently each time, it has a
different cache key and is stored separately in the artifact cache.
5.3.2. Run the hello world program with options
Since the --option command line option to bst
is a main option, it can be used in any command.
Let’s run the hello program using bst shell
three times in a row, each time using a different option so we can
observe the results.
user@host:~/directives$ bst shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732072.358459 2270 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][49a52215][ main:hello.bst ] START Staging dependencies
[00:00:00][49a52215][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][49a52215][ main:hello.bst ] START Integrating sandbox
[00:00:00][49a52215][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][49a52215][ main:hello.bst ] STATUS Running command
hello
Hello world !
user@host:~/directives$ bst --option flavor somber shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732072.862908 2314 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][14183452][ main:hello.bst ] START Staging dependencies
[00:00:00][14183452][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][14183452][ main:hello.bst ] START Integrating sandbox
[00:00:00][14183452][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][14183452][ main:hello.bst ] STATUS Running command
hello
Hey world.
user@host:~/directives$ bst --option flavor excited shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732073.369914 2358 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][72169f19][ main:hello.bst ] START Staging dependencies
[00:00:00][72169f19][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][72169f19][ main:hello.bst ] START Integrating sandbox
[00:00:00][72169f19][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][72169f19][ main:hello.bst ] STATUS Running command
hello
Howdy there, and what a world it is !
5.4. Summary
In this chapter we’ve demonstrated how to declare project options, how to use conditional directives, and also how to use include directives.
To get more familliar with these concepts, you may want to explore the remaining directives in the BuildStream YAML format, and also take a look at the various types of project options that are also supported.