3. Subproject includes
We’ve already discussed how we can add optionality to projects and explored how we can perform conditional statements and include fragments of BuildStream YAML in the earlier chapter about optionality and directives.
In this chapter we’re going to explore how we can use include directives to include YAML fragments from a subproject
referred to by a junction element, and how
project options can be specified in the configuration
of your junction.
Note
This example is distributed with BuildStream in the doc/examples/junction-includes subdirectory.
3.1. Overview
It is a goal of BuildStream to provide developers and integrators with the tools
they need to maintain software stacks which depend on eachother with least friction
as possible, such that one can integrate upgrades of projects one depends on
via junction elements regularly and with the least
hassle as possible.
Project options and include directives combined form the basis on which projects can maximize on code sharing effectively, and the basis on which BuildStream projects can form reliable APIs.
3.1.1. Project options
The options which a project exposes is a fairly limited API surface, it allows one to configure a limited set of options advertized by the project maintainers, and the options will affect what kind of artifacts will be produced by the project.
This kind of optionality however does not allow consumers to entirely redefine how artifacts are produced and how elements are configured.
On the one hand, this limitation can be frustrating, as one constantly finds themselves requiring a feature that their subproject does not support right now. On the other hand, the limitation of features which a given project chooses to support is what guards downstream project consumers against consuming artifacts which are not supported by the upstream.
Project options are designed to enforce a separation of concerns, where we expect that downstreams will either fork a project in order to support a new feature, or convince the upstream to start supporting a new feature. Furthermore, limited API surfaces for interdependent projects offers a possibility of API stability of projects, such that you can upgrade your dependencies with limited friction.
3.1.2. Includes
The includes which a project might advertize as “public”, form the output of the API exchange between a project and its subproject(s).
Cross-project include files allow a project to inherit configuration from a subproject. Include files can be used to define anything from the variables one needs to have in context in order to build into or link into alternative system prefixes, what special compiler flags to use when building for a specific machine architecture, to customized shell configurations to use when testing out applications in bst shell.
This chapter will provide an example of the mechanics of cross project includes when combined with project optionality.
3.2. Project structure
3.2.1. Project options
This example is comprised of two separate projects, both of which offer some project options. This is intended to emphasize how your toplevel project options can be used to select and configure options to use in the subprojects you depend on.
For convenience, the subproject is stored in the subdirectory of the toplevel project, while in the real world the subproject is probably hosted elsewhere.
First let’s take a look at the options declared in the respective
project.conf files.
3.2.1.1. Toplevel project.conf
name: project
min-version: 2.0
element-path: elements
aliases:
gnu: http://ftpmirror.gnu.org/gnu/automake/
plugins:
- origin: pip
package-name: buildstream-plugins
elements:
- autotools
# Define some options for this project
#
options:
funky:
type: bool
description: Whether this project is funky
default: False
3.2.1.2. Subproject project.conf
name: subproject
min-version: 2.0
element-path: elements
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
# Define some options for this project
#
options:
color:
type: enum
description: The color of this runtime
values:
- red
- green
- blue
default: blue
As we can see, these two projects both offer some arbitrarily named options.
3.2.2. Conditional configuration of subproject
The toplevel project here does some conditional configuration of the subproject.
3.2.2.1. Toplevel elements/subproject-junction.bst
kind: junction
config:
# Configure the options for subproject
#
# If our project is funky, then it requires
# a blue subproject, otherwise we use a red one.
#
options:
color: red
(?):
- funky == True:
color: blue
sources:
- kind: local
path: subproject
Here we can see that projects can use conditional statements to make decisions about subproject configuration based on their own configuration.
In this example, if the toplevel project is funky, then it will
configure its subproject with color set to blue, otherwise it
will use the red variant of the subproject color.
3.2.3. Including configuration from a subproject
Here there are a couple of aspects to observe, namely how the toplevel project includes files across a junction boundary, and how that include file might be implemented.
3.2.3.1. Toplevel elements/hello.bst
kind: autotools
description: |
Hello world example from automake
variables:
# The special paths.bst from our subproject is used to
# define the paths of some elements in this project.
#
(@): subproject-junction.bst:include/paths.bst
# The hello world example lives in the doc/amhello folder.
command-subdir: doc/amhello
sources:
- kind: tar
url: gnu:automake-1.16.tar.gz
ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f
depends:
- filename: base.bst
junction: subproject-junction.bst
Here we can see the same element which we discussed in the
autotools example, except that we’re including
a file from the subproject. As explained in the reference manual,
this is done by prefixing the include path with the local junction
element name and then a colon.
Note that in this case, the API contract is simply that hello.bst is
including paths.bst, and has the expectation that paths.bst will
in some way influence the variables, nothing more.
It can be that an include file is expected to create new variables, and it can be that the subproject might declare things differently depending on the subproject’s own configuration, as we will observe next.
3.2.3.2. Subproject include/paths.bst
# When this project color is blue, including this
# file causes installations to be relocated to /opt
#
prefix: /usr
(?):
- color == "blue":
prefix: /opt
Here, we can see the include file itself is making a conditional statement, in turn deciding what values to use depending on how the project was configured.
This decision will provide valuable context for any file including paths.bst,
whether it be an element, a project.conf which applies the variable as
a default for the entire project, whether it is being included by files
in the local project, or whether it is being included by a downstream
project which junctions this project, as is the case in this example.
3.3. Using the project
At this stage, you have probably already reasoned out what would happen if we tried to build and run the project.
Nevertheless, we will still present the outputs here for observation.
3.3.1. Building the project normally
Here we build the project without any special arguments.
user@host:~/junction-includes$ bst build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.0.1
Session Start: Friday, 17-02-2023 at 10:34:56
Project: project (/home/user/buildstream/doc/examples/junction-includes)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-marzbhof/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-marzbhof
Log Files: /home/user/buildstream/doc/run-bst-marzbhof/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-marzbhof/sources
Build Area: /home/user/buildstream/doc/run-bst-marzbhof/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: project
Project Options
funky: 0
Element Plugins
junction: core plugin
autotools: python package 'buildstream-plugins 1.91.0' at: /home/user/buildstream/.tox/docs/lib/python3.10/site-packages
Source Plugins
local: core plugin
tar: core plugin
Project: subproject
Junction path: subproject-junction.bst
Loaded by: hello.bst [line 11 column 7]
Project Options
color: red
Element Plugins
stack: core plugin
import: core plugin
Source Plugins
tar: core plugin
Pipeline
fetch needed 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst
waiting c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst
waiting f2b2aa995f2803923eb0c21eaf9df2265ce4a90fcfd9d4b0e3402807020ed2f0 hello.bst
===============================================================================
[--:--:--][2130295a][ fetch:subproject-junction.bst:base/alpine.bst] START subproject/base-alpine/2130295a-fetch.2755.log
[--:--:--][2130295a][ fetch:subproject-junction.bst:base/alpine.bst] START Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][c93bc58b][ fetch:subproject-junction.bst:base.bst] START subproject/base/c93bc58b-fetch.2755.log
[00:00:00][c93bc58b][ fetch:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-fetch.2755.log
[00:00:00][2130295a][ fetch:subproject-junction.bst:base/alpine.bst] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:10][2130295a][ fetch:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-fetch.2755.log
[--:--:--][2130295a][ build:subproject-junction.bst:base/alpine.bst] START subproject/base-alpine/2130295a-build.2755.log
[--:--:--][2130295a][ build:subproject-junction.bst:base/alpine.bst] START Staging sources
[00:00:00][2130295a][ build:subproject-junction.bst:base/alpine.bst] SUCCESS Staging sources
[--:--:--][2130295a][ build:subproject-junction.bst:base/alpine.bst] START Caching artifact
[00:00:00][2130295a][ build:subproject-junction.bst:base/alpine.bst] SUCCESS Caching artifact
[00:00:00][2130295a][ build:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-build.2755.log
[--:--:--][c93bc58b][ build:subproject-junction.bst:base.bst] START subproject/base/c93bc58b-build.2755.log
[--:--:--][c93bc58b][ build:subproject-junction.bst:base.bst] START Caching artifact
[00:00:00][c93bc58b][ build:subproject-junction.bst:base.bst] SUCCESS Caching artifact
[00:00:00][c93bc58b][ build:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-build.2755.log
[--:--:--][f2b2aa99][ build:hello.bst ] START project/hello/f2b2aa99-build.2755.log
[--:--:--][f2b2aa99][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][f2b2aa99][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][f2b2aa99][ build:hello.bst ] START Integrating sandbox
[00:00:00][f2b2aa99][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][f2b2aa99][ build:hello.bst ] START Staging sources
[00:00:00][f2b2aa99][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][f2b2aa99][ build:hello.bst ] START Running commands
export NOCONFIGURE=1;
if [ -x ./configure ]; then true;
elif [ -x ./autogen ]; then ./autogen;
elif [ -x ./autogen.sh ]; then ./autogen.sh;
elif [ -x ./bootstrap ]; then ./bootstrap;
elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
else autoreconf -ivf .;
fi
./configure --prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--datadir=/usr/share \
--includedir=/usr/include \
--libdir=/usr/lib \
--libexecdir=/usr/libexec \
--localstatedir=/var \
--sharedstatedir=/usr/com \
Message contains 23 additional lines
[00:00:04][f2b2aa99][ build:hello.bst ] SUCCESS Running commands
[--:--:--][f2b2aa99][ build:hello.bst ] START Caching artifact
[00:00:00][f2b2aa99][ build:hello.bst ] SUCCESS Caching artifact
[00:00:04][f2b2aa99][ build:hello.bst ] SUCCESS project/hello/f2b2aa99-build.2755.log
[00:00:16][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 2, skipped 1, failed 0
Build Queue: processed 3, skipped 0, failed 0
3.3.2. Building the project in funky mode
Now let’s see what happens when we build the project in funky mode
user@host:~/junction-includes$ bst --option funky True build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.0.1
Session Start: Friday, 17-02-2023 at 10:35:13
Project: project (/home/user/buildstream/doc/examples/junction-includes)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-marzbhof/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-marzbhof
Log Files: /home/user/buildstream/doc/run-bst-marzbhof/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-marzbhof/sources
Build Area: /home/user/buildstream/doc/run-bst-marzbhof/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: project
Project Options
funky: 1
Element Plugins
junction: core plugin
autotools: python package 'buildstream-plugins 1.91.0' at: /home/user/buildstream/.tox/docs/lib/python3.10/site-packages
Source Plugins
local: core plugin
tar: core plugin
Project: subproject
Junction path: subproject-junction.bst
Loaded by: hello.bst [line 11 column 7]
Project Options
color: blue
Element Plugins
stack: core plugin
import: core plugin
Source Plugins
tar: core plugin
Pipeline
cached 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst
cached c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst
buildable 8b7e1a9bfa0c4dea2b86d47589ff140f154cf08cc02c8d3544ccbd6f5c0f3c4c hello.bst
===============================================================================
[--:--:--][8b7e1a9b][ build:hello.bst ] START project/hello/8b7e1a9b-build.3758.log
[--:--:--][8b7e1a9b][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][8b7e1a9b][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][8b7e1a9b][ build:hello.bst ] START Integrating sandbox
[00:00:00][8b7e1a9b][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][8b7e1a9b][ build:hello.bst ] START Staging sources
[00:00:00][8b7e1a9b][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][8b7e1a9b][ build:hello.bst ] START Running commands
export NOCONFIGURE=1;
if [ -x ./configure ]; then true;
elif [ -x ./autogen ]; then ./autogen;
elif [ -x ./autogen.sh ]; then ./autogen.sh;
elif [ -x ./bootstrap ]; then ./bootstrap;
elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
else autoreconf -ivf .;
fi
./configure --prefix=/opt \
--exec-prefix=/opt \
--bindir=/opt/bin \
--sbindir=/opt/sbin \
--sysconfdir=/etc \
--datadir=/opt/share \
--includedir=/opt/include \
--libdir=/opt/lib \
--libexecdir=/opt/libexec \
--localstatedir=/var \
--sharedstatedir=/opt/com \
Message contains 23 additional lines
[00:00:04][8b7e1a9b][ build:hello.bst ] SUCCESS Running commands
[--:--:--][8b7e1a9b][ build:hello.bst ] START Caching artifact
[00:00:00][8b7e1a9b][ build:hello.bst ] SUCCESS Caching artifact
[00:00:04][8b7e1a9b][ build:hello.bst ] SUCCESS project/hello/8b7e1a9b-build.3758.log
[00:00:05][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 1, skipped 2, failed 0
As we can see, this time we’ve built the project into the /opt
system prefix instead of the standard /usr prefix.
Let’s just take a step back now and summarize the process which went into this decision:
The toplevel
project.confexposes the booleanfunkyoptionThe toplevel junction
subproject-junction.bstchooses to set the subprojectcolortobluewhen the toplevel project isfunkyThe subproject
include/paths.bstinclude file decides to set theprefixto/optin the case that the subproject isblueThe
hello.bstincludes theinclude/paths.bstfile, in order to inherit its path configuration from the subproject
3.3.3. Running the project in both modes
user@host:~/junction-includes$ bst shell hello.bst -- /usr/bin/hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][f2b2aa99][ main:hello.bst ] START Staging dependencies
[00:00:00][f2b2aa99][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][f2b2aa99][ main:hello.bst ] START Integrating sandbox
[00:00:00][f2b2aa99][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][f2b2aa99][ main:hello.bst ] STATUS Running command
/usr/bin/hello
2023-02-17T10:35:18.737+0000 [4780:140280298733568] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello World!
This is amhello 1.0.
user@host:~/junction-includes$ bst --option funky True shell hello.bst -- /opt/bin/hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][8b7e1a9b][ main:hello.bst ] START Staging dependencies
[00:00:00][8b7e1a9b][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][8b7e1a9b][ main:hello.bst ] START Integrating sandbox
[00:00:00][8b7e1a9b][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][8b7e1a9b][ main:hello.bst ] STATUS Running command
/opt/bin/hello
2023-02-17T10:35:19.636+0000 [4814:140285187866624] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello World!
This is amhello 1.0.
As expected, the funky variant of the toplevel project installs
the hello world program in the /opt prefix, and as such we
need to call it from there.
3.4. Summary
In this chapter we’ve discussed how conditional statements and include files play an essential role in the API surface of a project, and help to provide some configurability while preserving encapsulation of the API which a project exposes.
We’ve also gone over the mechanics of how these concepts interact and presented an example which shows how project options can be used in a recursive context, and how includes can help not only to share code, but to provide context to dependent projects about how their subprojects are configured.