3. Subproject includes

We’ve already discussed how we can add optionality to projects and explored how we can perform conditional statements and include fragments of BuildStream YAML in the earlier chapter about optionality and directives.

In this chapter we’re going to explore how we can use include directives to include YAML fragments from a subproject referred to by a junction element, and how project options can be specified in the configuration of your junction.

Note

This example is distributed with BuildStream in the doc/examples/junction-includes subdirectory.

3.1. Overview

It is a goal of BuildStream to provide developers and integrators with the tools they need to maintain software stacks which depend on eachother with least friction as possible, such that one can integrate upgrades of projects one depends on via junction elements regularly and with the least hassle as possible.

Project options and include directives combined form the basis on which projects can maximize on code sharing effectively, and the basis on which BuildStream projects can form reliable APIs.

3.1.1. Project options

The options which a project exposes is a fairly limited API surface, it allows one to configure a limited set of options advertized by the project maintainers, and the options will affect what kind of artifacts will be produced by the project.

This kind of optionality however does not allow consumers to entirely redefine how artifacts are produced and how elements are configured.

On the one hand, this limitation can be frustrating, as one constantly finds themselves requiring a feature that their subproject does not support right now. On the other hand, the limitation of features which a given project chooses to support is what guards downstream project consumers against consuming artifacts which are not supported by the upstream.

Project options are designed to enforce a separation of concerns, where we expect that downstreams will either fork a project in order to support a new feature, or convince the upstream to start supporting a new feature. Furthermore, limited API surfaces for interdependent projects offers a possibility of API stability of projects, such that you can upgrade your dependencies with limited friction.

3.1.2. Includes

The includes which a project might advertize as “public”, form the output of the API exchange between a project and its subproject(s).

Cross-project include files allow a project to inherit configuration from a subproject. Include files can be used to define anything from the variables one needs to have in context in order to build into or link into alternative system prefixes, what special compiler flags to use when building for a specific machine architecture, to customized shell configurations to use when testing out applications in bst shell.

This chapter will provide an example of the mechanics of cross project includes when combined with project optionality.

3.2. Project structure

3.2.1. Project options

This example is comprised of two separate projects, both of which offer some project options. This is intended to emphasize how your toplevel project options can be used to select and configure options to use in the subprojects you depend on.

For convenience, the subproject is stored in the subdirectory of the toplevel project, while in the real world the subproject is probably hosted elsewhere.

First let’s take a look at the options declared in the respective project.conf files.

3.2.1.1. Toplevel project.conf

name: project
min-version: 2.0
element-path: elements

aliases:
  gnu: http://ftpmirror.gnu.org/gnu/automake/

plugins:
- origin: pip
  package-name: buildstream-plugins
  elements:
  - autotools

# Define some options for this project
#
options:
  funky:
    type: bool
    description: Whether this project is funky
    default: False

3.2.1.2. Subproject project.conf

name: subproject
min-version: 2.0
element-path: elements

aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

# Define some options for this project
#
options:
  color:
    type: enum
    description: The color of this runtime
    values:
    - red
    - green
    - blue
    default: blue

As we can see, these two projects both offer some arbitrarily named options.

3.2.2. Conditional configuration of subproject

The toplevel project here does some conditional configuration of the subproject.

3.2.2.1. Toplevel elements/subproject-junction.bst

kind: junction

config:
  # Configure the options for subproject
  #
  # If our project is funky, then it requires
  # a blue subproject, otherwise we use a red one.
  #
  options:
    color: red
    (?):
    - funky == True:
        color: blue

sources:
- kind: local
  path: subproject

Here we can see that projects can use conditional statements to make decisions about subproject configuration based on their own configuration.

In this example, if the toplevel project is funky, then it will configure its subproject with color set to blue, otherwise it will use the red variant of the subproject color.

3.2.3. Including configuration from a subproject

Here there are a couple of aspects to observe, namely how the toplevel project includes files across a junction boundary, and how that include file might be implemented.

3.2.3.1. Toplevel elements/hello.bst

kind: autotools
description: |

    Hello world example from automake

variables:

  # The special paths.bst from our subproject is used to
  # define the paths of some elements in this project.
  #
  (@): subproject-junction.bst:include/paths.bst

  # The hello world example lives in the doc/amhello folder.
  command-subdir: doc/amhello

sources:
- kind: tar
  url: gnu:automake-1.16.tar.gz
  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f

depends:
  - filename: base.bst
    junction: subproject-junction.bst

Here we can see the same element which we discussed in the autotools example, except that we’re including a file from the subproject. As explained in the reference manual, this is done by prefixing the include path with the local junction element name and then a colon.

Note that in this case, the API contract is simply that hello.bst is including paths.bst, and has the expectation that paths.bst will in some way influence the variables, nothing more.

It can be that an include file is expected to create new variables, and it can be that the subproject might declare things differently depending on the subproject’s own configuration, as we will observe next.

3.2.3.2. Subproject include/paths.bst

# When this project color is blue, including this
# file causes installations to be relocated to /opt
#
prefix: /usr
(?):
- color == "blue":
    prefix: /opt

Here, we can see the include file itself is making a conditional statement, in turn deciding what values to use depending on how the project was configured.

This decision will provide valuable context for any file including paths.bst, whether it be an element, a project.conf which applies the variable as a default for the entire project, whether it is being included by files in the local project, or whether it is being included by a downstream project which junctions this project, as is the case in this example.

3.3. Using the project

At this stage, you have probably already reasoned out what would happen if we tried to build and run the project.

Nevertheless, we will still present the outputs here for observation.

3.3.1. Building the project normally

Here we build the project without any special arguments.

user@host:~/junction-includes$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732103.187746    3183 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.3.0
    Session Start: Thursday, 15-08-2024 at 14:28:23
    Project:       project (/home/user/buildstream/doc/examples/junction-includes)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-grwknr7r/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-grwknr7r
    Log Files:               /home/user/buildstream/doc/run-bst-grwknr7r/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-grwknr7r/sources
    Build Area:              /home/user/buildstream/doc/run-bst-grwknr7r/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: project

    Project Options
        funky: 0

    Element Plugins
        junction:  core plugin
        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages

    Source Plugins
        local: core plugin
        tar:   core plugin

Project: subproject
    Junction path: subproject-junction.bst
    Loaded by:     hello.bst [line 11 column 7]

    Project Options
        color: red

    Element Plugins
        stack:  core plugin
        import: core plugin

    Source Plugins
        tar: core plugin

Pipeline
fetch needed 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst 
     waiting c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst 
     waiting 75ba510d9aa73503f87fd9bc44db651a2cf7aca8c67f13d686abbe0b35dced0e hello.bst 
===============================================================================
[--:--:--][c93bc58b][   fetch:subproject-junction.bst:base.bst] START   subproject/base/c93bc58b-fetch.20240815-142823.log
[--:--:--][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] START   subproject/base-alpine/2130295a-fetch.20240815-142823.log
[--:--:--][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:00][c93bc58b][   fetch:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-fetch.20240815-142823.log
[00:00:00][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][2130295a][   fetch:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-fetch.20240815-142823.log
[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   subproject/base-alpine/2130295a-build.20240815-142829.log
[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   Staging sources
[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Staging sources
[--:--:--][2130295a][   build:subproject-junction.bst:base/alpine.bst] START   Caching artifact
[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Caching artifact
[00:00:00][2130295a][   build:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/2130295a-build.20240815-142829.log
[--:--:--][c93bc58b][   build:subproject-junction.bst:base.bst] START   subproject/base/c93bc58b-build.20240815-142829.log
[--:--:--][c93bc58b][   build:subproject-junction.bst:base.bst] START   Caching artifact
[00:00:00][c93bc58b][   build:subproject-junction.bst:base.bst] SUCCESS Caching artifact
[00:00:00][c93bc58b][   build:subproject-junction.bst:base.bst] SUCCESS subproject/base/c93bc58b-build.20240815-142829.log
[--:--:--][75ba510d][   build:hello.bst                     ] START   project/hello/75ba510d-build.20240815-142829.log
[--:--:--][75ba510d][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][75ba510d][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][75ba510d][   build:hello.bst                     ] START   Staging sources
[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][75ba510d][   build:hello.bst                     ] START   Running commands

    export NOCONFIGURE=1;
    
    if [ -x ./configure ]; then true;
    elif [ -x ./autogen ]; then ./autogen;
    elif [ -x ./autogen.sh ]; then ./autogen.sh;
    elif [ -x ./bootstrap ]; then ./bootstrap;
    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
    else autoreconf -ivf .;
    fi
    ./configure --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --includedir=/usr/include \
    --libdir=/usr/lib \
    --libexecdir=/usr/libexec \
    --localstatedir=/var \
    --sharedstatedir=/usr/com \
    Message contains 23 additional lines

[00:00:03][75ba510d][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][75ba510d][   build:hello.bst                     ] START   Caching artifact
[00:00:00][75ba510d][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:03][75ba510d][   build:hello.bst                     ] SUCCESS project/hello/75ba510d-build.20240815-142829.log
[00:00:10][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 2, skipped 1, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

3.3.2. Building the project in funky mode

Now let’s see what happens when we build the project in funky mode

user@host:~/junction-includes$ bst --option funky True build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732114.238509    4196 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.3.0
    Session Start: Thursday, 15-08-2024 at 14:28:34
    Project:       project (/home/user/buildstream/doc/examples/junction-includes)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-grwknr7r/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-grwknr7r
    Log Files:               /home/user/buildstream/doc/run-bst-grwknr7r/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-grwknr7r/sources
    Build Area:              /home/user/buildstream/doc/run-bst-grwknr7r/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: project

    Project Options
        funky: 1

    Element Plugins
        junction:  core plugin
        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages

    Source Plugins
        local: core plugin
        tar:   core plugin

Project: subproject
    Junction path: subproject-junction.bst
    Loaded by:     hello.bst [line 11 column 7]

    Project Options
        color: blue

    Element Plugins
        stack:  core plugin
        import: core plugin

    Source Plugins
        tar: core plugin

Pipeline
      cached 2130295a58bedd83aaf3998edb121ccd5942cd749f14b87c19a4be330c90d54f subproject-junction.bst:base/alpine.bst 
      cached c93bc58b2fe8afba05ea49c1a941775a15442a56ea826dee5c017641de7c469c subproject-junction.bst:base.bst 
   buildable df66a655e4245f317fd4f2784982198759448aea207966923e5d27c3d41082b0 hello.bst 
===============================================================================
[--:--:--][df66a655][   build:hello.bst                     ] START   project/hello/df66a655-build.20240815-142834.log
[--:--:--][df66a655][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][df66a655][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][df66a655][   build:hello.bst                     ] START   Staging sources
[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][df66a655][   build:hello.bst                     ] START   Running commands

    export NOCONFIGURE=1;
    
    if [ -x ./configure ]; then true;
    elif [ -x ./autogen ]; then ./autogen;
    elif [ -x ./autogen.sh ]; then ./autogen.sh;
    elif [ -x ./bootstrap ]; then ./bootstrap;
    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
    else autoreconf -ivf .;
    fi
    ./configure --prefix=/opt \
    --exec-prefix=/opt \
    --bindir=/opt/bin \
    --sbindir=/opt/sbin \
    --sysconfdir=/etc \
    --datadir=/opt/share \
    --includedir=/opt/include \
    --libdir=/opt/lib \
    --libexecdir=/opt/libexec \
    --localstatedir=/var \
    --sharedstatedir=/opt/com \
    Message contains 23 additional lines

[00:00:03][df66a655][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][df66a655][   build:hello.bst                     ] START   Caching artifact
[00:00:00][df66a655][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:03][df66a655][   build:hello.bst                     ] SUCCESS project/hello/df66a655-build.20240815-142834.log
[00:00:04][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 0, skipped 3, failed 0 
    Build Queue: processed 1, skipped 2, failed 0

As we can see, this time we’ve built the project into the /opt system prefix instead of the standard /usr prefix.

Let’s just take a step back now and summarize the process which went into this decision:

  • The toplevel project.conf exposes the boolean funky option

  • The toplevel junction subproject-junction.bst chooses to set the subproject color to blue when the toplevel project is funky

  • The subproject include/paths.bst include file decides to set the prefix to /opt in the case that the subproject is blue

  • The hello.bst includes the include/paths.bst file, in order to inherit its path configuration from the subproject

3.3.3. Running the project in both modes

user@host:~/junction-includes$ bst shell hello.bst -- /usr/bin/hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732118.653583    5206 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][75ba510d][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][75ba510d][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][75ba510d][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][75ba510d][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][75ba510d][    main:hello.bst                     ] STATUS  Running command

    /usr/bin/hello

Hello World!
This is amhello 1.0.

user@host:~/junction-includes$ bst --option funky True shell hello.bst -- /opt/bin/hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732119.262526    5250 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][df66a655][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][df66a655][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][df66a655][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][df66a655][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][df66a655][    main:hello.bst                     ] STATUS  Running command

    /opt/bin/hello

Hello World!
This is amhello 1.0.

As expected, the funky variant of the toplevel project installs the hello world program in the /opt prefix, and as such we need to call it from there.

3.4. Summary

In this chapter we’ve discussed how conditional statements and include files play an essential role in the API surface of a project, and help to provide some configurability while preserving encapsulation of the API which a project exposes.

We’ve also gone over the mechanics of how these concepts interact and presented an example which shows how project options can be used in a recursive context, and how includes can help not only to share code, but to provide context to dependent projects about how their subprojects are configured.