2. Running commands
In the first chapter we only imported a file to create an artifact, this time lets run some commands inside the isolated build sandbox.
Note
This example is distributed with BuildStream in the doc/examples/running-commands subdirectory.
2.1. Overview
In this chapter, we’ll be running commands inside the sandboxed execution environment and producing build output.
We’ll be compiling the following simple C file:
2.1.1. files/src/hello.c
/*
* hello.c - Simple hello world program
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("Hello World\n");
return 0;
}
And we’re going to build it using make, using the following Makefile:
2.1.2. files/src/Makefile
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $<
We’ll be using the most fundamental build element,
the manual build element.
The manual element is the backbone on which all the other
build elements are built, so understanding how it works at this level is helpful.
2.2. Project structure
In this project we have a project.conf, a directory with some source
code, and 3 element declarations.
Let’s first take a peek at what we need to build using bst show:
user@host:~/running-commands$ bst show hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888261.507356 6831 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst
waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst
waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst
This time we have loaded a pipeline with 3 elements, let’s go over what they do in detail.
2.2.1. project.conf
# Unique project name
name: running-commands
# Minimum required BuildStream version
min-version: 2.0
# Subdirectory where elements are stored
element-path: elements
# Define an alias for our alpine tarball
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
Our project.conf is very much like the last one, except that we
have defined a source alias for alpine.
Tip
Using source aliases for groups of sources which are generally hosted together is encouraged. This allows one to globally change the access scheme or URL for a group of repositories which belong together.
2.2.2. elements/base/alpine.bst
kind: import
description: |
Alpine Linux base runtime
sources:
- kind: tar
# This is a post doctored, trimmed down system image
# of the Alpine linux distribution.
#
url: alpine:integration-tests-base.v1.x86_64.tar.xz
ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
This import element uses a tar
source to download our Alpine Linux tarball to create our base runtime.
This tarball is a sysroot which provides the C runtime libraries and some programs - this is what will be providing the programs we’re going to run in this example.
2.2.3. elements/base.bst
kind: stack
description: Base stack
depends:
- base/alpine.bst
This is just a symbolic stack element which declares that
anything which depends on it, will implicitly depend on base/alpine.bst.
It is typical to use stack elements in places where the implementing logical software stack could change, but you rather not have your higher level components carry knowledge about those changing components.
Any element which runtime depends on
the base.bst will now be able to execute programs provided by the imported
base/alpine.bst runtime.
2.2.4. elements/hello.bst
kind: manual
description: |
Building manually
# Depend on the base system
depends:
- base.bst
# Stage the files/src directory for building
sources:
- kind: local
path: files/src
# Now configure the commands to run
config:
build-commands:
- make PREFIX="%{prefix}"
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Finally we have the element which executes commands. Looking at the
manual element’s documentation, we can see that
the element configuration exposes four command lists:
configure-commandsCommands which are run in preparation of a build. This is where you would normally call any configure stage build tools to configure the build how you like and generate some files needed for the build.
build-commandsCommands to run the build, usually a build system will invoke the compiler for you here.
install-commandsCommands to install the build results.
Commands to install the build results into the target system, these should install files somewhere under
%{install-root}.strip-commandsCommands to doctor the build results after the install.
Typically this involves stripping binaries of debugging symbols or stripping timestamps from build results to ensure reproducibility.
Tip
All other build elements implement exactly the same command lists too, except that they provide default commands specific to invoke the build systems they support.
The manual element however is the most basic
and does not provide any default commands, so we have instructed it
to use make to build and install our program.
2.3. Using the project
2.3.1. Build the hello.bst element
To build the project, run bst build in the following way:
user@host:~/running-commands$ bst build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888261.919086 6859 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0+26.g65e2074ad
Session Start: Wednesday, 06-11-2024 at 10:17:41
Project: running-commands (/home/user/buildstream/doc/examples/running-commands)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-drp0a3w0/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-drp0a3w0
Log Files: /home/user/buildstream/doc/run-bst-drp0a3w0/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-drp0a3w0/sources
Build Area: /home/user/buildstream/doc/run-bst-drp0a3w0/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: running-commands
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst
waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst
waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst
===============================================================================
[--:--:--][b59f4520][ fetch:base/alpine.bst ] START running-commands/base-alpine/b59f4520-fetch.20241106-101741.log
[--:--:--][b59f4520][ fetch:base/alpine.bst ] START Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][bb393e5a][ fetch:base.bst ] START running-commands/base/bb393e5a-fetch.20241106-101741.log
[00:00:00][bb393e5a][ fetch:base.bst ] SUCCESS running-commands/base/bb393e5a-fetch.20241106-101741.log
[00:00:00][b59f4520][ fetch:base/alpine.bst ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][b59f4520][ fetch:base/alpine.bst ] SUCCESS running-commands/base-alpine/b59f4520-fetch.20241106-101741.log
[--:--:--][b59f4520][ build:base/alpine.bst ] START running-commands/base-alpine/b59f4520-build.20241106-101748.log
[--:--:--][b59f4520][ build:base/alpine.bst ] START Staging sources
[00:00:00][b59f4520][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][b59f4520][ build:base/alpine.bst ] START Caching artifact
[00:00:00][b59f4520][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][b59f4520][ build:base/alpine.bst ] SUCCESS running-commands/base-alpine/b59f4520-build.20241106-101748.log
[--:--:--][bb393e5a][ build:base.bst ] START running-commands/base/bb393e5a-build.20241106-101748.log
[--:--:--][bb393e5a][ build:base.bst ] START Caching artifact
[00:00:00][bb393e5a][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][bb393e5a][ build:base.bst ] SUCCESS running-commands/base/bb393e5a-build.20241106-101748.log
[--:--:--][8cebf405][ build:hello.bst ] START running-commands/hello/8cebf405-build.20241106-101748.log
[--:--:--][8cebf405][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][8cebf405][ build:hello.bst ] START Integrating sandbox
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][8cebf405][ build:hello.bst ] START Staging sources
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][8cebf405][ build:hello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS Running commands
[--:--:--][8cebf405][ build:hello.bst ] START Caching artifact
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][8cebf405][ build:hello.bst ] SUCCESS running-commands/hello/8cebf405-build.20241106-101748.log
[00:00:06][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 2, skipped 1, failed 0
Build Queue: processed 3, skipped 0, failed 0
Now we’ve built our hello world program, using make
and the C compiler provided by the Alpine Linux image.
In the first chapter we observed that the inputs
and output of an element are directory trees. In this example, the directory tree
generated by base/alpine.bst is consumed by hello.bst due to the
implicit runtime dependency introduced by base.bst.
Tip
All of the dependencies which are required to run for the sake of a build, are staged at the root of the build sandbox. These comprise the runtime environment in which the depending element will run commands.
The result is that the make program and C compiler provided by base/alpine.bst
were already in $PATH and ready to run when the commands were needed by hello.bst.
Now observe that all of the elements in the loaded pipeline are cached,
the element is built:
user@host:~/running-commands$ bst show hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888269.262913 6919 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
cached b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst
cached bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst
cached 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst
2.3.2. Run the hello world program
Now that we’ve built everything, we can indulge ourselves in running the hello world program using bst shell:
user@host:~/running-commands$ bst shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888269.683298 6947 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][8cebf405][ main:hello.bst ] START Staging dependencies
[00:00:00][8cebf405][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][8cebf405][ main:hello.bst ] START Integrating sandbox
[00:00:00][8cebf405][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][8cebf405][ main:hello.bst ] STATUS Running command
hello
Hello World
Here, bst shell created a runtime environment for running
the hello.bst element. This was done by staging all of the dependencies of
hello.bst including the hello.bst output itself into a directory. Once a directory
with all of the dependencies was staged and ready, we ran the hello command from
within the build sandbox environment.
Tip
When specifying a command for bst shell to run,
we always specify -- first. This is a commonly understood shell syntax
to indicate that the remaining arguments are to be treated literally.
Specifying -- is optional and disambiguates BuildStream’s arguments
and options from those of the program being run by
bst shell.
2.4. Summary
In this chapter we’ve explored how to use the manual element,
which forms the basis of all build elements.
We’ve also observed how the directory tree from the output artifact of one element is later staged at the root of the sandbox, as input for use by any build elements which depend on that element.
Tip
The way that elements consume their dependency input can vary across the
different kinds of elements. This chapter describes how it works for
build elements implementations, which
are the most commonly used element type.