2. Running commands

In the first chapter we only imported a file to create an artifact, this time lets run some commands inside the isolated build sandbox.

Note

This example is distributed with BuildStream in the doc/examples/running-commands subdirectory.

2.1. Overview

In this chapter, we’ll be running commands inside the sandboxed execution environment and producing build output.

We’ll be compiling the following simple C file:

2.1.1. files/src/hello.c

/*
 * hello.c - Simple hello world program
 */
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("Hello World\n");
  return 0;
}

And we’re going to build it using make, using the following Makefile:

2.1.2. files/src/Makefile

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -m 755 hello ${DESTDIR}${PREFIX}/bin

hello: hello.c
	$(CC) -Wall -o $@ $<

We’ll be using the most fundamental build element, the manual build element.

The manual element is the backbone on which all the other build elements are built, so understanding how it works at this level is helpful.

2.2. Project structure

In this project we have a project.conf, a directory with some source code, and 3 element declarations.

Let’s first take a peek at what we need to build using bst show:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
     waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
     waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 

This time we have loaded a pipeline with 3 elements, let’s go over what they do in detail.

2.2.1. project.conf

# Unique project name
name: running-commands

# Minimum required BuildStream version
min-version: 2.0

# Subdirectory where elements are stored
element-path: elements

# Define an alias for our alpine tarball
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

Our project.conf is very much like the last one, except that we have defined a source alias for alpine.

Tip

Using source aliases for groups of sources which are generally hosted together is encouraged. This allows one to globally change the access scheme or URL for a group of repositories which belong together.

2.2.2. elements/base/alpine.bst

kind: import
description: |

    Alpine Linux base runtime

sources:
- kind: tar

  # This is a post doctored, trimmed down system image
  # of the Alpine linux distribution.
  #
  url: alpine:integration-tests-base.v1.x86_64.tar.xz
  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

This import element uses a tar source to download our Alpine Linux tarball to create our base runtime.

This tarball is a sysroot which provides the C runtime libraries and some programs - this is what will be providing the programs we’re going to run in this example.

2.2.3. elements/base.bst

kind: stack
description: Base stack

depends:
- base/alpine.bst

This is just a symbolic stack element which declares that anything which depends on it, will implicitly depend on base/alpine.bst.

It is typical to use stack elements in places where the implementing logical software stack could change, but you rather not have your higher level components carry knowledge about those changing components.

Any element which runtime depends on the base.bst will now be able to execute programs provided by the imported base/alpine.bst runtime.

2.2.4. elements/hello.bst

kind: manual
description: |

  Building manually

# Depend on the base system
depends:
- base.bst

# Stage the files/src directory for building
sources:
  - kind: local
    path: files/src

# Now configure the commands to run
config:

  build-commands:
  - make PREFIX="%{prefix}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

Finally we have the element which executes commands. Looking at the manual element’s documentation, we can see that the element configuration exposes four command lists:

• configure-commands

  Commands which are run in preparation of a build. This is where you would normally call any configure stage build tools to configure the build how you like and generate some files needed for the build.

• build-commands

  Commands to run the build, usually a build system will invoke the compiler for you here.

• install-commands

  Commands to install the build results.

  Commands to install the build results into the target system, these should install files somewhere under %{install-root}.

• strip-commands

  Commands to doctor the build results after the install.

  Typically this involves stripping binaries of debugging symbols or stripping timestamps from build results to ensure reproducibility.

Tip

All other build elements implement exactly the same command lists too, except that they provide default commands specific to invoke the build systems they support.

The manual element however is the most basic and does not provide any default commands, so we have instructed it to use make to build and install our program.

2.3. Using the project

2.3.1. Build the hello.bst element

To build the project, run bst build in the following way:

user@host:~/running-commands$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.0.1
    Session Start: Friday, 17-02-2023 at 10:36:00
    Project:       running-commands (/home/user/buildstream/doc/examples/running-commands)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-t8ybwht7/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-t8ybwht7
    Log Files:               /home/user/buildstream/doc/run-bst-t8ybwht7/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-t8ybwht7/sources
    Build Area:              /home/user/buildstream/doc/run-bst-t8ybwht7/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: running-commands

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
fetch needed b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
     waiting bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
     waiting 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 
===============================================================================
[--:--:--][b59f4520][   fetch:base/alpine.bst               ] START   running-commands/base-alpine/b59f4520-fetch.6160.log
[--:--:--][b59f4520][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][bb393e5a][   fetch:base.bst                      ] START   running-commands/base/bb393e5a-fetch.6160.log
[00:00:00][bb393e5a][   fetch:base.bst                      ] SUCCESS running-commands/base/bb393e5a-fetch.6160.log
[00:00:00][b59f4520][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:10][b59f4520][   fetch:base/alpine.bst               ] SUCCESS running-commands/base-alpine/b59f4520-fetch.6160.log
[--:--:--][b59f4520][   build:base/alpine.bst               ] START   running-commands/base-alpine/b59f4520-build.6160.log
[--:--:--][b59f4520][   build:base/alpine.bst               ] START   Staging sources
[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][b59f4520][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][b59f4520][   build:base/alpine.bst               ] SUCCESS running-commands/base-alpine/b59f4520-build.6160.log
[--:--:--][bb393e5a][   build:base.bst                      ] START   running-commands/base/bb393e5a-build.6160.log
[--:--:--][bb393e5a][   build:base.bst                      ] START   Caching artifact
[00:00:00][bb393e5a][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][bb393e5a][   build:base.bst                      ] SUCCESS running-commands/base/bb393e5a-build.6160.log
[--:--:--][8cebf405][   build:hello.bst                     ] START   running-commands/hello/8cebf405-build.6160.log
[--:--:--][8cebf405][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][8cebf405][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][8cebf405][   build:hello.bst                     ] START   Staging sources
[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][8cebf405][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][8cebf405][   build:hello.bst                     ] START   Caching artifact
[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][8cebf405][   build:hello.bst                     ] SUCCESS running-commands/hello/8cebf405-build.6160.log
[00:00:11][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 2, skipped 1, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

Now we’ve built our hello world program, using make and the C compiler provided by the Alpine Linux image.

In the first chapter we observed that the inputs and output of an element are directory trees. In this example, the directory tree generated by base/alpine.bst is consumed by hello.bst due to the implicit runtime dependency introduced by base.bst.

Tip

All of the dependencies which are required to run for the sake of a build, are staged at the root of the build sandbox. These comprise the runtime environment in which the depending element will run commands.

The result is that the make program and C compiler provided by base/alpine.bst were already in $PATH and ready to run when the commands were needed by hello.bst.

Now observe that all of the elements in the loaded pipeline are cached, the element is built:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
      cached b59f45208e6fb265550fb3f14180167e453f175f4f769b912d415f9fd66d76af base/alpine.bst 
      cached bb393e5a37b8eab166c9f76e829daaa3b9a17a4d24c55db1c1c57109d7ff431c base.bst 
      cached 8cebf405325640bbe2f5e824b558cd67b7a4037be13d8e6c3065ad7bca4f3e07 hello.bst 

2.3.2. Run the hello world program

Now that we’ve built everything, we can indulge ourselves in running the hello world program using bst shell:

user@host:~/running-commands$ bst shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][8cebf405][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][8cebf405][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][8cebf405][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][8cebf405][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][8cebf405][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:36:12.627+0000 [6256:140588767649792] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello World

Here, bst shell created a runtime environment for running the hello.bst element. This was done by staging all of the dependencies of hello.bst including the hello.bst output itself into a directory. Once a directory with all of the dependencies was staged and ready, we ran the hello command from within the build sandbox environment.

Tip

When specifying a command for bst shell to run, we always specify -- first. This is a commonly understood shell syntax to indicate that the remaining arguments are to be treated literally.

Specifying -- is optional and disambiguates BuildStream’s arguments and options from those of the program being run by bst shell.

2.4. Summary

In this chapter we’ve explored how to use the manual element, which forms the basis of all build elements.

We’ve also observed how the directory tree from the output artifact of one element is later staged at the root of the sandbox, as input for use by any build elements which depend on that element.

Tip

The way that elements consume their dependency input can vary across the different kinds of elements. This chapter describes how it works for build elements implementations, which are the most commonly used element type.