2. Running commands

In the first chapter we only imported a file to create an artifact, this time lets run some commands inside the isolated build sandbox.

Note

This example is distributed with BuildStream in the doc/examples/running-commands subdirectory.

2.1. Overview

In this chapter, we’ll be running commands inside the sandboxed execution environment and producing build output.

We’ll be compiling the following simple C file:

2.1.1. files/src/hello.c

/*
 * hello.c - Simple hello world program
 */
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("Hello World\n");
  return 0;
}

And we’re going to build it using make, using the following Makefile:

2.1.2. files/src/Makefile

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -m 755 hello ${DESTDIR}${PREFIX}/bin

hello: hello.c
	$(CC) -Wall -o $@ $<

We’ll be using the most fundamental build element, the manual build element.

The manual element is the backbone on which all the other build elements are built, so understanding how it works at this level is helpful.

2.2. Project structure

In this project we have a project.conf, a directory with some source code, and 3 element declarations.

Let’s first take a peek at what we need to build using bst show:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][][] STATUS  Cache usage recomputed: 12K / infinity (0%)
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
   buildable d5472eb60ceb8a45b0ed5907912b5d89ce91d8a4e9b454ea2663f052c6cf4dff base/alpine.bst 
     waiting 50110d08831106135148d991c33f15794b530938712216e84593eeff7c1ef8ca base.bst 
     waiting 2bee306eafed3b07aa9dae4c28185429984ca7b246585901709e10ba7ca52c8e hello.bst 

This time we have loaded a pipeline with 3 elements, let’s go over what they do in detail.

2.2.1. project.conf

# Unique project name
name: running-commands

# Required BuildStream format version
format-version: 9

# Subdirectory where elements are stored
element-path: elements

# Define an alias for our alpine tarball
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

Our project.conf is very much like the last one, except that we have defined a source alias for alpine.

Tip

Using source aliases for groups of sources which are generally hosted together is encouraged. This allows one to globally change the access scheme or URL for a group of repositories which belong together.

2.2.2. elements/base/alpine.bst

kind: import
description: |

    Alpine Linux base runtime

sources:
- kind: tar

  # This is a post doctored, trimmed down system image
  # of the Alpine linux distribution.
  #
  url: alpine:integration-tests-base.v1.x86_64.tar.xz
  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

This import element uses a tar source to download our Alpine Linux tarball to create our base runtime.

This tarball is a sysroot which provides the C runtime libraries and some programs - this is what will be providing the programs we’re going to run in this example.

2.2.3. elements/base.bst

kind: stack
description: Base stack

depends:
- base/alpine.bst

This is just a symbolic stack element which declares that anything which depends on it, will implicitly depend on base/alpine.bst.

It is typical to use stack elements in places where the implementing logical software stack could change, but you rather not have your higher level components carry knowledge about those changing components.

Any element which runtime depends on the base.bst will now be able to execute programs provided by the imported base/alpine.bst runtime.

2.2.4. elements/hello.bst

kind: manual
description: |

  Building manually

# Depend on the base system
depends:
- base.bst

# Stage the files/src directory for building
sources:
  - kind: local
    path: files/src

# Now configure the commands to run
config:

  build-commands:
  - make PREFIX="%{prefix}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

Finally we have the element which executes commands. Looking at the manual element’s documentation, we can see that the element configuration exposes four command lists:

• configure-commands

  Commands which are run in preparation of a build. This is where you would normally call any configure stage build tools to configure the build how you like and generate some files needed for the build.

• build-commands

  Commands to run the build, usually a build system will invoke the compiler for you here.

• install-commands

  Commands to install the build results.

  Commands to install the build results into the target system, these should install files somewhere under %{install-root}.

• strip-commands

  Commands to doctor the build results after the install.

  Typically this involves stripping binaries of debugging symbols or stripping timestamps from build results to ensure reproducibility.

Tip

All other build elements implement exactly the same command lists too, except that they provide default commands specific to invoke the build systems they support.

The manual element however is the most basic and does not provide any default commands, so we have instructed it to use make to build and install our program.

2.3. Using the project

2.3.1. Build the hello.bst element

To build the project, run bst build in the following way:

user@host:~/running-commands$ bst build hello.bst

[--:--:--][][] STATUS  Cache usage recomputed: 12K / infinity (0%)
[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START   Checking sources
[00:00:00][][] SUCCESS Checking sources

BuildStream Version 1.4.2+20.g89765b75
  Session Start: Wednesday, 13-05-2020 at 10:39:03
  Project:       running-commands (/home/user/running-commands)
  Targets:       hello.bst
  Cache Usage:   12K / infinity (0%)

User Configuration
  Configuration File:      /home/user/.config/buildstream.conf
  Log Files:               /home/user/.cache/buildstream/logs
  Source Mirrors:          /home/user/.cache/buildstream/sources
  Build Area:              /home/user/.cache/buildstream/build
  Artifact Cache:          /home/user/.cache/buildstream/artifacts
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Pipeline
   buildable d5472eb60ceb8a45b0ed5907912b5d89ce91d8a4e9b454ea2663f052c6cf4dff base/alpine.bst 
     waiting 50110d08831106135148d991c33f15794b530938712216e84593eeff7c1ef8ca base.bst 
     waiting 2bee306eafed3b07aa9dae4c28185429984ca7b246585901709e10ba7ca52c8e hello.bst 
===============================================================================
[--:--:--][d5472eb6][build:base/alpine.bst               ] START   running-commands/base-alpine/d5472eb6-build.574.log
[--:--:--][d5472eb6][build:base/alpine.bst               ] START   Staging sources
[00:00:10][d5472eb6][build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][d5472eb6][build:base/alpine.bst               ] START   Caching artifact
[00:00:05][d5472eb6][build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:17][d5472eb6][build:base/alpine.bst               ] SUCCESS running-commands/base-alpine/d5472eb6-build.574.log
[--:--:--][50110d08][build:base.bst                      ] START   running-commands/base/50110d08-build.576.log
[--:--:--][50110d08][build:base.bst                      ] START   Caching artifact
[00:00:00][50110d08][build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][50110d08][build:base.bst                      ] SUCCESS running-commands/base/50110d08-build.576.log
[--:--:--][2bee306e][build:hello.bst                     ] START   running-commands/hello/2bee306e-build.578.log
[--:--:--][2bee306e][build:hello.bst                     ] START   Staging dependencies
[00:00:01][2bee306e][build:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][2bee306e][build:hello.bst                     ] START   Integrating sandbox
[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][2bee306e][build:hello.bst                     ] START   Staging sources
[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][2bee306e][build:hello.bst                     ] START   Running build-commands
[--:--:--][2bee306e][build:hello.bst                     ] STATUS  Running build-commands

    make PREFIX="/usr"

[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Running build-commands
[--:--:--][2bee306e][build:hello.bst                     ] START   Running install-commands
[--:--:--][2bee306e][build:hello.bst                     ] STATUS  Running install-commands

    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Running install-commands
[--:--:--][2bee306e][build:hello.bst                     ] START   Running strip-commands
[--:--:--][2bee306e][build:hello.bst                     ] STATUS  Running strip-commands

    cd "/buildstream-install" && find -type f \
      '(' -perm -111 -o -name '*.so*' \
          -o -name '*.cmxs' -o -name '*.node' ')' \
      -exec sh -ec \
      'read -n4 hdr <"$1" # check for elf header
       if [ "$hdr" != "$(printf \\x7fELF)" ]; then
           exit 0
       fi
       debugfile="/buildstream-install/usr/lib/debug/$1"
       mkdir -p "$(dirname "$debugfile")"
       objcopy --only-keep-debug --compress-debug-sections "$1" "$debugfile"
       chmod 644 "$debugfile"
       strip --remove-section=.comment --remove-section=.note --strip-unneeded "$1"
       objcopy --add-gnu-debuglink "$debugfile" "$1"' - {} ';'

[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Running strip-commands
[--:--:--][2bee306e][build:hello.bst                     ] START   Caching artifact
[00:00:00][2bee306e][build:hello.bst                     ] SUCCESS Caching artifact
[00:00:01][2bee306e][build:hello.bst                     ] SUCCESS running-commands/hello/2bee306e-build.578.log
[00:00:18][][] SUCCESS Build

Pipeline Summary
  Total:       3
  Session:     3
  Fetch Queue: processed 0, skipped 3, failed 0 
  Build Queue: processed 3, skipped 0, failed 0

Now we’ve built our hello world program, using make and the C compiler provided by the Alpine Linux image.

In the first chapter we observed that the inputs and output of an element are directory trees. In this example, the directory tree generated by base/alpine.bst is consumed by hello.bst due to the implicit runtime dependency introduced by base.bst.

Tip

All of the dependencies which are required to run for the sake of a build, are staged at the root of the build sandbox. These comprise the runtime environment in which the depending element will run commands.

The result is that the make program and C compiler provided by base/alpine.bst were already in $PATH and ready to run when the commands were needed by hello.bst.

Now observe that all of the elements in the loaded pipeline are cached, the element is built:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
      cached d5472eb60ceb8a45b0ed5907912b5d89ce91d8a4e9b454ea2663f052c6cf4dff base/alpine.bst 
      cached 50110d08831106135148d991c33f15794b530938712216e84593eeff7c1ef8ca base.bst 
      cached 2bee306eafed3b07aa9dae4c28185429984ca7b246585901709e10ba7ca52c8e hello.bst 

2.3.2. Run the hello world program

Now that we’ve built everything, we can indulge ourselves in running the hello world program using bst shell:

user@host:~/running-commands$ bst shell hello.bst -- hello

[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][2bee306e][ main:hello.bst                     ] START   Staging dependencies
[00:00:00][2bee306e][ main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][2bee306e][ main:hello.bst                     ] START   Integrating sandbox
[00:00:00][2bee306e][ main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][2bee306e][ main:hello.bst                     ] STATUS  Running command

    hello

Hello World

Here, bst shell created a runtime environment for running the hello.bst element. This was done by staging all of the dependencies of hello.bst including the hello.bst output itself into a directory. Once a directory with all of the dependencies was staged and ready, we ran the hello command from within the build sandbox environment.

Tip

When specifying a command for bst shell to run, we always specify -- first. This is a commonly understood shell syntax to indicate that the remaining arguments are to be treated literally.

Specifying -- is optional and disambiguates BuildStream’s arguments and options from those of the program being run by bst shell.

2.4. Summary

In this chapter we’ve explored how to use the manual element, which forms the basis of all build elements.

We’ve also observed how the directory tree from the output artifact of one element is later staged at the root of the sandbox, as input for use by any build elements which depend on that element.

Tip

The way that elements consume their dependency input can vary across the different kinds of elements. This chapter describes how it works for build elements implementations, which are the most commonly used element type.