4. Integration commands¶
Sometimes a software requires more configuration or processing than what is performed at installation time, otherwise it will not run properly.
This is especially true in cases where a daemon or library interoperates with third party extensions and needs to maintain a system wide cache whenever it’s extensions are installed or removed; system wide font caches are an example of this.
In these cases we use integration commands to ensure that a runtime is ready to run after all of it’s components have been staged.
Note
This example is distributed with BuildStream in the doc/examples/integration-commands subdirectory.
4.1. Overview¶
In this chapter, we’ll be exploring integration commands, which will be our first look at public data.
4.2. Project structure¶
4.2.1. project.conf and elements/base.bst¶
The project.conf and base stack stack element are configured in the
same way as in the previous chapter: Running commands.
4.2.2. elements/base/alpine.bst¶
kind: import
description: |
Alpine Linux base runtime
sources:
- kind: tar
url: alpine:integration-tests-base.v1.x86_64.tar.xz
ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
#
# Run ldconfig in the libdir before running anything
#
public:
bst:
integration-commands:
- ldconfig "%{libdir}"
This is the same base/alpine.bst we’ve seen in previous chapters,
except that we’ve added an integration command.
This informs BuildStream that whenever the output of this element is
expected to run, this command should be run first. In this case we
are simply running ldconfig as a precautionary measure, to ensure
that the runtime linker is ready to find any shared libraries we may
have added to %{libdir}.
4.2.2.1. Looking at public data¶
The integration commands used here is the first time we’ve used any builtin public data.
Public data is a free form portion of an element’s configuration and is not necessarily understood by the element on which it is declared, public data is intended to be read by it’s reverse dependency elements.
This allows annotations on some elements to inform elements later in the dependency chain about details of it’s artifact, or to suggest how it should be processed.
4.2.3. elements/libhello.bst and elements/hello.bst¶
These are basically manual elements very similar to the ones we’ve seen in the previous chapter: Running commands.
These produce a library and a hello program which uses the library, we will consider these irrelevant to the topic and leave examination of their sources as an exercise for the reader.
4.3. Using the project¶
4.3.1. Build the hello.bst element¶
To build the project, run bst build in the following way:
user@host:~/integration-commands$ bst build hello.bst
[--:--:--][][] STATUS Cache usage recomputed: 12K / infinity (0%)
[--:--:--][][] START Build
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START Checking sources
[00:00:00][][] SUCCESS Checking sources
BuildStream Version 1.2.8
Session Start: Tuesday, 16-07-2019 at 07:44:32
Project: integration-commands (/home/user/integration-commands)
Targets: hello.bst
Cache Usage: 12K / infinity (0%)
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Artifact Cache: /home/user/.cache/buildstream/artifacts
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
buildable 30df5cfc02f182c03e62caab989fae75b64325d09694953040cf2be679e75269 base/alpine.bst
waiting b88faca38ae2b9297d92450cba0fff8a23df268f2b856aa93b1b450e0e22ed72 base.bst
waiting 8e6d7381ffa9a1409406e3c479f0ba55231e25329dcf15cb006cf3394117c243 libhello.bst
waiting 528867598298046d60225e2c3d462f343a70d69082cbebebbaf64a088905c5bd hello.bst
===============================================================================
[--:--:--][30df5cfc][build:base/alpine.bst ] START integration-commands/base-alpine/30df5cfc-build.2861.log
[--:--:--][30df5cfc][build:base/alpine.bst ] START Staging sources
[00:00:10][30df5cfc][build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][30df5cfc][build:base/alpine.bst ] START Caching artifact
[00:00:04][30df5cfc][build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:16][30df5cfc][build:base/alpine.bst ] SUCCESS integration-commands/base-alpine/30df5cfc-build.2861.log
[--:--:--][b88faca3][build:base.bst ] START integration-commands/base/b88faca3-build.2863.log
[--:--:--][b88faca3][build:base.bst ] START Caching artifact
[00:00:00][b88faca3][build:base.bst ] SUCCESS Caching artifact
[00:00:00][b88faca3][build:base.bst ] SUCCESS integration-commands/base/b88faca3-build.2863.log
[--:--:--][8e6d7381][build:libhello.bst ] START integration-commands/libhello/8e6d7381-build.2865.log
[--:--:--][8e6d7381][build:libhello.bst ] START Staging dependencies
[00:00:01][8e6d7381][build:libhello.bst ] SUCCESS Staging dependencies
[--:--:--][8e6d7381][build:libhello.bst ] START Integrating sandbox
[--:--:--][8e6d7381][build:libhello.bst ] STATUS Running integration command
ldconfig "/usr/lib"
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Integrating sandbox
[--:--:--][8e6d7381][build:libhello.bst ] START Staging sources
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Staging sources
[--:--:--][8e6d7381][build:libhello.bst ] START Running build-commands
[--:--:--][8e6d7381][build:libhello.bst ] STATUS Running build-commands
make PREFIX="/usr"
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Running build-commands
[--:--:--][8e6d7381][build:libhello.bst ] START Running install-commands
[--:--:--][8e6d7381][build:libhello.bst ] STATUS Running install-commands
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Running install-commands
[--:--:--][8e6d7381][build:libhello.bst ] START Running strip-commands
[--:--:--][8e6d7381][build:libhello.bst ] STATUS Running strip-commands
cd "/buildstream-install" && find -type f \
'(' -perm -111 -o -name '*.so*' \
-o -name '*.cmxs' -o -name '*.node' ')' \
-exec sh -ec \
'read -n4 hdr <"$1" # check for elf header
if [ "$hdr" != "$(printf \\x7fELF)" ]; then
exit 0
fi
debugfile="/buildstream-install/usr/lib/debug/$1"
mkdir -p "$(dirname "$debugfile")"
objcopy --only-keep-debug --compress-debug-sections "$1" "$debugfile"
chmod 644 "$debugfile"
strip --remove-section=.comment --remove-section=.note --strip-unneeded "$1"
objcopy --add-gnu-debuglink "$debugfile" "$1"' - {} ';'
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Running strip-commands
[--:--:--][8e6d7381][build:libhello.bst ] START Caching artifact
[00:00:00][8e6d7381][build:libhello.bst ] SUCCESS Caching artifact
[00:00:02][8e6d7381][build:libhello.bst ] SUCCESS integration-commands/libhello/8e6d7381-build.2865.log
[--:--:--][52886759][build:hello.bst ] START integration-commands/hello/52886759-build.2900.log
[--:--:--][52886759][build:hello.bst ] START Staging dependencies
[00:00:00][52886759][build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][52886759][build:hello.bst ] START Integrating sandbox
[--:--:--][52886759][build:hello.bst ] STATUS Running integration command
ldconfig "/usr/lib"
[00:00:00][52886759][build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][52886759][build:hello.bst ] START Staging sources
[00:00:00][52886759][build:hello.bst ] SUCCESS Staging sources
[--:--:--][52886759][build:hello.bst ] START Running build-commands
[--:--:--][52886759][build:hello.bst ] STATUS Running build-commands
make PREFIX="/usr"
[00:00:00][52886759][build:hello.bst ] SUCCESS Running build-commands
[--:--:--][52886759][build:hello.bst ] START Running install-commands
[--:--:--][52886759][build:hello.bst ] STATUS Running install-commands
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][52886759][build:hello.bst ] SUCCESS Running install-commands
[--:--:--][52886759][build:hello.bst ] START Running strip-commands
[--:--:--][52886759][build:hello.bst ] STATUS Running strip-commands
cd "/buildstream-install" && find -type f \
'(' -perm -111 -o -name '*.so*' \
-o -name '*.cmxs' -o -name '*.node' ')' \
-exec sh -ec \
'read -n4 hdr <"$1" # check for elf header
if [ "$hdr" != "$(printf \\x7fELF)" ]; then
exit 0
fi
debugfile="/buildstream-install/usr/lib/debug/$1"
mkdir -p "$(dirname "$debugfile")"
objcopy --only-keep-debug --compress-debug-sections "$1" "$debugfile"
chmod 644 "$debugfile"
strip --remove-section=.comment --remove-section=.note --strip-unneeded "$1"
objcopy --add-gnu-debuglink "$debugfile" "$1"' - {} ';'
[00:00:00][52886759][build:hello.bst ] SUCCESS Running strip-commands
[--:--:--][52886759][build:hello.bst ] START Caching artifact
[00:00:00][52886759][build:hello.bst ] SUCCESS Caching artifact
[00:00:01][52886759][build:hello.bst ] SUCCESS integration-commands/hello/52886759-build.2900.log
[00:00:21][][] SUCCESS Build
Pipeline Summary
Total: 4
Session: 4
Fetch Queue: processed 0, skipped 4, failed 0
Build Queue: processed 4, skipped 0, failed 0
Observe in the build process above, the integration command declared on the
base/alpine.bst element is run after staging the dependency artifacts
into the build sandbox and before running any of the build commands, for
both of the libhello.bst and hello.bst elements.
BuildStream assumes that commands which are to be run in the build sandbox need to be run in an integrated sandbox.
Tip
Integration commands can be taxing on your overall build process, because they need to run at the beginning of every build which runtime depends on the element declaring them.
For this reason, it is better to leave out more onerous tasks if they are not needed at software build time, and handle those specific tasks differently later in the pipeline, before deployment.
4.3.2. Run the hello world program¶
Unlike the previous chapters, this hello world program takes an argument, we can invoke the program using bst shell:
user@host:~/integration-commands$ bst shell hello.bst -- hello pony
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][52886759][ main:hello.bst ] START Staging dependencies
[00:00:01][52886759][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][52886759][ main:hello.bst ] START Integrating sandbox
[--:--:--][30df5cfc][ main:base/alpine.bst ] STATUS Running integration command
ldconfig "/usr/lib"
[00:00:00][52886759][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][52886759][ main:hello.bst ] STATUS Running command
hello pony
Hello pony
Here we see again, the integration commands are also used when preparing the shell to launch a command.
4.4. Summary¶
In this chapter we’ve observed how integration commands work, and we now know about public data, which plugins can read from their dependencies in order to influence their build process.