1. Composition
In this chapter we will explore how to create compositions of multiple
input filesystem trees, using the compose element.
Note
This example is distributed with BuildStream in the doc/examples/composition subdirectory.
1.1. Overview
Composing a directory tree based on a set of build dependencies is often one of the important steps you might perform in order to create a single artifact which can be checked out and deployed.
In order to use the compose element, it is important
to first understand the concept of split rules, which
we will cover in this chapter.
1.1.1. Introducing split rules
The split rules of an element declaration denote which sets of files in the given element’s resulting artifact belong to which domain name.
The domains can then be used in various ways, using plugins which understand split rule domains.
BuildStream’s default project configuration contains a sensible set of default split rule domains for the purpose of artifact splitting, they can be overridden in your project.conf, and finally on a per element basis in the public data of your element declarations.
Note
Projects are free to add additional split rule domains on top of the default domains provided by the default project configuration.
There is nothing wrong with defining split rule domains which overlap,
possibly capturing some of the same files also captured by another
domain, however you should be aware of this when later using your
split rules with a plugin which processes them, like the
compose element described in this chapter.
1.1.1.1. Example of split rule declaration
In an element, you might need to define or extend the split-rules
in order to capture files in custom locations in a logical domain.
Here is an example of how you might use the
list append directive
to append an additional rule to your split-rules list in order to
capture additional data files which your application or library might
want to include in the runtime domain:
# Add our .dat files to the runtime domain
public:
bst:
split-rules:
runtime:
(>):
- |
%{datadir}/foo/*.dat
Split rules are absolute paths which denote files within an artifact’s root
directory. The globbing patterns supported in split rules are defined in the
reference documentation here.
Important
Note that because of variable expansion, split rules can often be
resolved differently for elements which have overridden path
related variables, like %{prefix}.
This usually means that you do not need to explicitly extend or override split rules on a specific element unless your element installs files to special case locations.
1.2. Project structure
In this example we expand on the chapter about integration commands, so we will only discuss the files which are added or changed from that example.
1.2.1. elements/base/alpine.bst
kind: import
description: |
Alpine Linux base runtime
sources:
- kind: tar
url: alpine:integration-tests-base.v1.x86_64.tar.xz
ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
public:
bst:
#
# Run ldconfig in the libdir before running anything
#
integration-commands:
- ldconfig "%{libdir}"
#
# Extend the runtime split-rule domain for this element,
# such that we capture the runtime linker.
#
# There are various other things provided by this runtime
# such as tooling in /bin and an installation of python
# and perl, but we'll overlook these for the sake of
# this example.
#
split-rules:
runtime:
(>):
- "/lib/ld*.so*"
Here we have modified the base runtime, so as to specify that for this element, we want to also include the runtime linker into the runtime domain.
1.2.2. elements/runtime-only.bst
kind: compose
# Dependencies of a compose element cannot be transient,
# we can only build-depend on the inputs of a composition.
#
build-depends:
- hello.bst
config:
# Only include files from the runtime domain
#
include:
- runtime
# Don't include any files which do not match any existing
# split rule domains.
#
include-orphans: False
# Run integration commands before composition
#
integrate: True
As we can see, this compose element has
been configured to only include files from the runtime domain.
1.3. Using the project
Now that we’ve presented how split rules
work and shown how to use them in the context of this example, lets
use the compose element we’ve created and
observe the results.
1.3.1. Building the project
user@host:~/composition$ bst build runtime-only.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732040.474003 1483 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:20
Project: composition (/home/user/buildstream/doc/examples/composition)
Targets: runtime-only.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-omt4i24x/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-omt4i24x
Log Files: /home/user/buildstream/doc/run-bst-omt4i24x/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-omt4i24x/sources
Build Area: /home/user/buildstream/doc/run-bst-omt4i24x/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: composition
Element Plugins
compose: core plugin
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
fetch needed 2f643c2da05dd1f300aa1b5aa016a925a6fecfdb3c9ff947c66a97002496aac2 base/alpine.bst
waiting 82d7d3429a94cd01054753bbdc565d3f37069aacba51051764ec0f72129349f7 base.bst
waiting 0babd9cc5beb41704ffe9a110ed8b4ca69ff8241a630b46176052544c07a35b2 libhello.bst
waiting 6fb0f24a1fee36ba2f486fa0cd0cbedeb8f7a69df15908b9370aaa7799f8353d hello.bst
waiting c22b89ed1a1f72aac4ee28fe220f9efa2c2d4002a1b1c414e659cfb089435960 runtime-only.bst
===============================================================================
[--:--:--][2f643c2d][ fetch:base/alpine.bst ] START composition/base-alpine/2f643c2d-fetch.20240815-142720.log
[--:--:--][2f643c2d][ fetch:base/alpine.bst ] START Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][82d7d342][ fetch:base.bst ] START composition/base/82d7d342-fetch.20240815-142720.log
[--:--:--][0babd9cc][ fetch:libhello.bst ] START composition/libhello/0babd9cc-fetch.20240815-142720.log
[--:--:--][c22b89ed][ fetch:runtime-only.bst ] START composition/runtime-only/c22b89ed-fetch.20240815-142720.log
[00:00:00][82d7d342][ fetch:base.bst ] SUCCESS composition/base/82d7d342-fetch.20240815-142720.log
[00:00:00][0babd9cc][ fetch:libhello.bst ] SUCCESS composition/libhello/0babd9cc-fetch.20240815-142720.log
[00:00:00][c22b89ed][ fetch:runtime-only.bst ] SUCCESS composition/runtime-only/c22b89ed-fetch.20240815-142720.log
[00:00:00][2f643c2d][ fetch:base/alpine.bst ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][2f643c2d][ fetch:base/alpine.bst ] SUCCESS composition/base-alpine/2f643c2d-fetch.20240815-142720.log
[--:--:--][2f643c2d][ build:base/alpine.bst ] START composition/base-alpine/2f643c2d-build.20240815-142727.log
[--:--:--][2f643c2d][ build:base/alpine.bst ] START Staging sources
[00:00:00][2f643c2d][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][2f643c2d][ build:base/alpine.bst ] START Caching artifact
[00:00:00][2f643c2d][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][2f643c2d][ build:base/alpine.bst ] SUCCESS composition/base-alpine/2f643c2d-build.20240815-142727.log
[--:--:--][82d7d342][ build:base.bst ] START composition/base/82d7d342-build.20240815-142727.log
[--:--:--][82d7d342][ build:base.bst ] START Caching artifact
[00:00:00][82d7d342][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][82d7d342][ build:base.bst ] SUCCESS composition/base/82d7d342-build.20240815-142727.log
[--:--:--][0babd9cc][ build:libhello.bst ] START composition/libhello/0babd9cc-build.20240815-142727.log
[--:--:--][0babd9cc][ build:libhello.bst ] START Staging dependencies at: /
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][0babd9cc][ build:libhello.bst ] START Integrating sandbox
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS Integrating sandbox
[--:--:--][0babd9cc][ build:libhello.bst ] START Staging sources
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS Staging sources
[--:--:--][0babd9cc][ build:libhello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][0babd9cc][ build:libhello.bst ] START Caching artifact
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS Caching artifact
[00:00:00][0babd9cc][ build:libhello.bst ] SUCCESS composition/libhello/0babd9cc-build.20240815-142727.log
[--:--:--][6fb0f24a][ build:hello.bst ] START composition/hello/6fb0f24a-build.20240815-142728.log
[--:--:--][6fb0f24a][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][6fb0f24a][ build:hello.bst ] START Integrating sandbox
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][6fb0f24a][ build:hello.bst ] START Staging sources
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][6fb0f24a][ build:hello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS Running commands
[--:--:--][6fb0f24a][ build:hello.bst ] START Caching artifact
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][6fb0f24a][ build:hello.bst ] SUCCESS composition/hello/6fb0f24a-build.20240815-142728.log
[--:--:--][c22b89ed][ build:runtime-only.bst ] START composition/runtime-only/c22b89ed-build.20240815-142728.log
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Staging dependencies
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Computing split
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Computing split
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Integrating sandbox
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Running commands
[--:--:--][c22b89ed][ build:runtime-only.bst ] INFO Integration added 0 and removed 0 files
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Creating composition
Including files from domains: runtime
Excluding orphaned files
[--:--:--][c22b89ed][ build:runtime-only.bst ] INFO Composing 287 files
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Creating composition
[--:--:--][c22b89ed][ build:runtime-only.bst ] START Caching artifact
[00:00:00][c22b89ed][ build:runtime-only.bst ] SUCCESS Caching artifact
[00:00:01][c22b89ed][ build:runtime-only.bst ] SUCCESS composition/runtime-only/c22b89ed-build.20240815-142728.log
[00:00:09][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 5
Session: 5
Fetch Queue: processed 4, skipped 1, failed 0
Build Queue: processed 5, skipped 0, failed 0
As you can see in the output, this composition has only a few hundred
files, but the complete alpine.bst runtime has several thousand
files.
1.3.2. List the content
At the risk of this being a long list, let’s list the contents of this artifact
user@host:~/composition$ bst artifact list-contents runtime-only.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732050.326845 1634 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
runtime-only.bst:
lib
lib/ld-musl-x86_64.so.1
usr
usr/bin
usr/bin/2to3
usr/bin/2to3-3.6
usr/bin/[
usr/bin/[[
usr/bin/aclocal
usr/bin/aclocal-1.15
usr/bin/addr2line
usr/bin/ar
usr/bin/as
usr/bin/autoconf
usr/bin/autoheader
usr/bin/autom4te
usr/bin/automake
usr/bin/automake-1.15
usr/bin/autoreconf
usr/bin/autoscan
usr/bin/autoupdate
usr/bin/awk
usr/bin/basename
usr/bin/beep
usr/bin/blkdiscard
usr/bin/bunzip2
usr/bin/bzcat
usr/bin/bzip2
usr/bin/c++filt
usr/bin/c_rehash
usr/bin/cal
usr/bin/cc
usr/bin/ccmake
usr/bin/chvt
usr/bin/cksum
usr/bin/clear
usr/bin/cmake
usr/bin/cmp
usr/bin/comm
usr/bin/cpack
usr/bin/cpio
usr/bin/crontab
usr/bin/cryptpw
usr/bin/ctest
usr/bin/cut
usr/bin/dc
usr/bin/deallocvt
usr/bin/diff
usr/bin/dirname
usr/bin/dos2unix
usr/bin/du
usr/bin/dumpleases
usr/bin/dwp
usr/bin/easy_install-3.6
usr/bin/eject
usr/bin/elfedit
usr/bin/env
usr/bin/expand
usr/bin/expr
usr/bin/factor
usr/bin/fallocate
usr/bin/find
usr/bin/flock
usr/bin/fold
usr/bin/free
usr/bin/fuser
usr/bin/gdbm_dump
usr/bin/gdbm_load
usr/bin/gdbmtool
usr/bin/getconf
usr/bin/getent
usr/bin/gprof
usr/bin/groups
usr/bin/hd
usr/bin/head
usr/bin/hello
usr/bin/hexdump
usr/bin/hostid
usr/bin/iconv
usr/bin/id
usr/bin/ifnames
usr/bin/install
usr/bin/ipcrm
usr/bin/ipcs
usr/bin/killall
usr/bin/ld
usr/bin/ld.bfd
usr/bin/ldd
usr/bin/less
usr/bin/logger
usr/bin/lsof
usr/bin/lsusb
usr/bin/lzcat
usr/bin/lzma
usr/bin/lzopcat
usr/bin/m4
usr/bin/make
usr/bin/md5sum
usr/bin/mesg
usr/bin/microcom
usr/bin/mkfifo
usr/bin/mkpasswd
usr/bin/nc
usr/bin/nl
usr/bin/nm
usr/bin/nmeter
usr/bin/nohup
usr/bin/nproc
usr/bin/nsenter
usr/bin/nslookup
usr/bin/objcopy
usr/bin/objdump
usr/bin/od
usr/bin/openvt
usr/bin/passwd
usr/bin/paste
usr/bin/patch
usr/bin/perl
usr/bin/perl5.26.1
usr/bin/perldoc
usr/bin/pgrep
usr/bin/pip3
usr/bin/pip3.6
usr/bin/pkill
usr/bin/pmap
usr/bin/pod2html
usr/bin/pod2man
usr/bin/pod2text
usr/bin/pod2usage
usr/bin/podchecker
usr/bin/podselect
usr/bin/printf
usr/bin/pscan
usr/bin/pstree
usr/bin/pwdx
usr/bin/pydoc3
usr/bin/pydoc3.6
usr/bin/python3
usr/bin/python3.6
usr/bin/python3.6m
usr/bin/pyvenv
usr/bin/pyvenv-3.6
usr/bin/ranlib
usr/bin/readelf
usr/bin/readlink
usr/bin/realpath
usr/bin/renice
usr/bin/reset
usr/bin/resize
usr/bin/scanelf
usr/bin/seq
usr/bin/setkeycodes
usr/bin/setsid
usr/bin/sha1sum
usr/bin/sha256sum
usr/bin/sha3sum
usr/bin/sha512sum
usr/bin/showkey
usr/bin/shred
usr/bin/shuf
usr/bin/size
usr/bin/smemcap
usr/bin/sort
usr/bin/split
usr/bin/ssl_client
usr/bin/strings
usr/bin/strip
usr/bin/sum
usr/bin/tac
usr/bin/tail
usr/bin/tcc
usr/bin/tee
usr/bin/test
usr/bin/time
usr/bin/timeout
usr/bin/top
usr/bin/tr
usr/bin/traceroute
usr/bin/traceroute6
usr/bin/truncate
usr/bin/tty
usr/bin/ttysize
usr/bin/udhcpc6
usr/bin/unexpand
usr/bin/uniq
usr/bin/unix2dos
usr/bin/unlink
usr/bin/unlzma
usr/bin/unlzop
usr/bin/unshare
usr/bin/unxz
usr/bin/unzip
usr/bin/uptime
usr/bin/uudecode
usr/bin/uuencode
usr/bin/vi
usr/bin/vlock
usr/bin/volname
usr/bin/wc
usr/bin/wget
usr/bin/which
usr/bin/whoami
usr/bin/whois
usr/bin/xargs
usr/bin/xmlwf
usr/bin/xxd
usr/bin/xzcat
usr/bin/yes
usr/lib
usr/lib/libarchive.so.13
usr/lib/libarchive.so.13.3.2
usr/lib/libbfd-2.28.so
usr/lib/libbz2.so.1
usr/lib/libbz2.so.1.0.6
usr/lib/libc.so
usr/lib/libcrypto.so.42
usr/lib/libcrypto.so.42.0.0
usr/lib/libcurl.so.4
usr/lib/libcurl.so.4.5.0
usr/lib/libexpat.so.1
usr/lib/libexpat.so.1.6.7
usr/lib/libffi.so.6
usr/lib/libffi.so.6.0.4
usr/lib/libformw.so.6
usr/lib/libformw.so.6.0
usr/lib/libgcc_s.so.1
usr/lib/libgdbm.so.4
usr/lib/libgdbm.so.4.0.0
usr/lib/libgdbm_compat.so.4
usr/lib/libgdbm_compat.so.4.0.0
usr/lib/libhello.so
usr/lib/liblz4.so.1
usr/lib/liblz4.so.1.8.0
usr/lib/liblzma.so.5
usr/lib/liblzma.so.5.2.3
usr/lib/libmenuw.so.6
usr/lib/libmenuw.so.6.0
usr/lib/libncursesw.so.6
usr/lib/libncursesw.so.6.0
usr/lib/libopcodes-2.28.so
usr/lib/libpanelw.so.6
usr/lib/libpanelw.so.6.0
usr/lib/libpython3.6m.so.1.0
usr/lib/libpython3.so
usr/lib/libreadline.so.7
usr/lib/libreadline.so.7.0
usr/lib/librhash.so.0
usr/lib/libsqlite3.so.0
usr/lib/libsqlite3.so.0.8.6
usr/lib/libssh2.so.1
usr/lib/libssh2.so.1.0.1
usr/lib/libssl.so.44
usr/lib/libssl.so.44.0.1
usr/lib/libstdc++.so.6
usr/lib/libstdc++.so.6.0.22
usr/lib/libuv.so.1
usr/lib/libuv.so.1.0.0
usr/sbin
usr/sbin/add-shell
usr/sbin/addgroup
usr/sbin/adduser
usr/sbin/arping
usr/sbin/brctl
usr/sbin/chpasswd
usr/sbin/chroot
usr/sbin/crond
usr/sbin/delgroup
usr/sbin/deluser
usr/sbin/ether-wake
usr/sbin/fbset
usr/sbin/fdformat
usr/sbin/killall5
usr/sbin/loadfont
usr/sbin/lspci
usr/sbin/nanddump
usr/sbin/nandwrite
usr/sbin/nbd-client
usr/sbin/ntpd
usr/sbin/partprobe
usr/sbin/powertop
usr/sbin/rdate
usr/sbin/rdev
usr/sbin/readahead
usr/sbin/readprofile
usr/sbin/remove-shell
usr/sbin/rfkill
usr/sbin/sendmail
usr/sbin/setfont
usr/sbin/setlogcons
usr/sbin/update-ca-certificates
Some things to observe here:
The list does include the
/usr/bin/helloprogram and also the/usr/lib/libhello.soshared library.These paths are both captured by the default split rules for the runtime domain.
The list does not include the
/usr/include/libhello.hheader file which was used to compile/usr/bin/hello.The header file is not captured by the runtime domain by default. It is however captured by the devel domain.
The runtime linker
/lib/ld-musl-x86_64.so.1, as this was explicitly added to the runtime domain for thebase/alpine.bstelement which provides this file.
Tip
The reader at this time might want to list the content of
other elements built from this project, such as the
hello.bst element by itself, or the base/alpine.bst
element.
1.3.3. Run the program
Finally, lets just run the program we built.
user@host:~/composition$ bst shell runtime-only.bst -- hello audience
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732050.726468 1663 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][c22b89ed][ main:runtime-only.bst ] START Staging dependencies
[00:00:00][c22b89ed][ main:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][c22b89ed][ main:runtime-only.bst ] START Integrating sandbox
[00:00:00][c22b89ed][ main:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][c22b89ed][ main:runtime-only.bst ] STATUS Running command
hello audience
Hello audience
Here we can see that we at least have the required files to run
our hello world program, however we would not have if we were
missing the runtime linker which we added in base/alpine.bst.
1.4. Summary
In this chapter we’ve gotten familiar with split rules
annotations, and we’ve learned enough about the compose
element such that we can start creating our own compositions using
split domains.
We’ve also used the list append directive and we are now observing the contents of artifacts using bst artifact list-contents.