2. Filtering
In this chapter we will explore how to filter the files in an artifact
using the filter element, such that an element might
depend on a subset of the files provided by a filtered element.
Note
This example is distributed with BuildStream in the doc/examples/filtering subdirectory.
2.1. Overview
In some cases, it can be useful to depend on a subset of the files of an element, without depending on the entire element.
One scenario where filtering can be useful, is when you have an element which
will build differently depending on what is present in the system where it is
building. In an edge case where a module fails to offer configure time options to
disable an unwanted feature or behavior in the build, you might use
filter elements to ensure that special header files or
pkg-config files are filtered out from the system at build time, such that
the unwanted behavior cannot be built.
In many ways, a filter element is like a
compose element, except that it operates on a single
build dependency, without compositing the filtered
element with its runtime dependencies.
Tip
The filter element is special in the sense
that it acts as a window into it’s primary
build dependency.
As such, opening a workspace on a
filter element will result in opening a
workspace on the element which it filters. Any other workspace
commands will also be forwarded directly to the filtered element.
2.2. Project structure
This example again expands on the example presenting in the chapter about
integration commands. In this case
we will modify libhello.bst such that it produces a new file which,
if present, will affect the behavior of it’s reverse dependency hello.bst.
Let’s first take a look at how the sources have changed.
2.2.1. files/hello/Makefile
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
extra_flags=""; \
if [ -f "${PREFIX}/share/libhello/default-person.txt" ]; then \
extra_flags=-DDEFAULT_PERSON="\"$$(cat ${PREFIX}/share/libhello/default-person.txt)\""; \
fi; \
$(CC) $< -o $@ $${extra_flags} -Wall -lhello
Now we have our Makefile discovering the system defined default person to say hello to.
2.2.2. files/hello/hello.c
/*
* hello.c - Simple hello program
*/
#include <stdio.h>
#include <libhello.h>
int main(int argc, char *argv[])
{
const char *person = NULL;
if (argc > 1)
person = argv[1];
if (person)
hello(person);
else {
#ifdef DEFAULT_PERSON
hello(DEFAULT_PERSON);
#else
hello("stranger");
#endif
}
return 0;
}
If this program has been given a DEFAULT_PERSON, then it will
say hello to that person in the absence of any argument.
2.2.3. project.conf
# Unique project name
name: filtering
# Minimum required BuildStream version
min-version: 2.0
# Subdirectory where elements are stored
element-path: elements
# Define an alias for our alpine tarball
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
# Use an option to decide if we should use the filter
#
options:
use_filter:
type: bool
description: Whether to use a filter on the libhello.bst element
default: False
Here, we’ve added a project option to decide
whether to use the filter element or not.
This is merely for brevity, so that we can demonstrate the behavior
of depending on the filtered element without defining two separate versions
of the hello.bst element.
2.2.4. elements/libhello.bst
kind: manual
description: |
The libhello library
# Depend on the base system
depends:
- base.bst
# Stage the files/libhello directory for building
sources:
- kind: local
path: files/libhello
# Now configure the commands to run
config:
build-commands:
- make PREFIX="%{prefix}"
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
public:
bst:
# Define a split domain which captures the defaults
# which this library installs into %{datadir}
#
split-rules:
defaults:
- "%{datadir}/libhello/default-person.txt"
We’ve added some split rules here to declare
a new split domain named defaults, and we’ve added the new
default-person.txt file to this domain.
2.2.5. elements/libhello-filtered.bst
kind: filter
description: |
A filtered version of libhello which excludes the defaults
# Specify the build dependency to filter
build-depends:
- libhello.bst
# Propagate runtime dependencies
runtime-depends:
- base.bst
# Now configure the commands to run
config:
exclude:
- defaults
And we’ve added a new filter element to the project
which uses the exclude option of the filter configuration.
This is essentially a statement that any files mentioned in the the
defaults domain of the libhello.bst element should be excluded from
the resulting artifact.
Important
Notice that you need to explicitly declare any
runtime dependencies which are required by the
resulting artifact of a filter element, as runtime
dependencies of the build dependency are not transient.
2.2.6. elements/hello.bst
kind: manual
description: |
The hello application
# Depend on the hello library, or the filtered version
#
(?):
- use_filter == True:
depends:
- libhello-filtered.bst
- use_filter == False:
depends:
- libhello.bst
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make PREFIX="%{prefix}"
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Here we’ve merely added a conditional statement
which allows us to test the hello.bst element depending on the filtered
version of the library, or the unfiltered version.
2.3. Using the project
Let’s just skip over building the hello.bst element with the
use_filter option both True and False, these elements
are easily built with bst build as such:
bst --option use_filter True build hello.bst
bst --option use_filter False build hello.bst
2.3.1. Observing the artifacts
Let’s take a look at the built artifacts.
2.3.1.1. libhello.bst
user@host:~/filtering$ bst artifact list-contents libhello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888199.731437 2644 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
libhello.bst:
usr
usr/include
usr/include/libhello.h
usr/lib
usr/lib/libhello.so
usr/share
usr/share/libhello
usr/share/libhello/default-person.txt
Here we can see the full content of the libhello.bst artifact.
2.3.1.2. libhello-filtered.bst
user@host:~/filtering$ bst artifact list-contents libhello-filtered.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888200.141202 2672 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
libhello-filtered.bst:
usr
usr/include
usr/include/libhello.h
usr/lib
usr/lib/libhello.so
Here we can see that the default-person.txt file has been filtered
out of the libhello.bst artifact when creating the libhello-filtered.bst
artifact.
2.3.2. Running hello.bst
Now if we run the program built by hello.bst in either build
modes, we can observe the expected behavior.
2.3.2.1. Run hello.bst built directly against libhello.bst
user@host:~/filtering$ bst --option use_filter False shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888200.558882 2700 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][911b8a6d][ main:hello.bst ] START Staging dependencies
[00:00:00][911b8a6d][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][911b8a6d][ main:hello.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello.bst ] START Running commands
ldconfig "/usr/lib"
+ sh -e -c ldconfig "/usr/lib"
[00:00:00][ ][ main:hello.bst ] SUCCESS Running commands
[00:00:00][911b8a6d][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][911b8a6d][ main:hello.bst ] STATUS Running command
hello
Hello Sophia
Here we can see that the hello world program is using the system configured default person to say hello to.
2.3.2.2. Run hello.bst built against libhello-filtered.bst
user@host:~/filtering$ bst --option use_filter True shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730888201.788871 2765 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][db2ce036][ main:hello.bst ] START Staging dependencies
[00:00:00][db2ce036][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][db2ce036][ main:hello.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello.bst ] START Running commands
ldconfig "/usr/lib"
+ sh -e -c ldconfig "/usr/lib"
[00:00:00][ ][ main:hello.bst ] SUCCESS Running commands
[00:00:00][db2ce036][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][db2ce036][ main:hello.bst ] STATUS Running command
hello
Hello stranger
And now we’re reverting to the behavior we have when no system configured default person was installed at build time.
2.4. Summary
In this chapter, we’ve introduced the filter
element which allows one to filter the output of an element and
effectively create a dependency on a subset of an element’s files.