2. Filtering

In this chapter we will explore how to filter the files in an artifact using the filter element, such that an element might depend on a subset of the files provided by a filtered element.

Note

This example is distributed with BuildStream in the doc/examples/filtering subdirectory.

2.1. Overview

In some cases, it can be useful to depend on a subset of the files of an element, without depending on the entire element.

One scenario where filtering can be useful, is when you have an element which will build differently depending on what is present in the system where it is building. In an edge case where a module fails to offer configure time options to disable an unwanted feature or behavior in the build, you might use filter elements to ensure that special header files or pkg-config files are filtered out from the system at build time, such that the unwanted behavior cannot be built.

In many ways, a filter element is like a compose element, except that it operates on a single build dependency, without compositing the filtered element with its runtime dependencies.

Tip

The filter element is special in the sense that it acts as a window into it’s primary build dependency.

As such, opening a workspace on a filter element will result in opening a workspace on the element which it filters. Any other workspace commands will also be forwarded directly to the filtered element.

2.2. Project structure

This example again expands on the example presenting in the chapter about integration commands. In this case we will modify libhello.bst such that it produces a new file which, if present, will affect the behavior of it’s reverse dependency hello.bst.

Let’s first take a look at how the sources have changed.

2.2.1. `files/hello/Makefile`

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -m 755 hello ${DESTDIR}${PREFIX}/bin

hello: hello.c
	extra_flags="";                                               \
	if [ -f "${PREFIX}/share/libhello/default-person.txt" ]; then \
	  extra_flags=-DDEFAULT_PERSON="\"$$(cat ${PREFIX}/share/libhello/default-person.txt)\""; \
	fi;                                                           \
	$(CC) $< -o $@ $${extra_flags} -Wall -lhello

Now we have our Makefile discovering the system defined default person to say hello to.

2.2.2. `files/hello/hello.c`

/*
 * hello.c - Simple hello program
 */
#include <stdio.h>
#include <libhello.h>

int main(int argc, char *argv[])
{
  const char *person = NULL;

  if (argc > 1)
    person = argv[1];

  if (person)
    hello(person);
  else {
#ifdef DEFAULT_PERSON
    hello(DEFAULT_PERSON);
#else
    hello("stranger");
#endif
  }

  return 0;
}

If this program has been given a DEFAULT_PERSON, then it will say hello to that person in the absence of any argument.

2.2.3. `project.conf`

# Unique project name
name: filtering

# Minimum required BuildStream version
min-version: 2.0

# Subdirectory where elements are stored
element-path: elements

# Define an alias for our alpine tarball
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

# Use an option to decide if we should use the filter
#
options:
  use_filter:
    type: bool
    description: Whether to use a filter on the libhello.bst element
    default: False

Here, we’ve added a project option to decide whether to use the filter element or not.

This is merely for brevity, so that we can demonstrate the behavior of depending on the filtered element without defining two separate versions of the hello.bst element.

2.2.4. `elements/libhello.bst`

kind: manual
description: |

  The libhello library

# Depend on the base system
depends:
- base.bst

# Stage the files/libhello directory for building
sources:
  - kind: local
    path: files/libhello

# Now configure the commands to run
config:

  build-commands:
  - make PREFIX="%{prefix}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

public:
  bst:
    # Define a split domain which captures the defaults
    # which this library installs into %{datadir}
    #
    split-rules:
      defaults:
      - "%{datadir}/libhello/default-person.txt"

We’ve added some split rules here to declare a new split domain named defaults, and we’ve added the new default-person.txt file to this domain.

2.2.5. `elements/libhello-filtered.bst`

kind: filter
description: |

  A filtered version of libhello which excludes the defaults

# Specify the build dependency to filter
build-depends:
- libhello.bst

# Propagate runtime dependencies
runtime-depends:
- base.bst

# Now configure the commands to run
config:
  exclude:
  - defaults

And we’ve added a new filter element to the project which uses the exclude option of the filter configuration.

This is essentially a statement that any files mentioned in the the defaults domain of the libhello.bst element should be excluded from the resulting artifact.

Important

Notice that you need to explicitly declare any runtime dependencies which are required by the resulting artifact of a filter element, as runtime dependencies of the build dependency are not transient.

2.2.6. `elements/hello.bst`

kind: manual
description: |

  The hello application

# Depend on the hello library, or the filtered version
#
(?):
- use_filter == True:
    depends:
      - libhello-filtered.bst
- use_filter == False:
    depends:
      - libhello.bst

# Stage the files/hello directory for building
sources:
  - kind: local
    path: files/hello

# Now configure the commands to run
config:

  build-commands:
  - make PREFIX="%{prefix}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

Here we’ve merely added a conditional statement which allows us to test the hello.bst element depending on the filtered version of the library, or the unfiltered version.

2.3. Using the project

Let’s just skip over building the hello.bst element with the use_filter option both True and False, these elements are easily built with bst build as such:

bst --option use_filter True build hello.bst
bst --option use_filter False build hello.bst

2.3.1. Observing the artifacts

Let’s take a look at the built artifacts.

2.3.1.1. `libhello.bst`

user@host:~/filtering$ bst artifact list-contents libhello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
  libhello.bst:
	usr
	usr/include
	usr/include/libhello.h
	usr/lib
	usr/lib/libhello.so
	usr/share
	usr/share/libhello
	usr/share/libhello/default-person.txt

Here we can see the full content of the libhello.bst artifact.

2.3.1.2. `libhello-filtered.bst`

user@host:~/filtering$ bst artifact list-contents libhello-filtered.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
  libhello-filtered.bst:
	usr
	usr/include
	usr/include/libhello.h
	usr/lib
	usr/lib/libhello.so

Here we can see that the default-person.txt file has been filtered out of the libhello.bst artifact when creating the libhello-filtered.bst artifact.

2.3.2. Running hello.bst

Now if we run the program built by hello.bst in either build modes, we can observe the expected behavior.

2.3.2.1. Run `hello.bst` built directly against `libhello.bst`

user@host:~/filtering$ bst --option use_filter False shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][911b8a6d][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][911b8a6d][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][911b8a6d][    main:hello.bst                     ] START   Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] START   Running commands

    ldconfig "/usr/lib"

2023-02-17T10:34:29.760+0000 [2397:140183416750080] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][        ][    main:hello.bst                     ] SUCCESS Running commands
[00:00:01][911b8a6d][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][911b8a6d][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:34:30.898+0000 [2413:140492854636544] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello Sophia

Here we can see that the hello world program is using the system configured default person to say hello to.

2.3.2.2. Run `hello.bst` built against `libhello-filtered.bst`

user@host:~/filtering$ bst --option use_filter True shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][db2ce036][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][db2ce036][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][db2ce036][    main:hello.bst                     ] START   Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] START   Running commands

    ldconfig "/usr/lib"

2023-02-17T10:34:31.757+0000 [2447:140281321365504] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][        ][    main:hello.bst                     ] SUCCESS Running commands
[00:00:01][db2ce036][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][db2ce036][    main:hello.bst                     ] STATUS  Running command

    hello

2023-02-17T10:34:32.891+0000 [2463:139715354943488] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello stranger

And now we’re reverting to the behavior we have when no system configured default person was installed at build time.

2.4. Summary

In this chapter, we’ve introduced the filter element which allows one to filter the output of an element and effectively create a dependency on a subset of an element’s files.

2. Filtering

2.1. Overview

2.2. Project structure

2.2.1. files/hello/Makefile

2.2.2. files/hello/hello.c

2.2.3. project.conf

2.2.4. elements/libhello.bst

2.2.5. elements/libhello-filtered.bst

2.2.6. elements/hello.bst