4. Integration commands

Sometimes a software requires more configuration or processing than what is performed at installation time, otherwise it will not run properly.

This is especially true in cases where a daemon or library interoperates with third party extensions and needs to maintain a system wide cache whenever its extensions are installed or removed; system wide font caches are an example of this.

In these cases we use integration commands to ensure that a runtime is ready to run after all of its components have been staged.

Note

This example is distributed with BuildStream in the doc/examples/integration-commands subdirectory.

4.1. Overview

In this chapter, we’ll be exploring integration commands, which will be our first look at public data.

4.2. Project structure

4.2.1. project.conf and elements/base.bst

The project.conf and base stack stack element are configured in the same way as in the previous chapter: Running commands.

4.2.2. elements/base/alpine.bst

kind: import
description: |

    Alpine Linux base runtime

sources:
- kind: tar
  url: alpine:integration-tests-base.v1.x86_64.tar.xz
  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

#
# Run ldconfig in the libdir before running anything
#
public:
  bst:
    integration-commands:
    - ldconfig "%{libdir}"

This is the same base/alpine.bst we’ve seen in previous chapters, except that we’ve added an integration command.

This informs BuildStream that whenever the output of this element is expected to run, this command should be run first. In this case we are simply running ldconfig as a precautionary measure, to ensure that the runtime linker is ready to find any shared libraries we may have added to %{libdir}.

4.2.2.1. Looking at public data

The integration commands used here is the first time we’ve used any builtin public data.

Public data is a free form portion of an element’s configuration and is not necessarily understood by the element on which it is declared, public data is intended to be read by its reverse dependency elements.

This allows annotations on some elements to inform elements later in the dependency chain about details of its artifact, or to suggest how it should be processed.

4.2.3. elements/libhello.bst and elements/hello.bst

These are basically manual elements very similar to the ones we’ve seen in the previous chapter: Running commands.

These produce a library and a hello program which uses the library, we will consider these irrelevant to the topic and leave examination of their sources as an exercise for the reader.

4.3. Using the project

4.3.1. Build the hello.bst element

To build the project, run bst build in the following way:

user@host:~/integration-commands$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732090.850992    2953 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.3.0
    Session Start: Thursday, 15-08-2024 at 14:28:10
    Project:       integration-commands (/home/user/buildstream/doc/examples/integration-commands)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-3fkjlkkr/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-3fkjlkkr
    Log Files:               /home/user/buildstream/doc/run-bst-3fkjlkkr/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-3fkjlkkr/sources
    Build Area:              /home/user/buildstream/doc/run-bst-3fkjlkkr/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: integration-commands

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
fetch needed 5ff937aaa74f04d961f639c06d636e4a93abbc3a8d289a10ae8084f1d800a2de base/alpine.bst 
     waiting 0848bf1769eb7f774cfe146212b0cfe56f9d7b7320832c93a7823e50ab993138 base.bst 
     waiting 3cf9edd00c109f063e4a0c30d826420e6d240e59ee10241d59b22a7883ecef4f libhello.bst 
     waiting 76f82637c19f9595a18d2e480da027552cb53811e39b7481b2feb767f0d2311d hello.bst 
===============================================================================
[--:--:--][0848bf17][   fetch:base.bst                      ] START   integration-commands/base/0848bf17-fetch.20240815-142810.log
[--:--:--][5ff937aa][   fetch:base/alpine.bst               ] START   integration-commands/base-alpine/5ff937aa-fetch.20240815-142810.log
[--:--:--][3cf9edd0][   fetch:libhello.bst                  ] START   integration-commands/libhello/3cf9edd0-fetch.20240815-142810.log
[00:00:00][0848bf17][   fetch:base.bst                      ] SUCCESS integration-commands/base/0848bf17-fetch.20240815-142810.log
[--:--:--][5ff937aa][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:00][3cf9edd0][   fetch:libhello.bst                  ] SUCCESS integration-commands/libhello/3cf9edd0-fetch.20240815-142810.log
[00:00:00][5ff937aa][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][5ff937aa][   fetch:base/alpine.bst               ] SUCCESS integration-commands/base-alpine/5ff937aa-fetch.20240815-142810.log
[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   integration-commands/base-alpine/5ff937aa-build.20240815-142817.log
[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   Staging sources
[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][5ff937aa][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][5ff937aa][   build:base/alpine.bst               ] SUCCESS integration-commands/base-alpine/5ff937aa-build.20240815-142817.log
[--:--:--][0848bf17][   build:base.bst                      ] START   integration-commands/base/0848bf17-build.20240815-142817.log
[--:--:--][0848bf17][   build:base.bst                      ] START   Caching artifact
[00:00:00][0848bf17][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][0848bf17][   build:base.bst                      ] SUCCESS integration-commands/base/0848bf17-build.20240815-142817.log
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   integration-commands/libhello/3cf9edd0-build.20240815-142817.log
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Staging dependencies at: /
[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Staging dependencies at: /
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Integrating sandbox
[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Integrating sandbox
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Staging sources
[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Staging sources
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Running commands

    make PREFIX="/usr"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Running commands
[--:--:--][3cf9edd0][   build:libhello.bst                  ] START   Caching artifact
[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS Caching artifact
[00:00:00][3cf9edd0][   build:libhello.bst                  ] SUCCESS integration-commands/libhello/3cf9edd0-build.20240815-142817.log
[--:--:--][76f82637][   build:hello.bst                     ] START   integration-commands/hello/76f82637-build.20240815-142818.log
[--:--:--][76f82637][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][76f82637][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][76f82637][   build:hello.bst                     ] START   Staging sources
[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][76f82637][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][76f82637][   build:hello.bst                     ] START   Caching artifact
[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][76f82637][   build:hello.bst                     ] SUCCESS integration-commands/hello/76f82637-build.20240815-142818.log
[00:00:08][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       4
    Session:     4
    Fetch Queue: processed 3, skipped 1, failed 0 
    Build Queue: processed 4, skipped 0, failed 0

Observe in the build process above, the integration command declared on the base/alpine.bst element is run after staging the dependency artifacts into the build sandbox and before running any of the build commands, for both of the libhello.bst and hello.bst elements.

BuildStream assumes that commands which are to be run in the build sandbox need to be run in an integrated sandbox.

Tip

Integration commands can be taxing on your overall build process, because they need to run at the beginning of every build which runtime depends on the element declaring them.

For this reason, it is better to leave out more onerous tasks if they are not needed at software build time, and handle those specific tasks differently later in the pipeline, before deployment.

4.3.2. Run the hello world program

Unlike the previous chapters, this hello world program takes an argument, we can invoke the program using bst shell:

user@host:~/integration-commands$ bst shell hello.bst -- hello pony

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732099.696854    3084 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][76f82637][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][76f82637][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][76f82637][    main:hello.bst                     ] START   Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] START   Running commands

    ldconfig "/usr/lib"

+ sh -e -c ldconfig "/usr/lib"
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Running commands
[00:00:00][76f82637][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][76f82637][    main:hello.bst                     ] STATUS  Running command

    hello pony

Hello pony

Here we see again, the integration commands are also used when preparing the shell to launch a command.

4.4. Summary

In this chapter we’ve observed how integration commands work, and we now know about public data, which plugins can read from their dependencies in order to influence their build process.