3. Using the autotools element

In the last chapter we observed how the manual element works, allowing one to specify and run commands manually in the process of constructing an artifact.

In this chapter, we’ll go over a mostly automated build of a similar hello world example. We will observe how our configurations of the autotools element translate to configurations on the manual element, and observe how variable substitution works.

Note

This example is distributed with BuildStream in the doc/examples/autotools subdirectory.

3.1. Overview

Instead of using the local source as we have been using in the previous examples, we’re going to use a tar source this time to obtain the automake release tarball directly from the upstream hosting.

In this example we’re going to build the example program included in the upstream automake tarball itself, and we’re going to use the automated autotools build element to do so.

3.2. Project structure

3.2.1. project.conf

# Unique project name
name: autotools

# Minimum required BuildStream version
min-version: 2.0

# Subdirectory where elements are stored
element-path: elements

# Define some aliases for the tarballs we download
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
  gnu: http://ftpmirror.gnu.org/gnu/automake/

plugins:
- origin: pip
  package-name: buildstream-plugins
  elements:
  - autotools

Like the last project.conf, we’ve added another source alias for gnu, the location from which we’re going to download the automake tarball.

3.2.2. elements/base/alpine.bst and elements/base.bst

The alpine base and base stack element are defined in the same way as in the last chapter: Running commands.

3.2.3. elements/hello.bst

kind: autotools
description: |

    Hello world example from automake

variables:

  # The hello world example lives in the doc/amhello folder.
  #
  # Set the %{command-subdir} variable to that location
  # and just have the autotools element run its commands there.
  #
  command-subdir: doc/amhello

sources:
- kind: tar
  url: gnu:automake-1.16.tar.gz
  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f

depends:
- base.bst

In this case, we haven’t touched the element’s config section at all, instead we just slightly override the bahavior of the autotools build element by overriding the command-subdir variable

3.2.3.1. Looking at variables

Let’s take a moment and observe how element composition works with variables.

As the documentation mentions:

  • The initial settings of the project.conf variables are setup using BuildStream’s builtin defaults.

  • After this, your local project.conf may override some variables on a project wide basis. Those will in turn be overridden by any defaults provided by element classes, such as the variables set in the documentation of the autotools build element. The variables you set in your final <element.bst> element declarations, will have the final say on the value of a particular variable.

  • Finally, the variables, which may be composed of other variables, are resolved after all composition has taken place.

The variable we needed to override was command-subdir, which is an automatic variable provided by the BuildElement abstract class. This variable simply instructs the BuildElement in which subdirectory of the %{build-root} to run its commands in.

One can always display the resolved set of variables for a given element’s configuration using bst show:

user@host:~/autotools$ bst show --deps none --format "%{vars}" hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732026.852555     297 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
prefix: /usr
exec_prefix: /usr
bindir: /usr/bin
sbindir: /usr/sbin
libexecdir: /usr/libexec
datadir: /usr/share
sysconfdir: /etc
sharedstatedir: /usr/com
localstatedir: /var
lib: lib
libdir: /usr/lib
debugdir: /usr/lib/debug
includedir: /usr/include
docdir: /usr/share/doc
infodir: /usr/share/info
mandir: /usr/share/man
build-root: /buildstream/autotools/hello.bst
conf-root: .
install-root: /buildstream-install
strip-binaries:
project-name: autotools
max-jobs: 4
autogen: "export NOCONFIGURE=1;\n\nif [ -x ./configure ]; then true;\nelif [ -x ./autogen
  ]; then ./autogen;\nelif [ -x ./autogen.sh ]; then ./autogen.sh;\nelif [ -x ./bootstrap
  ]; then ./bootstrap;\nelif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;\nelse autoreconf
  -ivf .;\nfi"
conf-global:
conf-local:
conf-cmd: ./configure
conf-args: "--prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin \\\n--sbindir=/usr/sbin
  \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include \\\n
  --libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\n--sharedstatedir=/usr/com
  \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info  "
configure: "./configure --prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin
  \\\n--sbindir=/usr/sbin \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include
  \\\n--libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\n
  --sharedstatedir=/usr/com \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info\
  \  "
make-args:
make-install-args: ' DESTDIR="/buildstream-install" install'
make: 'make '
make-install: make -j1  DESTDIR="/buildstream-install" install
remove-libtool-modules: false
remove-libtool-libraries: false
delete-libtool-archives: "if false || false; then\n  find \"/buildstream-install\"\
  \ -name \"*.la\" -print0 | while read -d '' -r file; do\n    if grep '^shouldnotlink=yes$'
  \"${file}\" &>/dev/null; then\n      if false; then\n        echo \"Removing ${file}.\"\
  \n        rm \"${file}\"\n      else\n        echo \"Not removing ${file}.\"\n \
  \     fi\n    else\n      if false; then\n        echo \"Removing ${file}.\"\n \
  \       rm \"${file}\"\n      else\n        echo \"Not removing ${file}.\"\n   \
  \   fi\n    fi\n  done\nfi"
command-subdir: doc/amhello
element-name: hello.bst

As an exercise, we suggest that you modify the hello.bst element to set the prefix like so:

variables:
  prefix: "/opt"

And rerun the above bst show command to observe how this changes the output.

Observe where the variables are declared in the builtin defaults and autotools element documentation, and how overriding these effects the resolved set of variables.

3.3. Using the project

3.3.1. Build the hello.bst element

To build the project, run bst build in the following way:

user@host:~/autotools$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732027.269281     325 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.3.0
    Session Start: Thursday, 15-08-2024 at 14:27:07
    Project:       autotools (/home/user/buildstream/doc/examples/autotools)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-tc1hymrf/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-tc1hymrf
    Log Files:               /home/user/buildstream/doc/run-bst-tc1hymrf/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-tc1hymrf/sources
    Build Area:              /home/user/buildstream/doc/run-bst-tc1hymrf/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: autotools

    Element Plugins
        autotools: python package 'buildstream-plugins 2.2.0' at: /home/user/buildstream/.tox/docs/lib/python3.12/site-packages
        stack:     core plugin
        import:    core plugin

    Source Plugins
        tar: core plugin

Pipeline
fetch needed 6a78c03097648e558c803c20d8d20f61d3e87e3e616f3a45923704cfe95019af base/alpine.bst 
     waiting 644e964abe673c8a8723cf5a3fceeeda44a7bea364f55604fc8132af44a4fb59 base.bst 
     waiting ffeab61203b5940714c33b1cc29822b9391b9fea5876faef23c024bf6acd99cf hello.bst 
===============================================================================
[--:--:--][644e964a][   fetch:base.bst                      ] START   autotools/base/644e964a-fetch.20240815-142707.log
[--:--:--][6a78c030][   fetch:base/alpine.bst               ] START   autotools/base-alpine/6a78c030-fetch.20240815-142707.log
[--:--:--][6a78c030][   fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:00][644e964a][   fetch:base.bst                      ] SUCCESS autotools/base/644e964a-fetch.20240815-142707.log
[00:00:00][6a78c030][   fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:07][6a78c030][   fetch:base/alpine.bst               ] SUCCESS autotools/base-alpine/6a78c030-fetch.20240815-142707.log
[--:--:--][6a78c030][   build:base/alpine.bst               ] START   autotools/base-alpine/6a78c030-build.20240815-142714.log
[--:--:--][6a78c030][   build:base/alpine.bst               ] START   Staging sources
[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][6a78c030][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][6a78c030][   build:base/alpine.bst               ] SUCCESS autotools/base-alpine/6a78c030-build.20240815-142714.log
[--:--:--][644e964a][   build:base.bst                      ] START   autotools/base/644e964a-build.20240815-142714.log
[--:--:--][644e964a][   build:base.bst                      ] START   Caching artifact
[00:00:00][644e964a][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][644e964a][   build:base.bst                      ] SUCCESS autotools/base/644e964a-build.20240815-142714.log
[--:--:--][ffeab612][   build:hello.bst                     ] START   autotools/hello/ffeab612-build.20240815-142714.log
[--:--:--][ffeab612][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][ffeab612][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][ffeab612][   build:hello.bst                     ] START   Staging sources
[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][ffeab612][   build:hello.bst                     ] START   Running commands

    export NOCONFIGURE=1;
    
    if [ -x ./configure ]; then true;
    elif [ -x ./autogen ]; then ./autogen;
    elif [ -x ./autogen.sh ]; then ./autogen.sh;
    elif [ -x ./bootstrap ]; then ./bootstrap;
    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
    else autoreconf -ivf .;
    fi
    ./configure --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --includedir=/usr/include \
    --libdir=/usr/lib \
    --libexecdir=/usr/libexec \
    --localstatedir=/var \
    --sharedstatedir=/usr/com \
    Message contains 23 additional lines

[00:00:03][ffeab612][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][ffeab612][   build:hello.bst                     ] START   Caching artifact
[00:00:00][ffeab612][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:03][ffeab612][   build:hello.bst                     ] SUCCESS autotools/hello/ffeab612-build.20240815-142714.log
[00:00:11][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 2, skipped 1, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

3.3.2. Run the hello world program

We probably know by now what’s going to happen, but let’s run the program we’ve compiled anyway using bst shell:

user@host:~/autotools$ bst shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732038.954148    1408 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][ffeab612][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][ffeab612][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][ffeab612][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][ffeab612][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][ffeab612][    main:hello.bst                     ] STATUS  Running command

    hello

Hello World!
This is amhello 1.0.

3.4. Summary

Now we’ve used an external build element, from the buildstream-plugins package and we’ve taken a look into how variables work.

When browsing the build elements in their respective documentation, we are now equipped with a good idea of what an element is going to do, based on their default YAML configuration and any configurations we have in our project. We can also now observe what variables are in effect for the build of a given element, using bst show.