3. Using the autotools element
In the last chapter we observed how the
manual
element works, allowing one to specify and
run commands manually in the process of constructing an artifact.
In this chapter, we’ll go over a mostly automated build of a similar
hello world example. We will observe how our configurations of the
autotools
element translate to configurations
on the manual
element, and observe how
variable substitution works.
Note
This example is distributed with BuildStream in the doc/examples/autotools subdirectory.
3.1. Overview
Instead of using the local
source as we have been using
in the previous examples, we’re going to use a tar
source
this time to obtain the automake
release tarball directly from the upstream
hosting.
In this example we’re going to build the example program included in the
upstream automake
tarball itself, and we’re going to use the automated
autotools
build element to do so.
3.2. Project structure
3.2.1. project.conf
# Unique project name
name: autotools
# Required BuildStream format version
format-version: 9
# Subdirectory where elements are stored
element-path: elements
# Define some aliases for the tarballs we download
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
gnu: http://ftpmirror.gnu.org/gnu/automake/
Like the last project.conf, we’ve
added another source alias for gnu
, the location
from which we’re going to download the automake
tarball.
3.2.2. elements/base/alpine.bst
and elements/base.bst
The alpine base and base stack element are defined in the same way as in the last chapter: Running commands.
3.2.3. elements/hello.bst
kind: autotools
description: |
Hello world example from automake
variables:
# The hello world example lives in the doc/amhello folder.
#
# Set the %{command-subdir} variable to that location
# and just have the autotools element run it's commands there.
#
command-subdir: doc/amhello
sources:
- kind: tar
url: gnu:automake-1.16.tar.gz
ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f
depends:
- base.bst
In this case, we haven’t touched the element’s config
section
at all, instead we just slightly override the bahavior of the
autotools
build element by overriding
the command-subdir variable
3.2.3.1. Looking at variables
Let’s take a moment and observe how element composition works with variables.
As the documentation mentions:
The initial settings of the
project.conf
variables are setup using BuildStream’s builtin defaults.After this, your local
project.conf
may override some variables on a project wide basis. Those will in turn be overridden by any defaults provided by element classes, such as the variables set in the documentation of theautotools
build element. The variables you set in your final<element.bst>
element declarations, will have the final say on the value of a particular variable.Finally, the variables, which may be composed of other variables, are resolved after all composition has taken place.
The variable we needed to override was command-subdir
, which is an
automatic variable provided by the BuildElement
abstract class. This variable simply instructs the BuildElement
in which subdirectory of the %{build-root}
to run it’s commands in.
One can always display the resolved set of variables for a given element’s configuration using bst show:
user@host:~/autotools$ bst show --deps none --format "%{vars}" hello.bst
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
sysconfdir: /etc
debugdir: /usr/lib/debug
build-root: /buildstream/autotools/hello.bst
make: make
sharedstatedir: /usr/com
autogen: "export NOCONFIGURE=1;\n\nif [ -x ./configure ]; then true;\nelif [ -x autogen\
\ ]; then ./autogen;\nelif [ -x autogen.sh ]; then ./autogen.sh;\nelif [ -x bootstrap\
\ ]; then ./bootstrap;\nelif [ -x bootstrap.sh ]; then ./bootstrap.sh;\nelse autoreconf\
\ -ivf;\nfi"
prefix: /usr
exec_prefix: /usr
libdir: /usr/lib
objcopy-extract-args: --only-keep-debug --compress-debug-sections
bindir: /usr/bin
conf-args: "--prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin \\\n--sbindir=/usr/sbin\
\ \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include \\\
\n--libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\n\
--sharedstatedir=/usr/com \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info\
\ "
command-subdir: doc/amhello
sbindir: /usr/sbin
make-install: make -j1 DESTDIR="/buildstream-install" install
libexecdir: /usr/libexec
configure: "./configure --prefix=/usr \\\n--exec-prefix=/usr \\\n--bindir=/usr/bin\
\ \\\n--sbindir=/usr/sbin \\\n--sysconfdir=/etc \\\n--datadir=/usr/share \\\n--includedir=/usr/include\
\ \\\n--libdir=/usr/lib \\\n--libexecdir=/usr/libexec \\\n--localstatedir=/var \\\
\n--sharedstatedir=/usr/com \\\n--mandir=/usr/share/man \\\n--infodir=/usr/share/info\
\ "
objcopy-link-args: --add-gnu-debuglink
element-name: hello.bst
conf-local:
datadir: /usr/share
project-name: autotools
install-root: /buildstream-install
max-jobs: 2
strip-binaries: "cd \"/buildstream-install\" && find -type f \\\n '(' -perm -111\
\ -o -name '*.so*' \\\n -o -name '*.cmxs' -o -name '*.node' ')' \\\n -exec\
\ sh -ec \\\n 'read -n4 hdr <\"$1\" # check for elf header\n if [ \"$hdr\" !=\
\ \"$(printf \\\\x7fELF)\" ]; then\n exit 0\n fi\n debugfile=\"/buildstream-install/usr/lib/debug/$1\"\
\n mkdir -p \"$(dirname \"$debugfile\")\"\n objcopy --only-keep-debug --compress-debug-sections\
\ \"$1\" \"$debugfile\"\n chmod 644 \"$debugfile\"\n strip --remove-section=.comment\
\ --remove-section=.note --strip-unneeded \"$1\"\n objcopy --add-gnu-debuglink\
\ \"$debugfile\" \"$1\"' - {} ';'"
fix-pyc-timestamps: "find \"/buildstream-install\" -name '*.pyc' -exec \\\n dd if=/dev/zero\
\ of={} bs=1 count=4 seek=4 conv=notrunc ';'"
docdir: /usr/share/doc
conf-extra:
includedir: /usr/include
localstatedir: /var
infodir: /usr/share/info
strip-args: --remove-section=.comment --remove-section=.note --strip-unneeded
__bst_provenance_info: /home/user/buildstream/buildstream/data/projectconfig.yaml
[line 20 column 2]
mandir: /usr/share/man
conf-global:
lib: lib
conf-cmd: ./configure
As an exercise, we suggest that you modify the hello.bst
element to set the prefix like so:
variables:
prefix: "/opt"
And rerun the above bst show command to observe how this changes the output.
Observe where the variables are declared in the builtin defaults and autotools
element
documentation, and how overriding these affects the resolved set of variables.
3.3. Using the project
3.3.1. Build the hello.bst element
To build the project, run bst build in the following way:
user@host:~/autotools$ bst build hello.bst
[--:--:--][][] START Build
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START Checking sources
[00:00:00][][] SUCCESS Checking sources
BuildStream Version 1.6.9
Session Start: Tuesday, 21-02-2023 at 14:38:07
Project: autotools (/home/user/buildstream/doc/examples/autotools)
Targets: hello.bst
Cache Usage: 12K / infinity (0%)
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-c4z67wh_/buildstream.conf
Log Files: /home/user/buildstream/doc/run-bst-c4z67wh_/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-c4z67wh_/sources
Build Area: /home/user/buildstream/doc/run-bst-c4z67wh_/build
Artifact Cache: /home/user/buildstream/doc/run-bst-c4z67wh_/artifacts
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
buildable d5472eb60ceb8a45b0ed5907912b5d89ce91d8a4e9b454ea2663f052c6cf4dff base/alpine.bst
waiting 50110d08831106135148d991c33f15794b530938712216e84593eeff7c1ef8ca base.bst
waiting 3e973b0426351cd9948d72276133798f231564928935a941990d3bea8b1d3321 hello.bst
===============================================================================
[--:--:--][d5472eb6][build:base/alpine.bst ] START autotools/base-alpine/d5472eb6-build.255.log
[--:--:--][d5472eb6][build:base/alpine.bst ] START Staging sources
[00:00:05][d5472eb6][build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][d5472eb6][build:base/alpine.bst ] START Caching artifact
[00:00:02][d5472eb6][build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:09][d5472eb6][build:base/alpine.bst ] SUCCESS autotools/base-alpine/d5472eb6-build.255.log
[--:--:--][50110d08][build:base.bst ] START autotools/base/50110d08-build.257.log
[--:--:--][50110d08][build:base.bst ] START Caching artifact
[00:00:00][50110d08][build:base.bst ] SUCCESS Caching artifact
[00:00:00][50110d08][build:base.bst ] SUCCESS autotools/base/50110d08-build.257.log
[--:--:--][3e973b04][build:hello.bst ] START autotools/hello/3e973b04-build.259.log
[--:--:--][3e973b04][build:hello.bst ] START Staging dependencies
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][3e973b04][build:hello.bst ] START Integrating sandbox
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][3e973b04][build:hello.bst ] START Staging sources
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Staging sources
[--:--:--][3e973b04][build:hello.bst ] START Running configure-commands
[--:--:--][3e973b04][build:hello.bst ] STATUS Running configure-commands
export NOCONFIGURE=1;
if [ -x ./configure ]; then true;
elif [ -x autogen ]; then ./autogen;
elif [ -x autogen.sh ]; then ./autogen.sh;
elif [ -x bootstrap ]; then ./bootstrap;
elif [ -x bootstrap.sh ]; then ./bootstrap.sh;
else autoreconf -ivf;
fi
[--:--:--][3e973b04][build:hello.bst ] STATUS Running configure-commands
./configure --prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--datadir=/usr/share \
--includedir=/usr/include \
--libdir=/usr/lib \
--libexecdir=/usr/libexec \
--localstatedir=/var \
--sharedstatedir=/usr/com \
--mandir=/usr/share/man \
--infodir=/usr/share/info
[00:00:03][3e973b04][build:hello.bst ] SUCCESS Running configure-commands
[--:--:--][3e973b04][build:hello.bst ] START Running build-commands
[--:--:--][3e973b04][build:hello.bst ] STATUS Running build-commands
make
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Running build-commands
[--:--:--][3e973b04][build:hello.bst ] START Running install-commands
[--:--:--][3e973b04][build:hello.bst ] STATUS Running install-commands
make -j1 DESTDIR="/buildstream-install" install
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Running install-commands
[--:--:--][3e973b04][build:hello.bst ] START Running strip-commands
[--:--:--][3e973b04][build:hello.bst ] STATUS Running strip-commands
cd "/buildstream-install" && find -type f \
'(' -perm -111 -o -name '*.so*' \
-o -name '*.cmxs' -o -name '*.node' ')' \
-exec sh -ec \
'read -n4 hdr <"$1" # check for elf header
if [ "$hdr" != "$(printf \\x7fELF)" ]; then
exit 0
fi
debugfile="/buildstream-install/usr/lib/debug/$1"
mkdir -p "$(dirname "$debugfile")"
objcopy --only-keep-debug --compress-debug-sections "$1" "$debugfile"
chmod 644 "$debugfile"
strip --remove-section=.comment --remove-section=.note --strip-unneeded "$1"
objcopy --add-gnu-debuglink "$debugfile" "$1"' - {} ';'
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Running strip-commands
[--:--:--][3e973b04][build:hello.bst ] START Caching artifact
[00:00:00][3e973b04][build:hello.bst ] SUCCESS Caching artifact
[00:00:05][3e973b04][build:hello.bst ] SUCCESS autotools/hello/3e973b04-build.259.log
[00:00:14][][] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 3, skipped 0, failed 0
3.3.2. Run the hello world program
We probably know by now what’s going to happen, but let’s run the program we’ve compiled anyway using bst shell:
user@host:~/autotools$ bst shell hello.bst -- hello
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][3e973b04][ main:hello.bst ] START Staging dependencies
[00:00:00][3e973b04][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][3e973b04][ main:hello.bst ] START Integrating sandbox
[00:00:00][3e973b04][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][3e973b04][ main:hello.bst ] STATUS Running command
hello
Hello World!
This is amhello 1.0.
3.4. Summary
Now we’ve used a builtin build element, and we’ve taken a look into how variables work.
When browsing the build elements in the documentation, we are now equipped with a good idea of what an element is going to do, based on their default YAML configuration and any configurations we have in our project. We can also now observe what variables are in effect for the build of a given element, using bst show.