2. Strict mode
In this section, we will cover the usage of strict vs non-strict build plans in conjunction with workspaces, and how this can help to improve your edit/compile/test cycles.
Note
This example is distributed with BuildStream in the doc/examples/strict-mode subdirectory.
2.1. Overview
When working with BuildStream to create integrations, it is typical that you have a lot of components to build, and you frequently need to modify a component at various levels of the stack. When developing one or more applications, you might want to open a workspace and fix a bug in an application, or you might need to open a workspace on a low level shared library to fix the behavior of one or more misbehaving applications.
By default, BuildStream will always choose to be deterministic in order to produce the most correct build results as possible. As such, modifying a low level library will result in rebuilding all of it’s reverse dependencies, but this can be very time consuming and inconvenient for your edit/compile/test cycles.
This is when enabling non-strict build plans can be helpful.
To illustrate the facets of how this works, this example will present a project consisting of an application which is linked both statically and dynamically linked to a common library.
2.2. Project structure
This project is mostly based on the integration commands example, as such we will ignore large parts of this project and only focus on the elements which are of specific interest.
To illustrate the relationship of these two applications and the library, let’s briefly take a look at the underlying Makefiles which are used in this project, starting with the library and followed by both Makefiles used to build the application.
2.2.1. files/libhello/Makefile
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so libhello.a
install: all
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.a ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.a: libhello.o
$(AR) rcs $@ $^
libhello.so: libhello.o
$(CC) -shared -o $@ $<
2.2.2. files/hello/Makefile.dynamic
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< -lhello
2.2.3. files/hello/Makefile.static
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< /usr/lib/libhello.a
As we can see, we have a library that is distributed both as the dynamic
library libhello.so and also as the static archive libhello.a.
Now let’s take a look at the two separate elements which build the application, first the dynamically linked version and then the static one.
2.2.4. elements/hello-dynamic.bst
kind: manual
description: |
The dynamically linked hello application
# Depend on the hello library
depends:
- libhello.bst
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.dynamic PREFIX="%{prefix}"
install-commands:
- make -f Makefile.dynamic -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Nothing very special to observe about this hello program, just a
manual element quite similar to the one we’ve
already seen in the running commands
example.
2.2.5. elements/hello-static.bst
kind: manual
description: |
The statically linked hello application
# Depend on the hello library with the strict option
#
depends:
- filename: libhello.bst
strict: true
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.static PREFIX="%{prefix}"
install-commands:
- make -f Makefile.static -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Almost the same as the dynamic element, except here we have declared the
dependency to the libhello.bst element differently: this time we have enabled
the strict option in the dependency declaration.
The side effect of setting this option is that hello-static.bst will be
rebuilt any time that libhello.bst has changed, even when
non-strict build plans have been enabled.
Tip
Some element plugins are designed to consume the content of their
dependencies entirely, and output an artifact without any transient
runtime dependencies, an example of this is the compose
element.
In cases such as compose, it is not necessary to
explicitly annotate their dependencies as strict.
It is only helpful to set the strict attribute on a
dependency declaration in the case that the
specific dependency relationship causes data to be consumed verbatim,
as is the case with static linking.
2.3. Using the project
For the sake of brevity, let’s assume that you’ve already built all of the
elements of this project, and that you want to make some changes to the
libhello.bst element, and test how it might effect the hello program.
2.3.1. Everything is already built
user@host:~/strict-mode$ bst show hello-static.bst hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
cached cfc97a856b683a50e0b18ee6073c6283763d6ccc3937717d60f4b7281fa5e562 libhello.bst
cached 27d2a1506044342dd91b6cfd0ca7ca98b5ed14d9c500b8e8beb9b2202dfb21db hello-static.bst
cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst
2.3.2. Open a workspace and modify libhello.c
Now let’s open up a workspace on the hello library
user@host:~/strict-mode$ bst workspace open --directory workspace_libhello libhello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][ ][ main:core activity ] STATUS Creating workspace for element libhello.bst
[--:--:--][cfc97a85][ main:libhello.bst ] START Staging sources to /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
[--:--:--][ ][ main:libhello.bst ] START Staging local files into CAS
[00:00:00][ ][ main:libhello.bst ] SUCCESS Staging local files into CAS
[00:00:00][cfc97a85][ main:libhello.bst ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element libhello.bst
And go ahead and make a modification like this:
--- libhello.c
+++ libhello.c
@@ -5,5 +5,5 @@
void hello(const char *person)
{
- printf("Hello %s\n", person);
+ printf("Good morning %s\n", person);
}
2.3.3. Observing hello-dynamic.bst
Let’s take a look at the bst show output for
the dynamically linked hello-dynamic.bst element.
user@host:~/strict-mode$ bst show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
buildable 23a212e63f64f31fe8841581dd08ecc6427021e37d576a857334893ebc59f387 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
waiting 73999b0d669377530a575b2afb90fb1980be6661a9d31e06505c7e3e4950088f hello-dynamic.bst
As one might expect, the libhello.bst element is ready to be built
after having been modified, and the hello-dynamic.bst element is
waiting for libhello.bst to be built before it can build.
Now let’s take a look at the same elements if we pass the --no-strict
option to bst:
user@host:~/strict-mode$ bst --no-strict show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
buildable 23a212e63f64f31fe8841581dd08ecc6427021e37d576a857334893ebc59f387 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst
Note that this time, the libhello.bst still needs to be built,
but the hello-dymamic.bst element is showing up as cached.
Tip
The bst show output will show some cache keys dimmed out in the case that they are not entirely deterministic.
Here we can see that hello-dynamic.bst is dimmed out because
it will not be rebuilt against the changed libhello.bst element,
and it also has a different cache key because of this.
2.3.4. Observing hello-static.bst
Now let’s observe the hello-static.bst element with strict mode
disabled:
user@host:~/strict-mode$ bst --no-strict show hello-static.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
buildable 23a212e63f64f31fe8841581dd08ecc6427021e37d576a857334893ebc59f387 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
waiting a28002e3e490d5f32eb2fa50bc825f560c69f902dbf447780751c2d7f6c7ecf0 hello-static.bst
Note that in this case the hello-strict.bst is going to be
rebuilt even in strict mode. This is because we annotated the
declaration of the libhello.bst dependency with the strict
attribute.
We did this because hello-strict.bst consumes the input of
libhello.bst verbatim, by way of statically linking to it, instead
of merely being affected by the content of libhello.bst at runtime,
as would be the case of static linking.
2.3.5. Building and running hello-dynamic.bst
Now let’s build hello-dynamic.bst with strict mode disabled.
user@host:~/strict-mode$ bst --no-strict build hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.0.1
Session Start: Friday, 17-02-2023 at 10:36:30
Project: strict-mode (/home/user/buildstream/doc/examples/strict-mode)
Targets: hello-dynamic.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-k6m3swnv/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-k6m3swnv
Log Files: /home/user/buildstream/doc/run-bst-k6m3swnv/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-k6m3swnv/sources
Build Area: /home/user/buildstream/doc/run-bst-k6m3swnv/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: strict-mode
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
workspace: core plugin
tar: core plugin
Pipeline
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
buildable 23a212e63f64f31fe8841581dd08ecc6427021e37d576a857334893ebc59f387 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
cached 4aacf94bf921018c2224754cdba580590bcdcd26a293bc770f4fd08152369bf4 hello-dynamic.bst
===============================================================================
[--:--:--][23a212e6][ fetch:libhello.bst ] START strict-mode/libhello/23a212e6-fetch.6528.log
[00:00:00][23a212e6][ fetch:libhello.bst ] SUCCESS strict-mode/libhello/23a212e6-fetch.6528.log
[--:--:--][23a212e6][ build:libhello.bst ] START strict-mode/libhello/23a212e6-build.6528.log
[--:--:--][23a212e6][ build:libhello.bst ] START Staging dependencies at: /
[00:00:00][23a212e6][ build:libhello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][23a212e6][ build:libhello.bst ] START Integrating sandbox
[00:00:01][23a212e6][ build:libhello.bst ] SUCCESS Integrating sandbox
[--:--:--][23a212e6][ build:libhello.bst ] START Staging sources
[00:00:00][23a212e6][ build:libhello.bst ] SUCCESS Staging sources
[--:--:--][23a212e6][ build:libhello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][23a212e6][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][23a212e6][ build:libhello.bst ] START Caching artifact
[00:00:00][23a212e6][ build:libhello.bst ] SUCCESS Caching artifact
[00:00:01][23a212e6][ build:libhello.bst ] SUCCESS strict-mode/libhello/23a212e6-build.6528.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 4
Fetch Queue: processed 1, skipped 3, failed 0
Build Queue: processed 1, skipped 3, failed 0
Note that the bst build command completed without
having to build hello-dynamic.bst at all.
And now we can also run hello-dynamic.bst
user@host:~/strict-mode$ bst --no-strict shell hello-dynamic.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][4aacf94b][ main:hello-dynamic.bst ] START Staging dependencies
[00:00:00][4aacf94b][ main:hello-dynamic.bst ] SUCCESS Staging dependencies
[--:--:--][4aacf94b][ main:hello-dynamic.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-dynamic.bst ] START Running commands
ldconfig "/usr/lib"
2023-02-17T10:36:33.284+0000 [6618:140585413820416] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][ ][ main:hello-dynamic.bst ] SUCCESS Running commands
[00:00:01][4aacf94b][ main:hello-dynamic.bst ] SUCCESS Integrating sandbox
[--:--:--][4aacf94b][ main:hello-dynamic.bst ] STATUS Running command
hello
2023-02-17T10:36:34.421+0000 [6634:139678142787584] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Good morning stranger
When running hello-dynamic.bst with no-strict mode, we are
actually reusing the old build of hello-dynamic.bst staged against
the new build of the modified libhello.bst element.
2.3.6. Building and running hello-static.bst
Finally, if we build hello-static.bst with strict mode disabled,
we can see that it will be rebuilt regardless of strict mode being
enabled.
user@host:~/strict-mode$ bst --no-strict build hello-static.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.0.1
Session Start: Friday, 17-02-2023 at 10:36:35
Project: strict-mode (/home/user/buildstream/doc/examples/strict-mode)
Targets: hello-static.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-k6m3swnv/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-k6m3swnv
Log Files: /home/user/buildstream/doc/run-bst-k6m3swnv/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-k6m3swnv/sources
Build Area: /home/user/buildstream/doc/run-bst-k6m3swnv/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: strict-mode
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
workspace: core plugin
tar: core plugin
Pipeline
cached 96140a6856a5276cd062a2af48b425f332024bf2205a481fc1c965dfefa3bacd base/alpine.bst
cached ecfab63c8a863e21bdcecc1f2be78cfd7a03400d378710d981724674fbab7bb5 base.bst
cached 23a212e63f64f31fe8841581dd08ecc6427021e37d576a857334893ebc59f387 libhello.bst Workspace: /home/user/buildstream/doc/examples/strict-mode/workspace_libhello
buildable a28002e3e490d5f32eb2fa50bc825f560c69f902dbf447780751c2d7f6c7ecf0 hello-static.bst
===============================================================================
[--:--:--][a28002e3][ build:hello-static.bst ] START strict-mode/hello-static/a28002e3-build.6645.log
[--:--:--][a28002e3][ build:hello-static.bst ] START Staging dependencies at: /
[00:00:00][a28002e3][ build:hello-static.bst ] SUCCESS Staging dependencies at: /
[--:--:--][a28002e3][ build:hello-static.bst ] START Integrating sandbox
[00:00:01][a28002e3][ build:hello-static.bst ] SUCCESS Integrating sandbox
[--:--:--][a28002e3][ build:hello-static.bst ] START Staging sources
[00:00:00][a28002e3][ build:hello-static.bst ] SUCCESS Staging sources
[--:--:--][a28002e3][ build:hello-static.bst ] START Running commands
make -f Makefile.static PREFIX="/usr"
make -f Makefile.static -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][a28002e3][ build:hello-static.bst ] SUCCESS Running commands
[--:--:--][a28002e3][ build:hello-static.bst ] START Caching artifact
[00:00:00][a28002e3][ build:hello-static.bst ] SUCCESS Caching artifact
[00:00:01][a28002e3][ build:hello-static.bst ] SUCCESS strict-mode/hello-static/a28002e3-build.6645.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 4
Fetch Queue: processed 0, skipped 4, failed 0
Build Queue: processed 1, skipped 3, failed 0
This is of course because we declared its dependency on libhello.bst
as a strict dependency.
And by the same virtue, we can see that when we run the example it has properly relinked against the changed static archive, and has the updated text in the greeting:
user@host:~/strict-mode$ bst --no-strict shell hello-static.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][a28002e3][ main:hello-static.bst ] START Staging dependencies
[00:00:00][a28002e3][ main:hello-static.bst ] SUCCESS Staging dependencies
[--:--:--][a28002e3][ main:hello-static.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-static.bst ] START Running commands
ldconfig "/usr/lib"
2023-02-17T10:36:37.174+0000 [6729:140287041736704] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][ ][ main:hello-static.bst ] SUCCESS Running commands
[00:00:01][a28002e3][ main:hello-static.bst ] SUCCESS Integrating sandbox
[--:--:--][a28002e3][ main:hello-static.bst ] STATUS Running command
hello
2023-02-17T10:36:38.310+0000 [6745:140592244604928] [buildboxcommon_casclient.cpp:96] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Good morning stranger
2.4. Summary
In this chapter we’ve explored how to use non-strict build plans in order to avoid rebuilding reverse dependencies of a lower level element you might be working with in a workspace, consequently improving your edit/compile/test experience.
We’ve also explained how to ensure your project still works properly
with non-strict build plans when some elements perform static linking
(or other operations which consume data from their dependencies
verbatim), by annotating dependency declarations
as strict.