1. Workspaces
In this section we will cover the use of BuildStream’s workspaces feature when devloping a BuildStream project.
Note
This example is distributed with BuildStream in the doc/examples/developing subdirectory.
We will start with the project used in the running commands tutorial. Recall the element hello.bst, which builds the bellow C file:
/*
* hello.c - Simple hello world program
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("Hello World\n");
return 0;
}
Suppose we now want to alter the functionality of the hello command. We can make changes to the source code of Buildstream elements by making use of BuildStream’s workspace command.
1.1. Opening a workspace
First we need to open a workspace, we can do this by running
user@host:~/developing$ bst workspace open --directory workspace_hello hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732052.080148 1737 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][ ][ main:core activity ] STATUS Creating workspace for element hello.bst
[--:--:--][a6a39926][ main:hello.bst ] START Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:hello.bst ] START Staging local files into CAS
[00:00:00][ ][ main:hello.bst ] SUCCESS Staging local files into CAS
[00:00:00][a6a39926][ main:hello.bst ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element hello.bst
This command has created the workspace_hello directory in which you can see the source for the hello.bst element, i.e. hello.c and the corresponding makefile.
You can view existing workspaces using
user@host:~/developing$ bst workspace list
workspaces:
- element: hello.bst
directory: /home/user/buildstream/doc/examples/developing/workspace_hello
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732052.482593 1765 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
1.2. Making code changes
Let’s say we want to alter the message printed when the hello command is run. We can open workspace_hello/hello.c and make the following change:
-- hello.c 2018-06-25 14:48:32.077568920 +0100
+++ hello.c 2018-06-25 14:49:23.025553785 +0100
@@ -5,6 +5,6 @@
int main(int argc, char *argv[])
{
- printf("Hello World\n");
+ printf("Hello World\nWe can use workspaces!\n");
return 0;
}
Now, rebuild the hello.bst element.
user@host:~/developing$ bst build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732052.778889 1793 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:32
Project: developing (/home/user/buildstream/doc/examples/developing)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-axv8wi9e/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-axv8wi9e
Log Files: /home/user/buildstream/doc/run-bst-axv8wi9e/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-axv8wi9e/sources
Build Area: /home/user/buildstream/doc/run-bst-axv8wi9e/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: developing
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
workspace: core plugin
tar: core plugin
Pipeline
fetch needed b65366701aaab8b2f85afa7158f4036a1f55a8150db42fff45e0830cc8388d73 base/alpine.bst
waiting 07db7692b3828f7021ae8c1b0b6f715801e64cf24fcb1621a67b47c110fc3c70 base.bst
waiting 410bcb8e584bdeff697d7886018ece828e6d1139b86676ca21f1657fb98fa246 hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
===============================================================================
[--:--:--][b6536670][ fetch:base/alpine.bst ] START developing/base-alpine/b6536670-fetch.20240815-142732.log
[--:--:--][b6536670][ fetch:base/alpine.bst ] START Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[--:--:--][07db7692][ fetch:base.bst ] START developing/base/07db7692-fetch.20240815-142732.log
[--:--:--][410bcb8e][ fetch:hello.bst ] START developing/hello/410bcb8e-fetch.20240815-142732.log
[00:00:00][07db7692][ fetch:base.bst ] SUCCESS developing/base/07db7692-fetch.20240815-142732.log
[00:00:00][410bcb8e][ fetch:hello.bst ] SUCCESS developing/hello/410bcb8e-fetch.20240815-142732.log
[00:00:00][b6536670][ fetch:base/alpine.bst ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:06][b6536670][ fetch:base/alpine.bst ] SUCCESS developing/base-alpine/b6536670-fetch.20240815-142732.log
[--:--:--][b6536670][ build:base/alpine.bst ] START developing/base-alpine/b6536670-build.20240815-142739.log
[--:--:--][b6536670][ build:base/alpine.bst ] START Staging sources
[00:00:00][b6536670][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][b6536670][ build:base/alpine.bst ] START Caching artifact
[00:00:00][b6536670][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][b6536670][ build:base/alpine.bst ] SUCCESS developing/base-alpine/b6536670-build.20240815-142739.log
[--:--:--][07db7692][ build:base.bst ] START developing/base/07db7692-build.20240815-142739.log
[--:--:--][07db7692][ build:base.bst ] START Caching artifact
[00:00:00][07db7692][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][07db7692][ build:base.bst ] SUCCESS developing/base/07db7692-build.20240815-142739.log
[--:--:--][410bcb8e][ build:hello.bst ] START developing/hello/410bcb8e-build.20240815-142739.log
[--:--:--][410bcb8e][ build:hello.bst ] START Staging dependencies at: /
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS Staging dependencies at: /
[--:--:--][410bcb8e][ build:hello.bst ] START Integrating sandbox
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][410bcb8e][ build:hello.bst ] START Staging sources
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][410bcb8e][ build:hello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS Running commands
[--:--:--][410bcb8e][ build:hello.bst ] START Caching artifact
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][410bcb8e][ build:hello.bst ] SUCCESS developing/hello/410bcb8e-build.20240815-142739.log
[00:00:06][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 3, skipped 0, failed 0
Build Queue: processed 3, skipped 0, failed 0
Note that if you run the command from inside the workspace, the element name is optional.
user@host:~/workspace_hello$ bst build
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732060.125629 1853 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:27:40
Project: developing (/home/user/buildstream/doc/examples/developing)
Targets: hello.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-axv8wi9e/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-axv8wi9e
Log Files: /home/user/buildstream/doc/run-bst-axv8wi9e/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-axv8wi9e/sources
Build Area: /home/user/buildstream/doc/run-bst-axv8wi9e/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: developing
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
workspace: core plugin
tar: core plugin
Pipeline
cached b65366701aaab8b2f85afa7158f4036a1f55a8150db42fff45e0830cc8388d73 base/alpine.bst
cached 07db7692b3828f7021ae8c1b0b6f715801e64cf24fcb1621a67b47c110fc3c70 base.bst
cached 410bcb8e584bdeff697d7886018ece828e6d1139b86676ca21f1657fb98fa246 hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
===============================================================================
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 0, skipped 3, failed 0
Now running the hello command using bst shell:
user@host:~/developing$ bst shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732060.526545 1881 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][410bcb8e][ main:hello.bst ] START Staging dependencies
[00:00:00][410bcb8e][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][410bcb8e][ main:hello.bst ] START Integrating sandbox
[00:00:00][410bcb8e][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][410bcb8e][ main:hello.bst ] STATUS Running command
hello
Hello World
We can use workspaces!
This gives us the new message we changed in hello.c.
From this point we have several options. If the source is under version control we can commit our changes and push them to the remote repository.
1.3. Incremental builds
Once you have opened up your workspace, the workspace build directory will be reused for subsequent builds, which should improve your edit/compile/test cycle time when working with an open workspace.
In order to optimize incremental builds, BuildStream treats build configure steps
separately from the main build steps, and will only call the
Element.prepare() method on
an element plugin the first time it gets built. This avoids needlessly rebuilding
objects due to header files and such being unconditionally recreated by configuration
scripts (such as the typical ./configure script which is called for autotools
elements for instance).
A caveat of this optimization however is that changes you might make to the configuration scripts will not be taken into account by default on the next incremental build. A forced reconfiguration can also be required in some cases where build scripts automatically detect sources in it’s configuration phase, so newly added sources you add might be ignored.
In order to force the configuration step to be called again on the next build, you can use bst workspace reset –soft, like so:
In these cases, you can perform a hard reset on the workspace using bst workspace reset, like so:
user@host:~/developing$ bst workspace reset --soft hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732061.031870 1925 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] INFO Reset workspace state for hello.bst at: /home/user/buildstream/doc/examples/developing/workspace_hello
This will ensure that the next time you run the build, BuildStream will forcefully
rerun the Element.prepare() method
and cause the configuration step to occur again.
1.4. Closing your workspace
If we want to close the workspace and come back to our changes later, we can
user@host:~/developing$ bst workspace close hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732061.434564 1953 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] INFO Closed workspace for hello.bst
We can then reopen the workspace later using:
user@host:~/developing$ bst workspace open --no-checkout --directory workspace_hello hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732061.841632 1981 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] STATUS Creating workspace for element hello.bst
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element hello.bst
The –no-checkout option tells BuildStream not to check the source out but to instead hard-link to the workspace_hello directory.
Alternatively, if we wish to discard the changes we can use
user@host:~/developing$ bst workspace reset hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732062.238092 2009 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[00:00:00][ ][ main:core activity ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:core activity ] INFO Closed workspace for hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][ ][ main:core activity ] STATUS Creating workspace for element hello.bst
[--:--:--][a6a39926][ main:hello.bst ] START Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:hello.bst ] START Staging local files into CAS
[00:00:00][ ][ main:hello.bst ] SUCCESS Staging local files into CAS
[00:00:00][a6a39926][ main:hello.bst ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element hello.bst
This resets the workspace to its original state.
To discard the workspace completely we can do:
user@host:~/developing$ bst workspace close --remove-dir hello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732062.634733 2037 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[00:00:00][ ][ main:core activity ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][ ][ main:core activity ] INFO Closed workspace for hello.bst
This will close the workspace and completely remove the workspace_hello directory.