1. Workspaces

In this section we will cover the use of BuildStream’s workspaces feature when devloping a BuildStream project.

Note

This example is distributed with BuildStream in the doc/examples/developing subdirectory.

We will start with the project used in the running commands tutorial. Recall the element hello.bst, which builds the bellow C file:

/*
 * hello.c - Simple hello world program
 */
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("Hello World\n");
  return 0;
}

Suppose we now want to alter the functionality of the hello command. We can make changes to the source code of Buildstream elements by making use of BuildStream’s workspace command.

1.1. Opening a workspace

First we need to open a workspace, we can do this by running

user@host:~/developing$ bst workspace open --directory workspace_hello hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources
[--:--:--][        ][    main:core activity                 ] START   Preparing work plan
[00:00:00][        ][    main:core activity                 ] SUCCESS Preparing work plan
[--:--:--][        ][    main:core activity                 ] INFO    Creating workspace for element hello.bst
[--:--:--][e1dc9f80][    main:hello.bst                     ] START   Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
[00:00:00][e1dc9f80][    main:hello.bst                     ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element: hello.bst

This command has created the workspace_hello directory in which you can see the source for the hello.bst element, i.e. hello.c and the corresponding makefile.

You can view existing workspaces using

user@host:~/developing$ bst workspace list

workspaces:
- element: hello.bst
  directory: /home/user/buildstream/doc/examples/developing/workspace_hello

1.2. Making code changes

Let’s say we want to alter the message printed when the hello command is run. We can open workspace_hello/hello.c and make the following change:

-- hello.c	2018-06-25 14:48:32.077568920 +0100
+++ hello.c	2018-06-25 14:49:23.025553785 +0100
@@ -5,6 +5,6 @@
 
 int main(int argc, char *argv[])
 {
-  printf("Hello World\n");
+  printf("Hello World\nWe can use workspaces!\n");
   return 0;
 }

Now, rebuild the hello.bst element.

user@host:~/developing$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][        ][    main:core activity                 ] START   Preparing work plan
[00:00:00][        ][    main:core activity                 ] SUCCESS Preparing work plan

BuildStream Version 1.93.6+36.ga935429a
    Session Start: Thursday, 25-03-2021 at 15:42:20
    Project:       developing (/home/user/buildstream/doc/examples/developing)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-9ab9rep4/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-9ab9rep4
    Log Files:               /home/user/buildstream/doc/run-bst-9ab9rep4/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-9ab9rep4/sources
    Build Area:              /home/user/buildstream/doc/run-bst-9ab9rep4/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: developing

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        workspace: core plugin
        tar:       core plugin

Pipeline
   buildable b279a6f973c904fb098b039e32a29ac18d9d853345feb6a4612bca1bf9249ca0 base/alpine.bst 
     waiting 15c40959b4dfac10a4022b658a6d2764e5f578841328e17dcea9c807092263f5 base.bst 
     waiting 79f4a2966ac9cd8617f6a4f8d35a9ac75e5c96e9c58892614477f7c6d60b619f hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
===============================================================================
[--:--:--][b279a6f9][   build:base/alpine.bst               ] START   developing/base-alpine/b279a6f9-build.742.log
[--:--:--][b279a6f9][   build:base/alpine.bst               ] START   Staging sources
[--:--:--][79f4a296][   fetch:hello.bst                     ] START   developing/hello/79f4a296-fetch.742.log
[00:00:00][79f4a296][   fetch:hello.bst                     ] SUCCESS developing/hello/79f4a296-fetch.742.log
[00:00:00][b279a6f9][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][b279a6f9][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][b279a6f9][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][b279a6f9][   build:base/alpine.bst               ] SUCCESS developing/base-alpine/b279a6f9-build.742.log
[--:--:--][15c40959][   build:base.bst                      ] START   developing/base/15c40959-build.742.log
[--:--:--][15c40959][   build:base.bst                      ] START   Caching artifact
[00:00:00][15c40959][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][15c40959][   build:base.bst                      ] SUCCESS developing/base/15c40959-build.742.log
[--:--:--][79f4a296][   build:hello.bst                     ] START   developing/hello/79f4a296-build.742.log
[--:--:--][79f4a296][   build:hello.bst                     ] START   Staging dependencies at: /
[00:00:00][79f4a296][   build:hello.bst                     ] SUCCESS Staging dependencies at: /
[--:--:--][79f4a296][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][79f4a296][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][79f4a296][   build:hello.bst                     ] START   Staging sources
[00:00:00][79f4a296][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][79f4a296][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:02][79f4a296][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][79f4a296][   build:hello.bst                     ] START   Caching artifact
[00:00:00][79f4a296][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:02][79f4a296][   build:hello.bst                     ] SUCCESS developing/hello/79f4a296-build.742.log
[00:00:02][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 1, skipped 2, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

Note that if you run the command from inside the workspace, the element name is optional.

user@host:~/workspace_hello$ bst build

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][        ][    main:core activity                 ] START   Preparing work plan
[00:00:00][        ][    main:core activity                 ] SUCCESS Preparing work plan

BuildStream Version 1.93.6+36.ga935429a
    Session Start: Thursday, 25-03-2021 at 15:42:23
    Project:       developing (/home/user/buildstream/doc/examples/developing)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-9ab9rep4/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-9ab9rep4
    Log Files:               /home/user/buildstream/doc/run-bst-9ab9rep4/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-9ab9rep4/sources
    Build Area:              /home/user/buildstream/doc/run-bst-9ab9rep4/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: developing

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        workspace: core plugin
        tar:       core plugin

Pipeline
      cached b279a6f973c904fb098b039e32a29ac18d9d853345feb6a4612bca1bf9249ca0 base/alpine.bst 
      cached 15c40959b4dfac10a4022b658a6d2764e5f578841328e17dcea9c807092263f5 base.bst 
      cached 79f4a2966ac9cd8617f6a4f8d35a9ac75e5c96e9c58892614477f7c6d60b619f hello.bst Workspace: /home/user/buildstream/doc/examples/developing/workspace_hello
===============================================================================
[00:00:00][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 0, skipped 3, failed 0 
    Build Queue: processed 0, skipped 3, failed 0

Now running the hello command using bst shell:

user@host:~/developing$ bst shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][79f4a296][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][79f4a296][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][79f4a296][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][79f4a296][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][79f4a296][    main:hello.bst                     ] STATUS  Running command

    hello

WARNING: The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2021-03-25T15:42:24.090+0000 [828:140172167749440] [buildboxcommon_client.cpp:86] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello World
We can use workspaces!

This gives us the new message we changed in hello.c.

From this point we have several options. If the source is under version control we can commit our changes and push them to the remote repository.

1.3. Making bigger changes

The first time you build with your workspace BuildStream behaves very similarly to normal. However for subsequent builds it does not run the configuration commands and only runs the build commands. This can result in significant speed ups as the configuration commands can be slow (maybe 20+ minutes for a moderate sized element)

Sometimes you do need to rerun the configuration command for a open workspace, this can be done by resetting the workspace, and then running a build. However this will cause all of your incremental work to be lost. In this case running a soft reset will reset the trigger to run the configuration commands but will not change any files in you workspace.

Reasons to soft reset a workspace include:
  • changing the configuration command of a element.

  • adding a new source that the configure command will spot and enable more code.

The soft reset can be performed by:

user@host:~/developing$ bst workspace reset --soft hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] INFO    Reset workspace state for hello.bst at: /home/user/buildstream/doc/examples/developing/workspace_hello

Then the next build will include the configuration commands. You must reset the workspace every time you wish to trigger the configuration commands as only the first build after the reset will run them.

1.4. Closing your workspace

If we want to close the workspace and come back to our changes later, we can

user@host:~/developing$ bst workspace close hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst

We can then reopen the workspace later using:

user@host:~/developing$ bst workspace open --no-checkout --directory workspace_hello hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] INFO    Creating workspace for element hello.bst
[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element: hello.bst

The –no-checkout option tells BuildStream not to check the source out but to instead hard-link to the workspace_hello directory.

Alternatively, if we wish to discard the changes we can use

user@host:~/developing$ bst workspace reset hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[00:00:00][        ][    main:core activity                 ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources
[--:--:--][        ][    main:core activity                 ] START   Preparing work plan
[00:00:00][        ][    main:core activity                 ] SUCCESS Preparing work plan
[--:--:--][        ][    main:core activity                 ] INFO    Creating workspace for element hello.bst
[--:--:--][e1dc9f80][    main:hello.bst                     ] START   Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files into CAS
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files into CAS
[00:00:00][e1dc9f80][    main:hello.bst                     ] SUCCESS Staging sources to /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:core activity                 ] INFO    Created a workspace for element: hello.bst

This resets the workspace to its original state.

To discard the workspace completely we can do:

user@host:~/developing$ bst workspace close --remove-dir hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[--:--:--][        ][    main:hello.bst                     ] START   Staging local files
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging local files
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[00:00:00][        ][    main:core activity                 ] SUCCESS Removing workspace directory /home/user/buildstream/doc/examples/developing/workspace_hello
[--:--:--][        ][    main:core activity                 ] INFO    Closed workspace for hello.bst

This will close the workspace and completely remove the workspace_hello directory.