Remote Execution Servers
BuildStream supports building remotely using the Google Remote Execution API (REAPI)., which has various known implementations.
Some of these implementations include:
These various implementations implement the Google Remote Execution API (REAPI) to various degrees as these projects have different priorities.
On the client side, the remote execution service to use can be specified in the user configuration.
BuildStream specific requirements
In order for BuildStream to work correctly with a remote execution cluster, there are a couple of requirements that implementation needs to meet.
Implementation of platform properties
The remote execution service must properly implement platform properties.
This is crucial because BuildStream needs to be guaranteed the correct operating system and ISA which it requests from the service.
Staging the input root as the filesystem root
BuildStream requires that the input root given to the remote execution service be treated as the absolute filesystem root.
This is because BuildStream provides guarantees that all build dependencies, including the base runtime and compilers, are defined by elements and run within a sandboxed and isolated build environment, but the REAPI was originally developped without this determinism and control in mind. Instead, typically it is up to the user to configure a cluster to use a docker image to build payloads with, rather than allowing the REAPI client to control the entire sandbox.
Unfortunately the ability to dictate that the input root be treated as the filesystem root in a container on remote workers in the cluster is not yet standardized in the REAPI protocol.
The input root is referred to as the
input_root_digest member of the
as defined in the protocol
Example working configuration
## # BuildGrid Compose manifest for BuildStream. # # Spins-up a unnamed and unauthenticated grid: # - Controller + CAS + AC at http://localhost:50051 # - Ref. + CAS at: http://localhost:50052 # # BuildStream configuration snippet: # # artifacts: # url: http://localhost:50052 # push: true # remote-execution: # execution-service: # url: http://localhost:50051 # action-cache-service: # url: http://localhost:50051 # storage-service: # url: http://localhost:50051 # # Basic usage: # - docker-compose -f ci.buildgrid.yml up # - docker-compose -f ci.buildgrid.yml down # version: "3.2" services: controller: image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly command: [ "bgd", "server", "start", "-v", "/etc/buildgrid/default.yml"] ports: - 50051:50051 networks: - grid bot: image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildbox:nightly command: [ "sh", "-c", "sleep 15 && ( buildbox-casd --cas-remote=http://controller:50051 /var/lib/buildgrid/cache & buildbox-worker --request-timeout=30 --bots-remote=http://controller:50051 --cas-remote=unix:/var/lib/buildgrid/cache/casd.sock --buildbox-run=buildbox-run-bubblewrap --platform OSFamily=linux --platform ISA=x86-64 --verbose )"] privileged: true volumes: - type: volume source: cache target: /var/lib/buildgrid/cache depends_on: - controller networks: - grid storage: image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly command: [ "bgd", "server", "start", "-v", "/etc/buildgrid/artifacts.yml"] ports: - 50052:50052 networks: - grid networks: grid: driver: bridge volumes: cache: