Remote Execution Servers

BuildStream supports building remotely using the Google Remote Execution API (REAPI)., which has various known implementations.

Some of these implementations include:

These various implementations implement the Google Remote Execution API (REAPI) to various degrees as these projects have different priorities.

On the client side, the remote execution service to use can be specified in the user configuration.

BuildStream specific requirements

In order for BuildStream to work correctly with a remote execution cluster, there are a couple of requirements that implementation needs to meet.

Implementation of platform properties

The remote execution service must properly implement platform properties.

This is crucial because BuildStream needs to be guaranteed the correct operating system and ISA which it requests from the service.

Staging the input root as the filesystem root

BuildStream requires that the input root given to the remote execution service be treated as the absolute filesystem root.

This is because BuildStream provides guarantees that all build dependencies, including the base runtime and compilers, are defined by elements and run within a sandboxed and isolated build environment, but the REAPI was originally developped without this determinism and control in mind. Instead, typically it is up to the user to configure a cluster to use a docker image to build payloads with, rather than allowing the REAPI client to control the entire sandbox.

Unfortunately the ability to dictate that the input root be treated as the filesystem root in a container on remote workers in the cluster is not yet standardized in the REAPI protocol.

Note

The input root is referred to as the input_root_digest member of the Action message as defined in the protocol

Example working configuration

A simple configuration to spin up the BuildGrid service using docker compose follows:

##
# BuildGrid Compose manifest for BuildStream.
#
# Spins-up a unnamed and unauthenticated grid:
#  - Controller + CAS + AC at http://localhost:50051
#  - Ref. + CAS at: http://localhost:50052
#
# BuildStream configuration snippet:
#
#    artifacts:
#      url: http://localhost:50052
#      push: true
#    remote-execution:
#      execution-service:
#        url: http://localhost:50051
#      action-cache-service:
#        url: http://localhost:50051
#      storage-service:
#        url: http://localhost:50051
#
# Basic usage:
#  - docker-compose -f ci.buildgrid.yml up
#  - docker-compose -f ci.buildgrid.yml down
#
version: "3.2"

services:
  controller:
    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
    command: [
      "bgd", "server", "start", "-v",
      "/etc/buildgrid/default.yml"]
    ports:
      - 50051:50051
    networks:
      - grid

  bot:
    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildbox:nightly
    command: [
      "sh", "-c", "sleep 15 && ( buildbox-casd --cas-remote=http://controller:50051 /var/lib/buildgrid/cache & buildbox-worker --request-timeout=30 --bots-remote=http://controller:50051 --cas-remote=unix:/var/lib/buildgrid/cache/casd.sock --buildbox-run=buildbox-run-bubblewrap --platform OSFamily=linux --platform ISA=x86-64 --verbose )"]
    privileged: true
    volumes:
      - type: volume
        source: cache
        target: /var/lib/buildgrid/cache
    depends_on:
      - controller
    networks:
      - grid

  storage:
    image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
    command: [
      "bgd", "server", "start", "-v",
      "/etc/buildgrid/artifacts.yml"]
    ports:
      - 50052:50052
    networks:
      - grid

networks:
  grid:
    driver: bridge

volumes:
  cache: