Plugin - Base plugin class

BuildStream supports third party plugins to define additional kinds of Elements and Sources.

The common API is documented here, along with some information on how external plugin packages are structured.

Abstract Methods

For both Elements and Sources, it is mandatory to implement the following abstract methods:

Plugin.configure()

Loads the user provided configuration YAML for the given source or element
Plugin.preflight()

Early preflight checks allow plugins to bail out early with an error in the case that it can predict that failure is inevitable.
Plugin.get_unique_key()

Once all configuration has been loaded and preflight checks have passed, this method is used to inform the core of a plugin’s unique configuration.

Configurable Warnings

Warnings raised through calling Plugin.warn() can provide an optional parameter warning_token, this will raise a PluginError if the warning is configured as fatal within the project configuration.

Configurable warnings will be prefixed with Plugin.get_kind() within buildstream and must be prefixed as such in project configurations. For more detail on project configuration see Configurable Warnings.

It is important to document these warnings in your plugin documentation to allow users to make full use of them while configuring their projects.

Example

If the git plugin uses the warning "inconsistent-submodule" then it could be referenced in project configuration as "git:inconsistent-submodule".

Plugin Structure

A plugin should consist of a setuptools package that advertises contained plugins using entry points.

A plugin entry point must be a module that extends a class in the Plugin API reference to be discovered by BuildStream. A YAML file defining plugin default settings with the same name as the module can also be defined in the same directory as the plugin module.

Note

BuildStream does not support function/class entry points.

A sample plugin could be structured as such:

.
├── elements
│   ├── autotools.py
│   ├── autotools.yaml
│   └── __init__.py
├── MANIFEST.in
└── setup.py

The setuptools configuration should then contain at least:

setup.py:

from setuptools import setup, find_packages

setup(name='BuildStream Autotools',
      version="0.1",
      description="A better autotools element for BuildStream",
      packages=find_packages(),
      include_package_data=True,
      entry_points={
          'buildstream.plugins.elements': [
              'autotools = elements.autotools'
          ]
      })

MANIFEST.in:

global-include *.yaml

Class Reference

class Plugin

Bases: object

Base Plugin class.

Some common features to both Sources and Elements are found in this class.

Note

Derivation of plugins is not supported. Plugins may only derive from the base Source and Element types, and any convenience subclasses (like BuildElement) which are included in the buildstream namespace.

BST_MIN_VERSION: str | None = None

The minimum required version of BuildStream required by this plugin.

The version must be expressed as the string “<major>.<minor>”, where the major version number is the API version and the minor version number is the revision of the same BuildStream API where new symbols might have been added to the API.

Example:

The following statement means that this plugin works with BuildStream 2.X, only if X >= 2:

class Foo(Source):

    # Our plugin requires 2.2
    BST_MIN_VERSION = "2.2"

Note

This version works exactly the same was as the min-version which must be specified in the project.conf file.

BST_PLUGIN_DEPRECATED = False

True if this element plugin has been deprecated.

If this is set to true, BuildStream will emit a deprecation warning in any place where this plugin is used.

The deprecation warnings can be suppressed when defining the plugin origins in your project configuration

BST_PLUGIN_DEPRECATION_MESSAGE = None

An additional message to report when a plugin is deprecated

This can be used to refer the user to a suitable replacement or alternative approach when the plugin is deprecated.

name

The plugin name

For elements, this is the project relative bst filename, for sources this is the owning element’s name with a suffix indicating its index on the owning element.

For sources this is for display purposes only.

configure(node: MappingNode) → None

Configure the Plugin from loaded configuration data

Parameters:

node – The loaded configuration dictionary

Raises:

.SourceError – If it’s a Source implementation
.ElementError – If it’s an Element implementation

Plugin implementors should implement this method to read configuration data and store it.

The MappingNode.validate_keys() method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.

preflight() → None

Preflight Check

Raises:

.SourceError – If it’s a Source implementation
.ElementError – If it’s an Element implementation

This method is run after Plugin.configure() and after the pipeline is fully constructed.

Implementors should simply raise SourceError or ElementError with an informative message in the case that the host environment is unsuitable for operation.

Plugins which require host tools (only sources usually) should obtain them with utils.get_host_tool() which will raise an error automatically informing the user that a host tool is needed.

get_unique_key() → None | int | str | List[Any] | Dict[str, Any]

Return something which uniquely identifies the plugin input

Returns:: A string, list or dictionary which uniquely identifies the input

This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.

Elements and Sources should implement this by collecting any configurations which could possibly affect the output and return a dictionary of these settings.

For Sources, this is guaranteed to only be called if Source.is_resolved() has returned True which is to say that the Source is expected to have an exact source ref indicating exactly what source is going to be staged.

Note

If your plugin is concerned with API stability, then future extensions of your plugin YAML configuration which affect the unique key returned here should be added to this key with care.

A good rule of thumb is to only compute the new value in the returned key if the value of the newly added YAML key is not equal to it’s default value.

get_kind() → str

Fetches the kind of this plugin

Returns:: The kind of this plugin

node_get_project_path(node, *, check_is_file=False, check_is_dir=False) → str

Fetches a project path from a dictionary node and validates it

Paths are asserted to never lead to a directory outside of the project directory. In addition, paths can not point to symbolic links, fifos, sockets and block/character devices.

The check_is_file and check_is_dir parameters can be used to perform additional validations on the path. Note that an exception will always be raised if both parameters are set to True.

Parameters:

node (ScalarNode) – A Node loaded from YAML containing the path to validate
check_is_file (bool) – If True an error will also be raised if path does not point to a regular file. Defaults to False
check_is_dir (bool) – If True an error will also be raised if path does not point to a directory. Defaults to False

Returns:

The project path

Return type:

(str)

Raises:

.LoadError – In the case that the project path is not valid or does not exist

Example:

path = self.node_get_project_path(node, 'path')

debug(brief: str, *, detail: str | None = None) → None

Print a debugging message

Parameters:

brief – The brief message
detail – An optional detailed message, can be multiline output

status(brief: str, *, detail: str | None = None) → None

Print a status message

Parameters:

brief – The brief message
detail – An optional detailed message, can be multiline output

Note: Status messages tell about what a plugin is currently doing

info(brief: str, *, detail: str | None = None) → None

Print an informative message

Parameters:

brief – The brief message
detail – An optional detailed message, can be multiline output

Note: Informative messages tell the user something they might want: to know, like if refreshing an element caused it to change. The instance full name of the plugin will be generated with the message, this being the name of the given element, as appose to the class name of the underlying plugin __kind identifier.

warn(brief: str, *, detail: str | None = None, warning_token: str | None = None) → None

Print a warning message, checks warning_token against project configuration

Parameters:

brief – The brief message
detail – An optional detailed message, can be multiline output
warning_token – An optional configurable warning assosciated with this warning, this will cause PluginError to be raised if this warning is configured as fatal.

:raises (PluginError): When warning_token is considered fatal by the project configuration

log(brief: str, *, detail: str | None = None) → None

Log a message into the plugin’s log file

The message will not be shown in the master log at all (so it will not be displayed to the user on the console).

Parameters:

brief – The brief message
detail – An optional detailed message, can be multiline output

timed_activity(activity_name: str, *, detail: str | None = None, silent_nested: bool = False) → Generator[None, None, None]

Context manager for performing timed activities in plugins

Parameters:

activity_name – The name of the activity
detail – An optional detailed message, can be multiline output
silent_nested – If specified, nested messages will be silenced

This function lets you perform timed tasks in your plugin, the core will take care of timing the duration of your task and printing start / fail / success messages.

Example

# Activity will be logged and timed
with self.timed_activity("Mirroring {}".format(self.url)):

    # This will raise SourceError on its own
    self.call(... command which takes time ...)

blocking_activity(target: Callable[[...], T1], args: Sequence[Any], activity_name: str, *, detail: str | None = None, silent_nested: bool = False) → T1

Execute a blocking activity in the background.

This is to execute potentially blocking methods in the background, in order to avoid starving the scheduler.

The function, its arguments and return value must all be pickleable, as it will be run in another process. The function should not raise an exception.

This should be used whenever there is a potential for a blocking syscall to not return in a reasonable (<1s) amount of time. For example, you would use this if you were doing a request to a remote server, without a timeout.

Parameters:

target – the function to execute in the background
args – positional arguments to pass to the method to call
activity_name – The name of the activity
detail – An optional detailed message, can be multiline output
silent_nested – If specified, nested messages will be silenced

Returns:

the return value from target.

A wrapper for subprocess.call()

Parameters:

args – A sequence of program arguments or else a string or path-like-object
fail – A message to display if the process returns a non zero exit code
fail_temporarily – Whether any exceptions should be raised as temporary.
cwd – Optionally specify the working directory for the command
env – Optionally specify some environment variables for the command
stdin – Optionally specify standard input for the command
stdout – Optionally specify standard output for the command
stderr – Optionally specify standard error for the command

Returns:

The process exit code.

:raises (PluginError): If a non-zero return code is received and fail is specified

Attention

If fail is not specified, then the return value of subprocess.call() is returned even on error, and no exception is automatically raised.

Attention

If stderr and/or stdout are specified, then these streams are delivered directly to the plugin instead of being delivered to the associated log file.

Example

# Call some host tool
self.tool = utils.get_host_tool('toolname')
self.call(
    [self.tool, '--download-ponies', self.mirror_directory],
    "Failed to download ponies from {}".format(
        self.mirror_directory))

A wrapper for subprocess.check_output()

Parameters:

args –
A sequence of program arguments or else a string or path-like-object
fail – A message to display if the process returns a non zero exit code
fail_temporarily – Whether any exceptions should be raised as temporary.
cwd – Optionally specify the working directory for the command
env – Optionally specify some environment variables for the command
stdin – Optionally specify standard input for the command
stderr – Optionally specify standard error for the command

Returns:

A 2-tuple of form (process exit code, process standard output)

:raises (PluginError): If a non-zero return code is received and fail is specified

Attention

If fail is not specified, then the return value of subprocess.call() is returned even on error, and no exception is automatically raised.

Attention

If stderr and/or stdout are specified, then these streams are delivered directly to the plugin instead of being delivered to the associated log file.

Example

# Get the tool at preflight time
self.tool = utils.get_host_tool('toolname')

# Call the tool, automatically raise an error
_, output = self.check_output(
    [self.tool, '--print-ponies'],
    "Failed to print the ponies in {}".format(
        self.mirror_directory),
    cwd=self.mirror_directory)

# Call the tool, inspect exit code
exit_code, output = self.check_output(
    [self.tool, 'get-ref', tracking],
    cwd=self.mirror_directory)

if exit_code == 128:
    return
elif exit_code != 0:
    fmt = "{plugin}: Failed to get ref for tracking: {track}"
    raise SourceError(
        fmt.format(plugin=self, track=tracking)) from e