Plugin - Base plugin class

BuildStream supports third party plugins to define additional kinds of Elements and Sources.

The common API is documented here, along with some information on how external plugin packages are structured.

Abstract Methods

For both Elements and Sources, it is mandatory to implement the following abstract methods:

Plugin.configure()

Loads the user provided configuration YAML for the given source or element
Plugin.preflight()

Early preflight checks allow plugins to bail out early with an error in the case that it can predict that failure is inevitable.
Plugin.get_unique_key()

Once all configuration has been loaded and preflight checks have passed, this method is used to inform the core of a plugin’s unique configuration.

Configurable Warnings

Warnings raised through calling Plugin.warn() can provide an optional parameter warning_token, this will raise a PluginError if the warning is configured as fatal within the project configuration.

Configurable warnings will be prefixed with Plugin.get_kind() within buildstream and must be prefixed as such in project configurations. For more detail on project configuration see Configurable Warnings.

It is important to document these warnings in your plugin documentation to allow users to make full use of them while configuring their projects.

Example

If the git plugin uses the warning "inconsistent-submodule" then it could be referenced in project configuration as "git:inconsistent-submodule".

Plugin Structure

A plugin should consist of a setuptools package that advertises contained plugins using entry points.

A plugin entry point must be a module that extends a class in the Plugin API reference to be discovered by BuildStream. A YAML file defining plugin default settings with the same name as the module can also be defined in the same directory as the plugin module.

Note

BuildStream does not support function/class entry points.

A sample plugin could be structured as such:

.
├── elements
│   ├── autotools.py
│   ├── autotools.yaml
│   └── __init__.py
├── MANIFEST.in
└── setup.py

The setuptools configuration should then contain at least:

setup.py:

from setuptools import setup, find_packages

setup(name='BuildStream Autotools',
      version="0.1",
      description="A better autotools element for BuildStream",
      packages=find_packages(),
      install_requires=[
          'setuptools'
      ],
      include_package_data=True,
      entry_points={
          'buildstream.plugins': [
              'autotools = elements.autotools'
          ]
      })

MANIFEST.in:

global-include *.yaml

Class Reference

class Plugin

Bases: object

Base Plugin class.

Some common features to both Sources and Elements are found in this class.

Note

Derivation of plugins is not supported. Plugins may only derive from the base Source and Element types, and any convenience subclasses (like BuildElement) which are included in the buildstream namespace.

BST_REQUIRED_VERSION_MAJOR = 0: Minimum required major version

BST_REQUIRED_VERSION_MINOR = 0: Minimum required minor version

BST_FORMAT_VERSION = 0

The plugin’s YAML format version

This should be set to 1 the first time any new configuration is understood by your Plugin.configure() implementation and subsequently bumped every time your configuration is enhanced.

Note

Plugins are expected to maintain backward compatibility in the format and configurations they expose. The versioning is intended to track availability of new features only.

For convenience, the format version for plugins maintained and distributed with BuildStream are revisioned with BuildStream’s core format version core format version.

name

The plugin name

For elements, this is the project relative bst filename, for sources this is the owning element’s name with a suffix indicating it’s index on the owning element.

For sources this is for display purposes only.

configure(node)

Configure the Plugin from loaded configuration data

Parameters:

node (dict) – The loaded configuration dictionary

Raises:

.SourceError – If its a Source implementation
.ElementError – If its an Element implementation

Plugin implementors should implement this method to read configuration data and store it.

Plugins should use the Plugin.node_get_member() and Plugin.node_get_list_element() methods to fetch values from the passed node. This will ensure that a nice human readable error message will be raised if the expected configuration is not found, indicating the filename, line and column numbers.

Further the Plugin.node_validate() method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.

Note

For Elements, when variable substitution is desirable, the Element.node_subst_member() and Element.node_subst_list_element() methods can be used.

preflight()

Preflight Check

Raises:

.SourceError – If its a Source implementation
.ElementError – If its an Element implementation

This method is run after Plugin.configure() and after the pipeline is fully constructed.

Implementors should simply raise SourceError or ElementError with an informative message in the case that the host environment is unsuitable for operation.

Plugins which require host tools (only sources usually) should obtain them with utils.get_host_tool() which will raise an error automatically informing the user that a host tool is needed.

get_unique_key()

Return something which uniquely identifies the plugin input

Returns:: A string, list or dictionary which uniquely identifies the input

This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.

Elements and Sources should implement this by collecting any configurations which could possibly effect the output and return a dictionary of these settings.

For Sources, this is guaranteed to only be called if Source.get_consistency() has not returned Consistency.INCONSISTENT which is to say that the Source is expected to have an exact ref indicating exactly what source is going to be staged.

get_kind()

Fetches the kind of this plugin

Returns:: The kind of this plugin
Return type:: (str)

node_items(node)

Iterate over a dictionary loaded from YAML

Parameters:: node (dict) – The YAML loaded dictionary object
Returns:: List of key/value tuples to iterate over
Return type:: list

BuildStream holds some private data in dictionaries loaded from the YAML in order to preserve information to report in errors.

This convenience function should be used instead of the dict.items() builtin function provided by python.

node_provenance(node, member_name=None)

Gets the provenance for node and member_name

This reports a string with file, line and column information suitable for reporting an error or warning.

Parameters:

node (dict) – The YAML loaded dictionary object
member_name (str) – The name of the member to check, or None for the node itself

Returns:

A string describing the provenance of the node and member

Return type:

(str)

node_get_member(node, expected_type, member_name, default=<object object>)

Fetch the value of a node member, raising an error if the value is missing or incorrectly typed.

Parameters:

node (dict) – A dictionary loaded from YAML
expected_type (type) – The expected type of the node member
member_name (str) – The name of the member to fetch
default (expected_type) – A value to return when member_name is not specified in node

Returns:

The value of member_name in node, otherwise default

Raises:

.LoadError – When member_name is not found and no default was provided

Note

Returned strings are stripped of leading and trailing whitespace

Example:

# Expect a string 'name' in 'node'
name = self.node_get_member(node, str, 'name')

# Fetch an optional integer
level = self.node_get_member(node, int, 'level', -1)

node_get_project_path(node, key, *, check_is_file=False, check_is_dir=False)

Fetches a project path from a dictionary node and validates it

Paths are asserted to never lead to a directory outside of the project directory. In addition, paths can not point to symbolic links, fifos, sockets and block/character devices.

The check_is_file and check_is_dir parameters can be used to perform additional validations on the path. Note that an exception will always be raised if both parameters are set to True.

Parameters:

node (dict) – A dictionary loaded from YAML
key (str) – The key whose value contains a path to validate
check_is_file (bool) – If True an error will also be raised if path does not point to a regular file. Defaults to False
check_is_dir (bool) – If True an error will also be raised if path does not point to a directory. Defaults to False

Returns:

The project path

Return type:

(str)

Raises:

.LoadError – In the case that the project path is not valid or does not exist

Since: 1.2

Example:

path = self.node_get_project_path(node, 'path')

node_validate(node, valid_keys)

This should be used in configure() implementations to assert that users have only entered valid configuration keys.

Parameters:

node (dict) – A dictionary loaded from YAML
valid_keys (iterable) – A list of valid keys for the node

Raises:

.LoadError – When an invalid key is found

Example:

# Ensure our node only contains valid autotools config keys
self.node_validate(node, [
    'configure-commands', 'build-commands',
    'install-commands', 'strip-commands'
])

node_get_list_element(node, expected_type, member_name, indices)

Fetch the value of a list element from a node member, raising an error if the value is incorrectly typed.

Parameters:

node (dict) – A dictionary loaded from YAML
expected_type (type) – The expected type of the node member
member_name (str) – The name of the member to fetch
indices (list of int) – List of indices to search, in case of nested lists

Returns:

The value of the list element in member_name at the specified indices

Raises:

.LoadError –

Note

Returned strings are stripped of leading and trailing whitespace

Example:

# Fetch the list itself
things = self.node_get_member(node, list, 'things')

# Iterate over the list indices
for i in range(len(things)):

    # Fetch dict things
    thing = self.node_get_list_element(
        node, dict, 'things', [ i ])

debug(brief, *, detail=None)

Print a debugging message

Parameters:

brief (str) – The brief message
detail (str) – An optional detailed message, can be multiline output

status(brief, *, detail=None)

Print a status message

Parameters:

brief (str) – The brief message
detail (str) – An optional detailed message, can be multiline output

Note: Status messages tell about what a plugin is currently doing

info(brief, *, detail=None)

Print an informative message

Parameters:

brief (str) – The brief message
detail (str) – An optional detailed message, can be multiline output

Note: Informative messages tell the user something they might want: to know, like if refreshing an element caused it to change.

warn(brief, *, detail=None, warning_token=None)

Print a warning message, checks warning_token against project configuration

Parameters:

brief (str) – The brief message
detail (str) – An optional detailed message, can be multiline output
warning_token (str) – An optional configurable warning assosciated with this warning, this will cause PluginError to be raised if this warning is configured as fatal. (Since 1.4)

:raises (PluginError): When warning_token is considered fatal by the project configuration

log(brief, *, detail=None)

Log a message into the plugin’s log file

The message will not be shown in the master log at all (so it will not be displayed to the user on the console).

Parameters:

brief (str) – The brief message
detail (str) – An optional detailed message, can be multiline output

timed_activity(activity_name, *, detail=None, silent_nested=False)

Context manager for performing timed activities in plugins

Parameters:

activity_name (str) – The name of the activity
detail (str) – An optional detailed message, can be multiline output
silent_nested (bool) – If specified, nested messages will be silenced

This function lets you perform timed tasks in your plugin, the core will take care of timing the duration of your task and printing start / fail / success messages.

Example

# Activity will be logged and timed
with self.timed_activity("Mirroring {}".format(self.url)):

    # This will raise SourceError on its own
    self.call(... command which takes time ...)

call(*popenargs, fail=None, fail_temporarily=False, **kwargs)

A wrapper for subprocess.call()

Parameters:

popenargs (list) – Popen() arguments
fail (str) – A message to display if the process returns a non zero exit code
fail_temporarily (bool) – Whether any exceptions should be raised as temporary. (Since: 1.2)
rest_of_args (kwargs) – Remaining arguments to subprocess.call()

Returns:

The process exit code.

Return type:

(int)

:raises (PluginError): If a non-zero return code is received and fail is specified

Note: If fail is not specified, then the return value of subprocess.call(): is returned even on error, and no exception is automatically raised.

Example

# Call some host tool
self.tool = utils.get_host_tool('toolname')
self.call(
    [self.tool, '--download-ponies', self.mirror_directory],
    "Failed to download ponies from {}".format(
        self.mirror_directory))

check_output(*popenargs, fail=None, fail_temporarily=False, **kwargs)

A wrapper for subprocess.check_output()

Parameters:

popenargs (list) – Popen() arguments
fail (str) – A message to display if the process returns a non zero exit code
fail_temporarily (bool) – Whether any exceptions should be raised as temporary. (Since: 1.2)
rest_of_args (kwargs) – Remaining arguments to subprocess.call()

Returns:

The process exit code (str): The process standard output

Return type:

(int)

:raises (PluginError): If a non-zero return code is received and fail is specified

Note: If fail is not specified, then the return value of subprocess.check_output(): is returned even on error, and no exception is automatically raised.

Example

# Get the tool at preflight time
self.tool = utils.get_host_tool('toolname')

# Call the tool, automatically raise an error
_, output = self.check_output(
    [self.tool, '--print-ponies'],
    "Failed to print the ponies in {}".format(
        self.mirror_directory),
    cwd=self.mirror_directory)

# Call the tool, inspect exit code
exit_code, output = self.check_output(
    [self.tool, 'get-ref', tracking],
    cwd=self.mirror_directory)

if exit_code == 128:
    return
elif exit_code != 0:
    fmt = "{plugin}: Failed to get ref for tracking: {track}"
    raise SourceError(
        fmt.format(plugin=self, track=tracking)) from e