Source - Base source class¶

Abstract Methods¶

For loading and configuration purposes, Sources must implement the Plugin base class abstract methods.

Attention

In order to ensure that all configuration data is processed at load time, it is important that all URLs have been processed during Plugin.configure().

Source implementations must either call Source.translate_url() or Source.mark_download_url() for every URL that has been specified in the configuration during Plugin.configure()

Sources expose the following abstract methods. Unless explicitly mentioned, these methods are mandatory to implement.

Source.get_consistency()

Report the sources consistency state.
Source.load_ref()

Load the ref from a specific YAML node
Source.get_ref()

Fetch the source ref
Source.set_ref()

Set a new ref explicitly
Source.track()

Automatically derive a new ref from a symbolic tracking branch
Source.fetch()

Fetch the actual payload for the currently set ref
Source.stage()

Stage the sources for a given ref at a specified location
Source.init_workspace()

Stage sources in a local directory for use as a workspace.

Optional: If left unimplemented, this will default to calling Source.stage()
Source.get_source_fetchers()

Get the objects that are used for fetching.

Optional: This only needs to be implemented for sources that need to download from multiple URLs while fetching (e.g. a git repo and its submodules). For details on how to define a SourceFetcher, see SourceFetcher.
Source.validate_cache()

Perform any validations which require the sources to be cached.

Optional: This is completely optional and will do nothing if left unimplemented.

Accessing previous sources¶

Since: 1.4

In the general case, all sources are fetched and tracked independently of one another. In situations where a source needs to access previous source(s) in order to perform its own track and/or fetch, following attributes can be set to request access to previous sources:

BST_REQUIRES_PREVIOUS_SOURCES_TRACK

Indicate that access to previous sources is required during track
BST_REQUIRES_PREVIOUS_SOURCES_FETCH

Indicate that access to previous sources is required during fetch

The intended use of such plugins is to fetch external dependencies of other sources, typically using some kind of package manager, such that all the dependencies of the original source(s) are available at build time.

When implementing such a plugin, implementors should adhere to the following guidelines:

Implementations must be able to store the obtained artifacts in a subdirectory.
Implementations must be able to deterministically generate a unique ref, such that two refs are different if and only if they produce different outputs.
Implementations must not introduce host contamination.

SourceFetcher - Object for fetching individual URLs¶

Abstract Methods¶

SourceFetchers expose the following abstract methods. Unless explicitly mentioned, these methods are mandatory to implement.

SourceFetcher.fetch()

Fetches the URL associated with this SourceFetcher, optionally taking an alias override.

Class Reference¶

class Consistency¶

Bases: object

INCONSISTENT = 0¶

Inconsistent

Inconsistent sources have no explicit reference set. They cannot produce a cache key, be fetched or staged. They can only be tracked.

RESOLVED = 1¶

Resolved

Resolved sources have a reference and can produce a cache key and be fetched, however they cannot be staged.

CACHED = 2¶

Cached

Cached sources have a reference which is present in the local source cache. Only cached sources can be staged.

exception SourceError(message, *, detail=None, reason=None, temporary=False)¶

Bases: buildstream._exceptions.BstError

This exception should be raised by Source implementations to report errors to the user.

Parameters:	message (str) – The breif error description to report to the user detail (str) – A possibly multiline, more detailed error message reason (str) – An optional machine readable reason string, used for test cases temporary (bool) – An indicator to whether the error may occur if the operation was run again. (Since: 1.2)

class SourceFetcher¶

Bases: object

This interface exists so that a source that downloads from multiple places (e.g. a git source with submodules) has a consistent interface for fetching and substituting aliases.

Since: 1.2

Attention

When implementing a SourceFetcher, remember to call Source.mark_download_url() for every URL found in the configuration data at Plugin.configure() time.

fetch(alias_override=None, **kwargs)¶

Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.

Parameters:	alias_override (str) – The alias to use instead of the default one defined by the aliases field in the project’s config.
Raises:	`SourceError`

Implementors should raise SourceError if the there is some network error or if the source reference could not be matched.

mark_download_url(url)¶

Identifies the URL that this SourceFetcher uses to download

This must be called during the fetcher’s initialization

Parameters:	url (str) – The url used to download.

class Source¶

Bases: buildstream.plugin.Plugin

Base Source class.

All Sources derive from this class, this interface defines how the core will be interacting with Sources.

BST_REQUIRES_PREVIOUS_SOURCES_TRACK = False¶

Whether access to previous sources is required during track

When set to True:

all sources listed before this source in the given element will be fetched before this source is tracked
Source.track() will be called with an additional keyword argument previous_sources_dir where previous sources will be staged
this source can not be the first source for an element

Since: 1.4

BST_REQUIRES_PREVIOUS_SOURCES_FETCH = False¶

Whether access to previous sources is required during fetch

When set to True:

all sources listed before this source in the given element will be fetched before this source is fetched
Source.fetch() will be called with an additional keyword argument previous_sources_dir where previous sources will be staged
this source can not be the first source for an element

Since: 1.4

COMMON_CONFIG_KEYS = ['kind', 'directory']¶

Common source config keys

Source config keys that must not be accessed in configure(), and should be checked for using node_validate().

get_consistency()¶

Report whether the source has a resolved reference

Returns:	The source consistency
Return type:	(`Consistency`)

load_ref(node)¶

Loads the ref for this Source from the specified node.

Parameters:	node (dict) – The YAML node to load the ref from

Note

The ref for the Source is expected to be read at Plugin.configure() time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.

Since: 1.2

get_ref()¶

Fetch the internal ref, however it is represented

Returns:	The internal source reference, or `None`
Return type:	(simple object)

Note

The reference is the user provided (or track resolved) value the plugin uses to represent a specific input, like a commit in a VCS or a tarball’s checksum. Usually the reference is a string, but the plugin may choose to represent it with a tuple or such.

Implementations must return a None value in the case that the ref was not loaded. E.g. a (None, None) tuple is not acceptable.

set_ref(ref, node)¶

Applies the internal ref, however it is represented

Parameters:	ref (simple object) – The internal source reference to set, or `None` node (dict) – The same dictionary which was previously passed to `Plugin.configure()`

See Source.get_ref() for a discussion on the ref parameter.

Note

Implementors must support the special None value here to allow clearing any existing ref.

track(**kwargs)¶

Resolve a new ref from the plugin’s track option

Parameters:	previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` is set to True.
Returns:	A new internal source reference, or None
Return type:	(simple object)

If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst track commands.

This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.

See Source.get_ref() for a discussion on the ref parameter.

fetch(**kwargs)¶

Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.

Parameters:	previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when `BST_REQUIRES_PREVIOUS_SOURCES_FETCH` is set to True.
Raises:	`SourceError`

Implementors should raise SourceError if the there is some network error or if the source reference could not be matched.

stage(directory)¶

Stage the sources to a directory

Parameters:	directory (str) – Path to stage the source
Raises:	`SourceError`

Implementors should assume that directory already exists and stage already cached sources to the passed directory.

Implementors should raise SourceError when encountering some system error.

init_workspace(directory)¶

Initialises a new workspace

Parameters:	directory (str) – Path of the workspace to init
Raises:	`SourceError`

Default implementation is to call Source.stage().

Implementors overriding this method should assume that directory already exists.

Implementors should raise SourceError when encountering some system error.

get_source_fetchers()¶

Get the objects that are used for fetching

If this source doesn’t download from multiple URLs, returning None and falling back on the default behaviour is recommended.

Returns:	The Source’s SourceFetchers, if any.
Return type:	iterable

Note

Implementors can implement this as a generator.

The SourceFetcher.fetch() method will be called on the returned fetchers one by one, before consuming the next fetcher in the list.

Since: 1.2

validate_cache()¶

Implement any validations once we know the sources are cached

This is guaranteed to be called only once for a given session once the sources are known to be Consistency.CACHED, if source tracking is enabled in the session for this source, then this will only be called if the sources become cached after tracking completes.

Since: 1.4

get_mirror_directory()¶

Fetches the directory where this source should store things

Returns:	The directory belonging to this source
Return type:	(str)

translate_url(url, *, alias_override=None, primary=True)¶

Translates the given url which may be specified with an alias into a fully qualified url.

Parameters:	url (str) – A URL, which may be using an alias alias_override (str) – Optionally, an URI to override the alias with. (Since: 1.2) primary (bool) – Whether this is the primary URL for the source. (Since: 1.2)
Returns:	The fully qualified URL, with aliases resolved
Return type:	str

Note

This must be called for every URL in the configuration during Plugin.configure() if Source.mark_download_url() is not called.

mark_download_url(url, *, primary=True)¶

Identifies the URL that this Source uses to download

Parameters:	url (str) – The URL used to download primary (bool) – Whether this is the primary URL for the source

Note

This must be called for every URL in the configuration during Plugin.configure() if Source.translate_url() is not called.

Since: 1.2

get_project_directory()¶

Fetch the project base directory

This is useful for sources which need to load resources stored somewhere inside the project.

Returns:	The project base directory
Return type:	str

tempdir()¶

Context manager for working in a temporary directory

Yields:	(str) – A path to a temporary directory

This should be used by source plugins directly instead of the tempfile module. This one will automatically cleanup in case of termination by catching the signal before os._exit(). It will also use the ‘mirror directory’ as expected for a source.