Source - Base source class
Abstract Methods
For loading and configuration purposes, Sources must implement the Plugin base class abstract methods.
Attention
In order to ensure that all configuration data is processed at
load time, it is important that all URLs have been processed during
Plugin.configure()
.
Source implementations must either call
Source.translate_url()
or
Source.mark_download_url()
for every URL that has been specified in the configuration during
Plugin.configure()
Sources expose the following abstract methods. Unless explicitly mentioned, these methods are mandatory to implement.
-
Report the sources consistency state.
-
Load the ref from a specific YAML node
-
Fetch the source ref
-
Set a new ref explicitly
-
Automatically derive a new ref from a symbolic tracking branch
-
Fetch the actual payload for the currently set ref
-
Stage the sources for a given ref at a specified location
-
Stage sources in a local directory for use as a workspace.
Optional: If left unimplemented, this will default to calling
Source.stage()
-
Get the objects that are used for fetching.
Optional: This only needs to be implemented for sources that need to download from multiple URLs while fetching (e.g. a git repo and its submodules). For details on how to define a SourceFetcher, see SourceFetcher.
-
Perform any validations which require the sources to be cached.
Optional: This is completely optional and will do nothing if left unimplemented.
Accessing previous sources
Since: 1.4
In the general case, all sources are fetched and tracked independently of one another. In situations where a source needs to access previous source(s) in order to perform its own track and/or fetch, following attributes can be set to request access to previous sources:
BST_REQUIRES_PREVIOUS_SOURCES_TRACK
Indicate that access to previous sources is required during track
BST_REQUIRES_PREVIOUS_SOURCES_FETCH
Indicate that access to previous sources is required during fetch
The intended use of such plugins is to fetch external dependencies of other sources, typically using some kind of package manager, such that all the dependencies of the original source(s) are available at build time.
When implementing such a plugin, implementors should adhere to the following guidelines:
Implementations must be able to store the obtained artifacts in a subdirectory.
Implementations must be able to deterministically generate a unique ref, such that two refs are different if and only if they produce different outputs.
Implementations must not introduce host contamination.
SourceFetcher - Object for fetching individual URLs
Abstract Methods
SourceFetchers expose the following abstract methods. Unless explicitly mentioned, these methods are mandatory to implement.
-
Fetches the URL associated with this SourceFetcher, optionally taking an alias override.
Class Reference
- class Consistency
Bases:
object
- INCONSISTENT = 0
Inconsistent
Inconsistent sources have no explicit reference set. They cannot produce a cache key, be fetched or staged. They can only be tracked.
- RESOLVED = 1
Resolved
Resolved sources have a reference and can produce a cache key and be fetched, however they cannot be staged.
- CACHED = 2
Cached
Cached sources have a reference which is present in the local source cache. Only cached sources can be staged.
- exception SourceError(message, *, detail=None, reason=None, temporary=False)
Bases:
BstError
This exception should be raised by
Source
implementations to report errors to the user.- Parameters:
message (str) – The breif error description to report to the user
detail (str) – A possibly multiline, more detailed error message
reason (str) – An optional machine readable reason string, used for test cases
temporary (bool) – An indicator to whether the error may occur if the operation was run again. (Since: 1.2)
- class SourceFetcher
Bases:
object
This interface exists so that a source that downloads from multiple places (e.g. a git source with submodules) has a consistent interface for fetching and substituting aliases.
Since: 1.2
Attention
When implementing a SourceFetcher, remember to call
Source.mark_download_url()
for every URL found in the configuration data atPlugin.configure()
time.- fetch(alias_override=None, **kwargs)
Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.
- Parameters:
alias_override (str) – The alias to use instead of the default one defined by the aliases field in the project’s config.
- Raises:
.SourceError –
Implementors should raise
SourceError
if the there is some network error or if the source reference could not be matched.
- mark_download_url(url)
Identifies the URL that this SourceFetcher uses to download
This must be called during the fetcher’s initialization
- Parameters:
url (str) – The url used to download.
- class Source
Bases:
Plugin
Base Source class.
All Sources derive from this class, this interface defines how the core will be interacting with Sources.
- BST_REQUIRES_PREVIOUS_SOURCES_TRACK = False
Whether access to previous sources is required during track
- When set to True:
all sources listed before this source in the given element will be fetched before this source is tracked
Source.track() will be called with an additional keyword argument previous_sources_dir where previous sources will be staged
this source can not be the first source for an element
Since: 1.4
- BST_REQUIRES_PREVIOUS_SOURCES_FETCH = False
Whether access to previous sources is required during fetch
- When set to True:
all sources listed before this source in the given element will be fetched before this source is fetched
Source.fetch() will be called with an additional keyword argument previous_sources_dir where previous sources will be staged
this source can not be the first source for an element
Since: 1.4
- COMMON_CONFIG_KEYS = ['kind', 'directory']
Common source config keys
Source config keys that must not be accessed in configure(), and should be checked for using node_validate().
- get_consistency()
Report whether the source has a resolved reference
- Returns:
The source consistency
- Return type:
- load_ref(node)
Loads the ref for this Source from the specified node.
- Parameters:
node (dict) – The YAML node to load the ref from
Note
The ref for the Source is expected to be read at
Plugin.configure()
time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.Since: 1.2
- get_ref()
Fetch the internal ref, however it is represented
- Returns:
The internal source reference, or
None
- Return type:
(simple object)
Note
The reference is the user provided (or track resolved) value the plugin uses to represent a specific input, like a commit in a VCS or a tarball’s checksum. Usually the reference is a string, but the plugin may choose to represent it with a tuple or such.
Implementations must return a
None
value in the case that the ref was not loaded. E.g. a(None, None)
tuple is not acceptable.
- set_ref(ref, node)
Applies the internal ref, however it is represented
- Parameters:
ref (simple object) – The internal source reference to set, or
None
node (dict) – The same dictionary which was previously passed to
Plugin.configure()
See
Source.get_ref()
for a discussion on the ref parameter.Note
Implementors must support the special
None
value here to allow clearing any existing ref.
- track(**kwargs)
Resolve a new ref from the plugin’s track option
- Parameters:
previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when
BST_REQUIRES_PREVIOUS_SOURCES_TRACK
is set to True.- Returns:
A new internal source reference, or None
- Return type:
(simple object)
If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst track commands.
This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.
See
Source.get_ref()
for a discussion on the ref parameter.
- fetch(**kwargs)
Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.
- Parameters:
previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when
BST_REQUIRES_PREVIOUS_SOURCES_FETCH
is set to True.- Raises:
.SourceError –
Implementors should raise
SourceError
if the there is some network error or if the source reference could not be matched.
- stage(directory)
Stage the sources to a directory
- Parameters:
directory (str) – Path to stage the source
- Raises:
.SourceError –
Implementors should assume that directory already exists and stage already cached sources to the passed directory.
Implementors should raise
SourceError
when encountering some system error.
- init_workspace(directory)
Initialises a new workspace
- Parameters:
directory (str) – Path of the workspace to init
- Raises:
.SourceError –
Default implementation is to call
Source.stage()
.Implementors overriding this method should assume that directory already exists.
Implementors should raise
SourceError
when encountering some system error.
- get_source_fetchers()
Get the objects that are used for fetching
If this source doesn’t download from multiple URLs, returning None and falling back on the default behaviour is recommended.
- Returns:
The Source’s SourceFetchers, if any.
- Return type:
iterable
Note
Implementors can implement this as a generator.
The
SourceFetcher.fetch()
method will be called on the returned fetchers one by one, before consuming the next fetcher in the list.Since: 1.2
- validate_cache()
Implement any validations once we know the sources are cached
This is guaranteed to be called only once for a given session once the sources are known to be
Consistency.CACHED
, if source tracking is enabled in the session for this source, then this will only be called if the sources become cached after tracking completes.Since: 1.4
- get_mirror_directory()
Fetches the directory where this source should store things
- Returns:
The directory belonging to this source
- Return type:
(str)
- translate_url(url, *, alias_override=None, primary=True)
Translates the given url which may be specified with an alias into a fully qualified url.
- Parameters:
url (str) – A URL, which may be using an alias
alias_override (str) – Optionally, an URI to override the alias with. (Since: 1.2)
primary (bool) – Whether this is the primary URL for the source. (Since: 1.2)
- Returns:
The fully qualified URL, with aliases resolved
- Return type:
str
Note
This must be called for every URL in the configuration during
Plugin.configure()
ifSource.mark_download_url()
is not called.
- mark_download_url(url, *, primary=True)
Identifies the URL that this Source uses to download
- Parameters:
url (str) – The URL used to download
primary (bool) – Whether this is the primary URL for the source
Note
This must be called for every URL in the configuration during
Plugin.configure()
ifSource.translate_url()
is not called.Since: 1.2
- get_project_directory()
Fetch the project base directory
This is useful for sources which need to load resources stored somewhere inside the project.
- Returns:
The project base directory
- Return type:
str
- tempdir()
Context manager for working in a temporary directory
- Yields:
(str) – A path to a temporary directory
This should be used by source plugins directly instead of the tempfile module. This one will automatically cleanup in case of termination by catching the signal before os._exit(). It will also use the ‘mirror directory’ as expected for a source.