DownloadableFileSource - Abstract class for sources downloaded from a URI
This DownloadableFileSource class is a convenience class on can derive for implementing sources that get downloaded from a URI.
It provides utilities around handling mirrors, tracking and fetching the source.
Any derived classes must write their own stage() and get_unique_key() implementation.
Built-in functionality
The DownloadableFileSource class provides built in keys which can be set when intantiating any Source which derives from DownloadableFileSource
Guess pattern
The
version-guess-pattern
sets the regular expression which will be used to attempt to guess the version of a source when parsing the source’s URI.The DownloadableFileSource provides a default implementation of
Source.collect_source_info()
, which will use theversion-guess-pattern
to attempt to extract a human readable version string from the specified URI, in order to fill out the reportedversion_guess
.The URI will be searched using this regular expression, and is allowed to yield a number of groups. For example the value
(\d+)_(\d+)_(\d+)
would report 3 groups if 3 numerical values separated by underscores were found in the URI.The default value for
version-guess-pattern
is\d+\.\d+(?:\.\d+)?
.Since: 2.5.
Version
The
version
explicitly sets theversion_guess
attribute of theSourceInfo
reported for this source, overriding any guessing.This is useful for remote files which do not express their version in their filenames.
Since: 2.5.
SourceMirror extra data “http-auth”
The DownloadableFileSource, and consequently any Source
implementations which derive from DownloadableFileSource, support the “http-auth”
extra data returned by SourceMirror
plugins
through Source.translate_url()
.
This functionality is available Since: 2.2.
This allows one to use SourceMirror
plugins
to add an authorization header to the GET
requests.
Example:
class MySourceMirror(SourceMirror):
def translate_url(
self,
*,
alias: str,
alias_url: str,
source_url: str,
extra_data: Optional[Dict[str, Any]],
) -> str:
#
# Set the "http-auth" extra data
#
if extra_data is not None:
extra_data["http-auth"] = "bearer"
# ...
Only the “http-auth” value bearer
is supported.
Example:
If the URL reported by SourceMirror.translate_url()
is http://flying-ponies.com/downloads/pony.tgz
, then a corresponding entry will be expected in the
user’s ~/.netrc
:
flying-ponies.com
password 1234
DownloadableFileSource will add the following header to the GET
request to download the file:
Authorization: Bearer 1234
Default reporting of SourceInfo
Source plugins which derive from the DownloadableFileSource, unless overridden and specified in the documentation of the specific source plugin, will behave as described here.
The DownloadableFileSource reports the URL of the remote file as the url.
Further, the DownloadableFileSourcebzr source reports the
SourceInfoMedium.REMOTE_FILE
medium and the
SourceVersionType.SHA256
version_type,
for which it reports the sha256 checksum of the remote file content as the version.
An attempt to guess the version based on the remote filename will be made for the reporting of the guess_version. Control over how the guess is made or overridden is explained above in the built-in functionality documentation.
- class DownloadableFileSource(context: Context, project: Project, meta: MetaSource, variables: Variables, *, alias_override: Tuple[str, AliasSubstitution] | None = None, unique_id: int | None = None)
Bases:
Source
- COMMON_CONFIG_KEYS = ['kind', 'directory', 'url', 'ref', 'version-guess-pattern', 'version']
Common source config keys
Source config keys that must not be accessed in configure(), and should be checked for using node.validate_keys().
- configure(node)
Configure the Plugin from loaded configuration data
- Parameters:
node – The loaded configuration dictionary
- Raises:
Plugin implementors should implement this method to read configuration data and store it.
The
MappingNode.validate_keys()
method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.
- preflight()
Preflight Check
- Raises:
This method is run after
Plugin.configure()
and after the pipeline is fully constructed.Implementors should simply raise
SourceError
orElementError
with an informative message in the case that the host environment is unsuitable for operation.Plugins which require host tools (only sources usually) should obtain them with
utils.get_host_tool()
which will raise an error automatically informing the user that a host tool is needed.
- get_unique_key()
Return something which uniquely identifies the plugin input
- Returns:
A string, list or dictionary which uniquely identifies the input
This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.
Elements and Sources should implement this by collecting any configurations which could possibly affect the output and return a dictionary of these settings.
For Sources, this is guaranteed to only be called if
Source.is_resolved()
has returnedTrue
which is to say that theSource
is expected to have an exact source ref indicating exactly what source is going to be staged.Note
If your plugin is concerned with API stability, then future extensions of your plugin YAML configuration which affect the unique key returned here should be added to this key with care.
A good rule of thumb is to only compute the new value in the returned key if the value of the newly added YAML key is not equal to it’s default value.
- is_cached() bool
Get whether the source has a local copy of its data.
This method is guaranteed to only be called whenever
Source.is_resolved()
returns True.Returns: whether the source is cached locally or not.
- load_ref(node)
Loads the
SourceRef
for this Source from the specified node.- Parameters:
node – The YAML node to load the ref from
Working with the source ref is discussed here.
Note
The
SourceRef
for the Source is expected to be read atPlugin.configure()
time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.
- get_ref()
Fetch the
SourceRef
- Returns:
The internal
SourceRef
, orNone
Working with the source ref is discussed here.
- set_ref(ref, node)
Applies the internal ref, however it is represented
- Parameters:
ref – The internal
SourceRef
to set, orNone
node – The same node which was previously passed to
Plugin.configure()
andSource.load_ref()
The implementor must update the node parameter to reflect the new ref, and it should store the passed ref so that it will be returned in any later calls to
Source.get_ref()
.The passed ref parameter is guaranteed to either be a value which has been previously retrieved by the
Source.get_ref()
method on the same plugin, orNone
.Example:
# Implementation of Source.set_ref() # def set_ref(self, ref, node): # Update internal state of the ref self.ref = ref # Update the passed node so that we will read the new ref # next time this source plugin is configured with this node. # node["ref"] = self.ref
Working with the source ref is discussed here.
- track()
Resolve a new ref from the plugin’s track option
- Parameters:
previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when
BST_REQUIRES_PREVIOUS_SOURCES_TRACK
is set to True.- Returns:
A new
SourceRef
, or None
If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst source track commands.
This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.
Working with the source ref is discussed here.
- fetch()
Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.
- Parameters:
previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when
BST_REQUIRES_PREVIOUS_SOURCES_FETCH
is set to True.- Raises:
.SourceError –
Implementors should raise
SourceError
if the there is some network error or if the source reference could not be matched.
- collect_source_info()
Get the
SourceInfo
objects describing this sourceThis method should only be called whenever
Source.is_resolved()
returnsTrue
.SourceInfo objects created by implementors should be created with
Source.create_source_info()
.Returns: the
SourceInfo
objects describing this source- Raises:
.SourceImplError – if the source class does not implement this method and does not implement
SourceFether.get_source_info()
Note
If your plugin uses
SourceFetcher
objects, you can implementSource.collect_source_info()
instead.Since: 2.5