DownloadableFileSource - Abstract class for sources downloaded from a URI

This DownloadableFileSource class is a convenience class on can derive for implementing sources that get downloaded from a URI.

It provides utilities around handling mirrors, tracking and fetching the source.

Any derived classes must write their own stage() and get_unique_key() implementation.

Built-in functionality

The DownloadableFileSource class provides built in keys which can be set when intantiating any Source which derives from DownloadableFileSource

  • Guess pattern

    The version-guess-pattern sets the regular expression which will be used to attempt to guess the version of a source when parsing the source’s URI.

    The DownloadableFileSource provides a default implementation of Source.collect_source_info(), which will use the version-guess-pattern to attempt to extract a human readable version string from the specified URI, in order to fill out the reported version_guess.

    This is done using the utils.guess_version() utility function, please refer to that function documentation to understand how the guessing mechanics works, and what kind of string you should provide here.

    Since: 2.5.

  • Version

    The version explicitly sets the version_guess attribute of the SourceInfo reported for this source, overriding any guessing.

    This is useful for remote files which do not express their version in their filenames.

    Since: 2.5.

SourceMirror extra data “http-auth”

The DownloadableFileSource, and consequently any Source implementations which derive from DownloadableFileSource, support the “http-auth” extra data returned by SourceMirror plugins through Source.translate_url().

This functionality is available Since: 2.2.

This allows one to use SourceMirror plugins to add an authorization header to the GET requests.

Example:

class MySourceMirror(SourceMirror):

     def translate_url(
         self,
         *,
         alias: str,
         alias_url: str,
         source_url: str,
         extra_data: Optional[Dict[str, Any]],
     ) -> str:

         #
         # Set the "http-auth" extra data
         #
         if extra_data is not None:
             extra_data["http-auth"] = "bearer"

         # ...

Only the “http-auth” value bearer is supported.

Example:

If the URL reported by SourceMirror.translate_url() is http://flying-ponies.com/downloads/pony.tgz, then a corresponding entry will be expected in the user’s ~/.netrc:

flying-ponies.com
    password 1234

DownloadableFileSource will add the following header to the GET request to download the file:

Authorization: Bearer 1234

Default reporting of SourceInfo

Source plugins which derive from the DownloadableFileSource, unless overridden and specified in the documentation of the specific source plugin, will behave as described here.

The DownloadableFileSource reports the URL of the remote file as the url.

Further, the DownloadableFileSourcebzr source reports the SourceInfoMedium.REMOTE_FILE medium and the SourceVersionType.SHA256 version_type, for which it reports the sha256 checksum of the remote file content as the version.

An attempt to guess the version based on the remote filename will be made for the reporting of the version_guess. Control over how the guess is made or overridden is explained above in the built-in functionality documentation.

class DownloadableFileSource(context: Context, project: Project, meta: MetaSource, variables: Variables, *, alias_override: Tuple[str, AliasSubstitution] | None = None, unique_id: int | None = None)

Bases: Source

COMMON_CONFIG_KEYS = ['kind', 'directory', 'url', 'ref', 'version-guess-pattern', 'version']

Common source config keys

Source config keys that must not be accessed in configure(), and should be checked for using node.validate_keys().

configure(node)

Configure the Plugin from loaded configuration data

Parameters:

node – The loaded configuration dictionary

Raises:

  • .SourceError – If it’s a Source implementation

  • .ElementError – If it’s an Element implementation

Plugin implementors should implement this method to read configuration data and store it.

The MappingNode.validate_keys() method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.

preflight()

Preflight Check

Raises:

  • .SourceError – If it’s a Source implementation

  • .ElementError – If it’s an Element implementation

This method is run after Plugin.configure() and after the pipeline is fully constructed.

Implementors should simply raise SourceError or ElementError with an informative message in the case that the host environment is unsuitable for operation.

Plugins which require host tools (only sources usually) should obtain them with utils.get_host_tool() which will raise an error automatically informing the user that a host tool is needed.

get_unique_key()

Return something which uniquely identifies the plugin input

Returns:

A string, list or dictionary which uniquely identifies the input

This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.

Elements and Sources should implement this by collecting any configurations which could possibly affect the output and return a dictionary of these settings.

For Sources, this is guaranteed to only be called if Source.is_resolved() has returned True which is to say that the Source is expected to have an exact source ref indicating exactly what source is going to be staged.

Note

If your plugin is concerned with API stability, then future extensions of your plugin YAML configuration which affect the unique key returned here should be added to this key with care.

A good rule of thumb is to only compute the new value in the returned key if the value of the newly added YAML key is not equal to it’s default value.

is_cached() → bool

Get whether the source has a local copy of its data.

This method is guaranteed to only be called whenever Source.is_resolved() returns True.

Returns: whether the source is cached locally or not.

load_ref(node)

Loads the SourceRef for this Source from the specified node.

Parameters:

node – The YAML node to load the ref from

Working with the source ref is discussed here.

Note

The SourceRef for the Source is expected to be read at Plugin.configure() time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.

get_ref()

Fetch the SourceRef

Returns:

The internal SourceRef, or None

Working with the source ref is discussed here.

set_ref(ref, node)

Applies the internal ref, however it is represented

Parameters:

The implementor must update the node parameter to reflect the new ref, and it should store the passed ref so that it will be returned in any later calls to Source.get_ref().

The passed ref parameter is guaranteed to either be a value which has been previously retrieved by the Source.get_ref() method on the same plugin, or None.

Example:

# Implementation of Source.set_ref()
#
def set_ref(self, ref, node):

    # Update internal state of the ref
    self.ref = ref

    # Update the passed node so that we will read the new ref
    # next time this source plugin is configured with this node.
    #
    node["ref"] = self.ref

Working with the source ref is discussed here.

track()

Resolve a new ref from the plugin’s track option

Parameters:

previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_TRACK is set to True.

Returns:

A new SourceRef, or None

If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst source track commands.

This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.

Working with the source ref is discussed here.

fetch()

Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.

Parameters:

previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_FETCH is set to True.

Raises:

.SourceError –

Implementors should raise SourceError if the there is some network error or if the source reference could not be matched.

collect_source_info()

Get the SourceInfo objects describing this source

This method should only be called whenever Source.is_resolved() returns True.

SourceInfo objects created by implementors should be created with Source.create_source_info().

Returns: the SourceInfo objects describing this source

Raises:

.SourceImplError – if the source class does not implement this method and does not implement SourceFether.get_source_info()

Note

If your plugin uses SourceFetcher objects, you can implement Source.get_source_info() instead.

Since: 2.5