DownloadableFileSource - Abstract class for sources downloaded from a URI

This DownloadableFileSource class is a convenience class on can derive for implementing sources that get downloaded from a URI.

It provides utilities around handling mirrors, tracking and fetching the source.

Any derived classes must write their own stage() and get_unique_key() implementation.

SourceMirror extra data “http-auth”

The DownloadableFileSource, and consequently any Source implementations which derive from DownloadableFileSource, support the “http-auth” extra data returned by SourceMirror plugins through Source.translate_url().

This functionality is available Since: 2.2.

This allows one to use SourceMirror plugins to add an authorization header to the GET requests.

Example:

class MySourceMirror(SourceMirror):

     def translate_url(
         self,
         *,
         alias: str,
         alias_url: str,
         source_url: str,
         extra_data: Optional[Dict[str, Any]],
     ) -> str:

         #
         # Set the "http-auth" extra data
         #
         if extra_data is not None:
             extra_data["http-auth"] = "bearer"

         # ...

Only the “http-auth” value bearer is supported.

Example:

If the URL reported by SourceMirror.translate_url() is http://flying-ponies.com/downloads/pony.tgz, then a corresponding entry will be expected in the user’s ~/.netrc:

flying-ponies.com
    password 1234

DownloadableFileSource will add the following header to the GET request to download the file:

Authorization: Bearer 1234

class DownloadableFileSource(context: Context, project: Project, meta: MetaSource, variables: Variables, *, alias_override: Tuple[str, AliasSubstitution] | None = None, unique_id: int | None = None)

Bases: Source

COMMON_CONFIG_KEYS = ['kind', 'directory', 'url', 'ref']

Common source config keys

Source config keys that must not be accessed in configure(), and should be checked for using node.validate_keys().

configure(node)

Configure the Plugin from loaded configuration data

Parameters:

node – The loaded configuration dictionary

Raises:

.SourceError – If it’s a Source implementation
.ElementError – If it’s an Element implementation

Plugin implementors should implement this method to read configuration data and store it.

The MappingNode.validate_keys() method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.

preflight()

Preflight Check

Raises:

.SourceError – If it’s a Source implementation
.ElementError – If it’s an Element implementation

This method is run after Plugin.configure() and after the pipeline is fully constructed.

Implementors should simply raise SourceError or ElementError with an informative message in the case that the host environment is unsuitable for operation.

Plugins which require host tools (only sources usually) should obtain them with utils.get_host_tool() which will raise an error automatically informing the user that a host tool is needed.

get_unique_key()

Return something which uniquely identifies the plugin input

Returns:: A string, list or dictionary which uniquely identifies the input

This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.

Elements and Sources should implement this by collecting any configurations which could possibly affect the output and return a dictionary of these settings.

For Sources, this is guaranteed to only be called if Source.is_resolved() has returned True which is to say that the Source is expected to have an exact source ref indicating exactly what source is going to be staged.

Note

If your plugin is concerned with API stability, then future extensions of your plugin YAML configuration which affect the unique key returned here should be added to this key with care.

A good rule of thumb is to only compute the new value in the returned key if the value of the newly added YAML key is not equal to it’s default value.

is_cached() → bool

Get whether the source has a local copy of its data.

This method is guaranteed to only be called whenever Source.is_resolved() returns True.

Returns: whether the source is cached locally or not.

load_ref(node)

Loads the SourceRef for this Source from the specified node.

Parameters:: node – The YAML node to load the ref from

Working with the source ref is discussed here.

Note

The SourceRef for the Source is expected to be read at Plugin.configure() time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.

get_ref()

Fetch the SourceRef

Returns:: The internal SourceRef, or None

Working with the source ref is discussed here.

set_ref(ref, node)

Applies the internal ref, however it is represented

Parameters:

ref – The internal SourceRef to set, or None
node – The same node which was previously passed to Plugin.configure() and Source.load_ref()

The implementor must update the node parameter to reflect the new ref, and it should store the passed ref so that it will be returned in any later calls to Source.get_ref().

The passed ref parameter is guaranteed to either be a value which has been previously retrieved by the Source.get_ref() method on the same plugin, or None.

Example:

# Implementation of Source.set_ref()
#
def set_ref(self, ref, node):

    # Update internal state of the ref
    self.ref = ref

    # Update the passed node so that we will read the new ref
    # next time this source plugin is configured with this node.
    #
    node["ref"] = self.ref

Working with the source ref is discussed here.

track()

Resolve a new ref from the plugin’s track option

Parameters:: previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_TRACK is set to True.
Returns:: A new SourceRef, or None

If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst source track commands.

This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.

Working with the source ref is discussed here.

fetch()

Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.

Parameters:: previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_FETCH is set to True.
Raises:: .SourceError –

Implementors should raise SourceError if the there is some network error or if the source reference could not be matched.