Utilities

exception UtilError(message, reason=None)

Bases: buildstream._exceptions.BstError

Raised by utility functions when system calls fail.

This will be handled internally by the BuildStream core, if you need to handle this error, then it should be reraised, or either of the ElementError or SourceError exceptions should be raised from this error.

exception ProgramNotFoundError(message, reason=None)

Bases: buildstream._exceptions.BstError

Raised if a required program is not found.

It is normally unneeded to handle this exception from plugin code.

exception DirectoryExistsError

Bases: OSError

Raised when a os.rename is attempted but the destination is an existing directory.

class FileListResult

Bases: object

An object which stores the result of one of the operations which run on a list of files.

overwritten = None

List of files which were overwritten in the target directory

ignored = None

List of files which were ignored, because they would have replaced a non empty directory

failed_attributes = None

List of files for which attributes could not be copied over

files_written = None

List of files that were written.

combine(other)

Create a new FileListResult that contains the results of both.

list_relative_paths(directory)

A generator for walking directory relative paths

This generator is useful for checking the full manifest of a directory.

Symbolic links will not be followed, but will be included in the manifest.

Parameters:directory (str) – The directory to list files in
Yields:Relative filenames in directory
glob(paths, pattern)

A generator to yield paths which match the glob pattern

Parameters:
  • paths (iterable) – The paths to check
  • pattern (str) – A glob pattern

This generator will iterate over the passed paths and yield only the filenames which matched the provided pattern.

Meta Description
* Zero or more of any character, excepting path separators
** Zero or more of any character, including path separators
? One of any character, except for path separators
[abc] One of any of the specified characters
[a-z] One of the characters in the specified range
[!abc] Any single character, except the specified characters
[!a-z] Any single character, except those in the specified range

Note

Escaping of the metacharacters is not possible

sha256sum(filename)

Calculate the sha256sum of a file

Parameters:filename (str) – A path to a file on disk
Returns:An sha256 checksum string
Return type:(str)
Raises:UtilError – In the case there was an issue opening or reading filename
safe_copy(src, dest, *, result=None)

Copy a file while preserving attributes

Parameters:
  • src (str) – The source filename
  • dest (str) – The destination filename
  • result (FileListResult) – An optional collective result
Raises:

UtilError – In the case of unexpected system call failures

This is almost the same as shutil.copy2(), except that we unlink dest before overwriting it if it exists, just incase dest is a hardlink to a different file.

Try to create a hardlink, but resort to copying in the case of cross device links.

Parameters:
  • src (str) – The source filename
  • dest (str) – The destination filename
  • result (FileListResult) – An optional collective result
Raises:

UtilError – In the case of unexpected system call failures

safe_remove(path)

Removes a file or directory

This will remove a file if it exists, and will remove a directory if the directory is empty.

Parameters:path (str) – The path to remove
Returns:True if path was removed or did not exist, False if path was a non empty directory.
Raises:UtilError – In the case of unexpected system call failures
copy_files(src, dest, *, files=None, ignore_missing=False, report_written=False)

Copy files from source to destination.

Parameters:
  • src (str) – The source file or directory
  • dest (str) – The destination directory
  • files (list) – Optional list of files in src to copy
  • ignore_missing (bool) – Dont raise any error if a source file is missing
  • report_written (bool) – Add to the result object the full list of files written
Returns:

The result describing what happened during this file operation

Return type:

(FileListResult)

Raises:

UtilError – In the case of unexpected system call failures

Note

Directories in dest are replaced with files from src, unless the existing directory in dest is not empty in which case the path will be reported in the return value.

UNIX domain socket files from src are ignored.

Hardlink files from source to destination.

Parameters:
  • src (str) – The source file or directory
  • dest (str) – The destination directory
  • files (list) – Optional list of files in src to link
  • ignore_missing (bool) – Dont raise any error if a source file is missing
  • report_written (bool) – Add to the result object the full list of files written
Returns:

The result describing what happened during this file operation

Return type:

(FileListResult)

Raises:

UtilError – In the case of unexpected system call failures

Note

Directories in dest are replaced with files from src, unless the existing directory in dest is not empty in which case the path will be reported in the return value.

Note

If a hardlink cannot be created due to crossing filesystems, then the file will be copied instead.

UNIX domain socket files from src are ignored.

get_host_tool(name)

Get the full path of a host tool

Parameters:name (str) – The name of the program to search for
Returns:The full path to the program, if found
Raises:ProgramNotFoundError
url_directory_name(url)

Normalizes a url into a directory name

Parameters:url (str) – A url string
Returns:A string which can be used as a directory name
get_bst_version()

Gets the major, minor release portion of the BuildStream version.

Returns:The major version (int): The minor version
Return type:(int)
move_atomic(source, destination, *, ensure_parents=True)

Move the source to the destination using atomic primitives.

This uses os.rename to move a file or directory to a new destination. It wraps some OSError thrown errors to ensure their handling is correct.

The main reason for this to exist is that rename can throw different errors for the same symptom (https://www.unix.com/man-page/POSIX/3posix/rename/) when we are moving a directory.

We are especially interested here in the case when the destination already exists, is a directory and is not empty. In this case, either EEXIST or ENOTEMPTY can be thrown.

In order to ensure consistent handling of these exceptions, this function should be used instead of os.rename

Parameters:
  • source (str or Path) – source to rename
  • destination (str or Path) – destination to which to move the source
  • ensure_parents (bool) – Whether or not to create the parent’s directories of the destination (default: True)
Raises:
  • DirectoryExistsError – if the destination directory already exists and is not empty
  • OSError – if another filesystem level error occured
save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None, tempdir=None)

Save a file with a temporary name and rename it into place when ready.

This is a context manager which is meant for saving data to files. The data is written to a temporary file, which gets renamed to the target name when the context is closed. This avoids readers of the file from getting an incomplete file.

Example:

with save_file_atomic('/path/to/foo', 'w') as f:
    f.write(stuff)

The file will be called something like tmpCAFEBEEF until the context block ends, at which point it gets renamed to foo. The temporary file will be created in the same directory as the output file. The filename parameter must be an absolute path.

If an exception occurs or the process is terminated, the temporary file will be deleted.