3. Overlapping files
In this chapter, we will discuss what happens when files from multiple element artifacts conflict with eachother, and what strategies we can use to resolve these situations.
Note
This example is distributed with BuildStream in the doc/examples/overlaps subdirectory.
3.1. Overview
This project builds on the previous chapter on composition, and as such we’ll only go over what has changed from there, which is not much.
3.2. Project structure
In this example we’ve just extended the libhello.bst and the hello.bst
elements such that they both install an additional file: %{docdir}/hello.txt.
We’ve updated the following Makefiles:
3.2.1. files/libhello/Makefile
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so
install:
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.so: libhello.o
$(CC) -shared -o $@ $<
3.2.2. files/hello/Makefile
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 755 hello ${DESTDIR}${PREFIX}/bin
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
hello: hello.c
$(CC) $< -o $@ -Wall -lhello
As you can see, this now presents a conflict of sorts, where multiple elements in the pipeline want to install the same file.
3.3. Using the project
In this chapter, we’re only going to present the warning and then discuss how to mitigate this situation.
See what happens when we try to build the runtime-only.bst
compose element:
user@host:~/overlaps$ bst build runtime-only.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732145.247261 6612 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
BuildStream Version 2.3.0
Session Start: Thursday, 15-08-2024 at 14:29:05
Project: overlaps (/home/user/buildstream/doc/examples/overlaps)
Targets: runtime-only.bst
User Configuration
Configuration File: /home/user/buildstream/doc/run-bst-iaanumqp/buildstream.conf
Cache Directory: /home/user/buildstream/doc/run-bst-iaanumqp
Log Files: /home/user/buildstream/doc/run-bst-iaanumqp/logs
Source Mirrors: /home/user/buildstream/doc/run-bst-iaanumqp/sources
Build Area: /home/user/buildstream/doc/run-bst-iaanumqp/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: overlaps
Element Plugins
compose: core plugin
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 617c4e863e3404763b4b4860156e4eb510ea5de1f99384d71465f79ee9495504 base/alpine.bst
cached 1f56d7f344afc5f3f3738600e43bd523ed68d3208419e3d49d1325ff0b83a843 base.bst
cached eb9542a1a6a86e71e839554cf67aca2e2fa52c29eefbb8707889bae8ebd9d429 libhello.bst
cached 5ba078b2324d29d7a455f3a1e754443279bd62fc685c0849bb86ef214cbc51aa hello.bst
buildable 06c0be7b5c5692db9a37354e4e1210104d4ebd717f35d49e03ff7508ba3e1bb2 runtime-only.bst
===============================================================================
[--:--:--][06c0be7b][ build:runtime-only.bst ] START overlaps/runtime-only/06c0be7b-build.20240815-142905.log
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Staging dependencies
[--:--:--][06c0be7b][ build:runtime-only.bst ] WARNING [overlaps]: Non-whitelisted overlaps detected
Staged files overwrite existing files in staging area: /
/usr/share/doc/hello.txt: hello.bst is not permitted to overlap other elements, order hello.bst above libhello.bst
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Computing split
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Computing split
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Integrating sandbox
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Running commands
[--:--:--][06c0be7b][ build:runtime-only.bst ] INFO Integration added 0 and removed 0 files
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Creating composition
Including files from domains: runtime
Excluding orphaned files
[--:--:--][06c0be7b][ build:runtime-only.bst ] INFO Composing 287 files
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Creating composition
[--:--:--][06c0be7b][ build:runtime-only.bst ] START Caching artifact
[00:00:00][06c0be7b][ build:runtime-only.bst ] SUCCESS Caching artifact
[00:00:01][06c0be7b][ build:runtime-only.bst ] SUCCESS overlaps/runtime-only/06c0be7b-build.20240815-142905.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 5
Session: 5
Fetch Queue: processed 0, skipped 5, failed 0
Build Queue: processed 1, skipped 4, failed 0
Notice the warning message about the conflicting file, it is there to inform the user about which files are overlapping, and also which elements are being staged in which order.
Note also that BuildStream does not discover the overlap until the moment that you build a reverse dependency which will require staging of both artifacts.
Tip
The overlaps warning discussed here can be configured to be
a fatal warning. This is useful
in the case that you want to be strict about avoiding overlapping
files in your project.
3.4. Mitigating overlapping files
Since we recently discussed filtering of artifacts,
we should note that it is of course possible to handle this case by
having hello.bst depend on a filtered version of libhello.bst with the
offending file excluded.
However, working with filter elements just for the
sake of handling a conflicting artifact would be quite inconvenient, so we
have other means.
3.4.1. Whitelisting overlapping files
BuildStream allows explicitly ignoring such errors by adding the files
to an overlap whitelist, you could achieve
this in the given example by adding the following to the hello.bst element:
public:
bst:
overlap-whitelist:
- |
%{docdir}/hello.txt
Note
Note that glob patterns are also
supported in the whitelist.
3.4.2. Artifact munging
Another way around this situation is the munge the artifacts at install time such that there is no conflict.
This is the easiest approach in the case that you might want to keep the
underlying %{docdir}/hello.txt from libhello.bst and discard the
same file from hello.bst.
In this case, we might modify the hello.bst file so that it’s install
command contain an rm statement, as such:
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
- |
# Rid ourselves of the unwanted file at install time
rm -f %{install-root}%{docdir}/hello.txt
This would cause later builds of runtime-only.bst to no longer
conflict on the given file.
3.5. Summary
In this chapter we’ve presented a situation where an artifact can conflict with another artifact by way of providing the same files.
We’ve presented the overlap whitelist public data which is the typical solution for silencing the error when the outcome is desired, and also presented a strategy to deal with cases where you want to keep files from the overlapped artifact instead.