3. Overlapping files

In this chapter, we will discuss what happens when files from multiple element artifacts conflict with eachother, and what strategies we can use to resolve these situations.

Note

This example is distributed with BuildStream in the doc/examples/overlaps subdirectory.

3.1. Overview

This project builds on the previous chapter on composition, and as such we’ll only go over what has changed from there, which is not much.

3.2. Project structure

In this example we’ve just extended the libhello.bst and the hello.bst elements such that they both install an additional file: %{docdir}/hello.txt.

We’ve updated the following Makefiles:

3.2.1. files/libhello/Makefile

# Sample makefile for hello library
#
.PHONY: all install

all: libhello.so

install:
	install -d ${DESTDIR}${PREFIX}/lib
	install -d ${DESTDIR}${PREFIX}/include
	install -d ${DESTDIR}${PREFIX}/share/doc
	install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
	install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
	install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc

%.o: %.c %.h
	$(CC) -c $< -o $@ -Wall

libhello.so: libhello.o
	$(CC) -shared -o $@ $<

3.2.2. files/hello/Makefile

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -d ${DESTDIR}${PREFIX}/share/doc
	install -m 755 hello ${DESTDIR}${PREFIX}/bin
	install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc

hello: hello.c
	$(CC) $< -o $@ -Wall -lhello

As you can see, this now presents a conflict of sorts, where multiple elements in the pipeline want to install the same file.

3.3. Using the project

In this chapter, we’re only going to present the warning and then discuss how to mitigate this situation.

See what happens when we try to build the runtime-only.bst compose element:

user@host:~/overlaps$ bst build runtime-only.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723732145.247261    6612 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Initializing remote caches
[00:00:00][        ][    main:core activity                 ] SUCCESS Initializing remote caches
[--:--:--][        ][    main:core activity                 ] START   Query cache
[00:00:00][        ][    main:core activity                 ] SUCCESS Query cache

BuildStream Version 2.3.0
    Session Start: Thursday, 15-08-2024 at 14:29:05
    Project:       overlaps (/home/user/buildstream/doc/examples/overlaps)
    Targets:       runtime-only.bst

User Configuration
    Configuration File:      /home/user/buildstream/doc/run-bst-iaanumqp/buildstream.conf
    Cache Directory:         /home/user/buildstream/doc/run-bst-iaanumqp
    Log Files:               /home/user/buildstream/doc/run-bst-iaanumqp/logs
    Source Mirrors:          /home/user/buildstream/doc/run-bst-iaanumqp/sources
    Build Area:              /home/user/buildstream/doc/run-bst-iaanumqp/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: overlaps

    Element Plugins
        compose: core plugin
        manual:  core plugin
        stack:   core plugin
        import:  core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
      cached 617c4e863e3404763b4b4860156e4eb510ea5de1f99384d71465f79ee9495504 base/alpine.bst 
      cached 1f56d7f344afc5f3f3738600e43bd523ed68d3208419e3d49d1325ff0b83a843 base.bst 
      cached eb9542a1a6a86e71e839554cf67aca2e2fa52c29eefbb8707889bae8ebd9d429 libhello.bst 
      cached 5ba078b2324d29d7a455f3a1e754443279bd62fc685c0849bb86ef214cbc51aa hello.bst 
   buildable 06c0be7b5c5692db9a37354e4e1210104d4ebd717f35d49e03ff7508ba3e1bb2 runtime-only.bst 
===============================================================================
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   overlaps/runtime-only/06c0be7b-build.20240815-142905.log
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Staging dependencies
[--:--:--][06c0be7b][   build:runtime-only.bst              ] WARNING [overlaps]: Non-whitelisted overlaps detected

    Staged files overwrite existing files in staging area: /
    /usr/share/doc/hello.txt: hello.bst is not permitted to overlap other elements, order hello.bst above libhello.bst 

[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Staging dependencies
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Computing split
[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Computing split
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Integrating sandbox
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Running commands

    ldconfig "/usr/lib"

[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Running commands
[--:--:--][06c0be7b][   build:runtime-only.bst              ] INFO    Integration added 0 and removed 0 files
[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Integrating sandbox
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Creating composition

    Including files from domains: runtime
    Excluding orphaned files

[--:--:--][06c0be7b][   build:runtime-only.bst              ] INFO    Composing 287 files
[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Creating composition
[--:--:--][06c0be7b][   build:runtime-only.bst              ] START   Caching artifact
[00:00:00][06c0be7b][   build:runtime-only.bst              ] SUCCESS Caching artifact
[00:00:01][06c0be7b][   build:runtime-only.bst              ] SUCCESS overlaps/runtime-only/06c0be7b-build.20240815-142905.log
[00:00:01][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       5
    Session:     5
    Fetch Queue: processed 0, skipped 5, failed 0 
    Build Queue: processed 1, skipped 4, failed 0

Notice the warning message about the conflicting file, it is there to inform the user about which files are overlapping, and also which elements are being staged in which order.

Note also that BuildStream does not discover the overlap until the moment that you build a reverse dependency which will require staging of both artifacts.

Tip

The overlaps warning discussed here can be configured to be a fatal warning. This is useful in the case that you want to be strict about avoiding overlapping files in your project.

3.4. Mitigating overlapping files

Since we recently discussed filtering of artifacts, we should note that it is of course possible to handle this case by having hello.bst depend on a filtered version of libhello.bst with the offending file excluded.

However, working with filter elements just for the sake of handling a conflicting artifact would be quite inconvenient, so we have other means.

3.4.1. Whitelisting overlapping files

BuildStream allows explicitly ignoring such errors by adding the files to an overlap whitelist, you could achieve this in the given example by adding the following to the hello.bst element:

public:
  bst:
    overlap-whitelist:
    - |
      %{docdir}/hello.txt

Note

Note that glob patterns are also supported in the whitelist.

3.4.2. Artifact munging

Another way around this situation is the munge the artifacts at install time such that there is no conflict.

This is the easiest approach in the case that you might want to keep the underlying %{docdir}/hello.txt from libhello.bst and discard the same file from hello.bst.

In this case, we might modify the hello.bst file so that it’s install command contain an rm statement, as such:

install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

- |
  # Rid ourselves of the unwanted file at install time
  rm -f %{install-root}%{docdir}/hello.txt

This would cause later builds of runtime-only.bst to no longer conflict on the given file.

3.5. Summary

In this chapter we’ve presented a situation where an artifact can conflict with another artifact by way of providing the same files.

We’ve presented the overlap whitelist public data which is the typical solution for silencing the error when the outcome is desired, and also presented a strategy to deal with cases where you want to keep files from the overlapped artifact instead.