Writing documentation

BuildStream starts out as a documented project from day one and uses sphinx to document itself.

This section discusses formatting policies for editing files in the doc/source directory, and describes the details of how the docs are generated so that you can easily generate and view the docs yourself before submitting patches to the documentation.

For details on how API documenting comments and docstrings are formatted, refer to the documenting section of the coding guidelines.

Documentation formatting policy

The BuildStream documentation style is as follows:

  • Titles and headings require two leading empty lines above them. Only the first word in a title should be capitalized.

    • If there is an .. _internal_link: anchor, there should be two empty lines above the anchor, followed by one leading empty line.

  • Within a section, paragraphs should be separated by one empty line.

  • Notes are defined using: .. note:: blocks, followed by an empty line and then indented (3 spaces) text.

    • Other kinds of notes can be used throughout the documentation and will be decorated in different ways, these work in the same way as .. note:: does.

      Feel free to also use .. attention:: or .. important:: to call special attention to a paragraph, .. tip:: to give the reader a special tip on how to use an advanced feature or .. warning:: to warn the user about a potential misuse of the API and explain its consequences.

  • Code blocks are defined using: .. code:: LANGUAGE blocks, followed by an empty line and then indented (3 spaces) text. Note that the default language is python.

  • Cross references should be of the form :role:`target`.

    • Explicit anchors can be declared as .. _anchor_name: on a line by itself.

    • To cross reference arbitrary locations with, for example, the anchor anchor_name, always provide some explicit text in the link instead of deriving the text from the target, e.g.: :ref:`Link text <anchor_name>`. Note that the “_” prefix is not used when referring to the target.

For further information about using the reStructuredText with sphinx, please see the Sphinx Documentation.

Building Docs

Before you can build the docs, you will end to ensure that you have installed the required build dependencies as mentioned in the testing section above.

To build the documentation, just run the following:

tox -e docs

This will give you a doc/build/html directory with the html docs which you can view in your browser locally to test.

Regenerating session html

The documentation build will build the session files if they are missing, or if explicitly asked to rebuild. We revision the generated session html files in order to reduce the burden on documentation contributors.

To explicitly rebuild the session snapshot html files, it is recommended that you first set the BST_SOURCE_CACHE environment variable to your source cache, this will make the docs build reuse already downloaded sources:

export BST_SOURCE_CACHE=~/.cache/buildstream/sources

To force rebuild session html while building the doc, simply run tox with the BST_FORCE_SESSION_REBUILD environment variable set, like so:

env BST_FORCE_SESSION_REBUILD=1 tox -e docs

Man pages

Unfortunately it is quite difficult to integrate the man pages build into the setup.py, as such, whenever the frontend command line interface changes, the static man pages should be regenerated and committed with that.

To do this, run the following from the the toplevel directory of BuildStream:

tox -e man

And commit the result, ensuring that you have added anything in the man/ subdirectory, which will be automatically included in the buildstream distribution.

User guide

The user guide is comprised of free form documentation in manually written .rst files and is split up into a few sections, of main interest are the tutorial and the examples.

The distinction of the two categories of user guides is important to understand too.

  • Tutorial

    The tutorial is structured as a series of exercises which start with the most basic concepts and build upon the previous chapters in order to arrive at a basic understanding of how to create BuildStream projects.

    This series of examples should be easy enough to complete in a matter of a few hours for a new user, and should provide just enough insight to get the user started in creating their own projects.

    Going through the tutorial step by step should also result in the user becoming proficient enough with the reference manual to get by on their own.

  • Examples

    These exist to demonstrate how to accomplish more advanced tasks which are not always obvious and discoverable.

    Alternatively, these also demonstrate elegant and recommended ways of accomplishing some tasks which could be done in various ways.

Guidelines

Here are some general guidelines for adding new free form documentation to the user guide.

  • Focus on a single subject

    It is important to stay focused on a single subject and avoid getting into tangential material when creating a new entry, so that the articles remain concise and the user is not distracted by unrelated subject material.

    A single tutorial chapter or example should not introduce any additional subject material than the material being added for the given example.

  • Reuse existing sample project elements

    To help avoid distracting from the topic at hand, it is always preferable to reuse the same project sample material from other examples and only deviate slightly to demonstrate the new material, than to create completely new projects.

    This helps us remain focused on a single topic at a time, and reduces the amount of unrelated material the reader needs to learn in order to digest the new example.

  • Don’t be redundant

    When something has already been explained in the tutorial or in another example, it is best to simply refer to the other user guide entry in a new example.

    Always prefer to link to the tutorial if an explanation exists in the tutorial, rather than linking to another example, where possible.

  • Link into the reference manual at every opportunity

    The format and plugin API is 100% documented at all times. Whenever discussing anything about the format or plugin API, always do so while providing a link into the more terse reference material.

    We don’t want users to have to search for the material themselves, and we also want the user to become proficient at navigating the reference material over time.

  • Use concise terminology

    As developers, we tend to come up with code names for features we develop, and then end up documenting a new feature in an example.

    Never use a code name or shorthand to refer to a feature in the user guide, instead always use fully qualified sentences outlining very explicitly what we are doing in the example, or what the example is for in the case of a title.

    We need to be considerate that the audience of our user guide is probably a proficient developer or integrator, but has no idea what we might have decided to name a given activity.

Structure of an example

The tutorial and the examples sections of the documentation contain a series of sample projects, each chapter in the tutorial, or standalone example uses a sample project.

Here is the the structure for adding new examples and tutorial chapters.

  • The example has a ${name}.

  • The example has a project users can copy and use.

    • This project is added in the directory doc/examples/${name}.

  • The example has a documentation component.

    • This is added at doc/source/examples/${name}.rst

    • An entry for examples/${name} is added to the toctree in doc/source/using_examples.rst

    • This documentation discusses the project elements declared in the project and may provide some BuildStream command examples.

    • This documentation links out to the reference manual at every opportunity.

    Note

    In the case of a tutorial chapter, the .rst file is added in at doc/source/tutorial/${name}.rst and an entry for tutorial/${name} is added to doc/source/using_tutorial.rst.

  • The example has a CI test component.

    • This is an integration test added at tests/examples/${name}.

    • This test runs BuildStream in the ways described in the example and assert that we get the results which we advertize to users in the said examples.

Adding BuildStream command output

As a part of building the docs, BuildStream will run itself and extract some html for the colorized output which is produced.

If you want to run BuildStream to produce some nice html for your documentation, then you can do so by adding new .run files to the doc/sessions/ directory.

Any files added as doc/sessions/${example}.run will result in generated file at doc/source/sessions/${example}.html, and these files can be included in the reStructuredText documentation at any time with:

.. raw:: html
   :file: sessions/${example}.html

The .run file format is just another YAML dictionary which consists of a commands list, instructing the program what to do command by command.

Each command is a dictionary, the members of which are listed here:

  • directory: The input file relative project directory.

  • output: The input file relative output html file to generate (optional).

  • fake-output: Don’t really run the command, just pretend to and pretend this was the output, an empty string will enable this too.

  • command: The command to run, without the leading bst.

  • shell: Specifying True indicates that command should be run as a shell command from the project directory, instead of a bst command (optional).

When adding a new .run file, one should normally also commit the new resulting generated .html file(s) into the doc/source/sessions-stored/ directory at the same time, this ensures that other developers do not need to regenerate them locally in order to build the docs.

Example:

commands:

# Make it fetch first
- directory: ../examples/foo
  command: source fetch hello.bst

# Capture a build output
- directory: ../examples/foo
  output: ../source/sessions/foo-build.html
  command: build hello.bst