[buildstream] 02/02: doc: Document setting up remote execution services.

[tv...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tvb pushed a commit to branch tristan/document-remote-execution-configuration
in repository https://gitbox.apache.org/repos/asf/buildstream.git

commit 9680ac4def18c9005ec0f9cb0af627e792eab344
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Dec 29 16:48:09 2021 +0900

    doc: Document setting up remote execution services.
    
    This fixes #1324
---
 doc/source/main_using.rst                         |  1 +
 doc/source/using_configuring_remote_execution.rst | 67 +++++++++++++++++++++++
 2 files changed, 68 insertions(+)

diff --git a/doc/source/main_using.rst b/doc/source/main_using.rst
index ddda44a..c6cd385 100644
--- a/doc/source/main_using.rst
+++ b/doc/source/main_using.rst
@@ -18,4 +18,5 @@ guides and information on user preferences and configuration.
    using_config
    using_commands
    using_configuring_cache_server
+   using_configuring_remote_execution
    using_examples
diff --git a/doc/source/using_configuring_remote_execution.rst b/doc/source/using_configuring_remote_execution.rst
new file mode 100644
index 0000000..759e40f
--- /dev/null
+++ b/doc/source/using_configuring_remote_execution.rst
@@ -0,0 +1,67 @@
+
+
+.. _remote_execution_servers:
+
+Remote Execution Servers
+========================
+BuildStream supports building remotely using the
+`Google Remote Execution API (REAPI). <https://github.com/bazelbuild/remote-apis>`_, which
+has various known implementations.
+
+Some of these implementations include:
+
+* `BuildGrid <https://buildgrid.build/>`_
+* `BuildBarn <https://github.com/buildbarn>`_
+* `Buildfarm <https://github.com/bazelbuild/bazel-buildfarm>`_
+
+These various implementations implement the `Google Remote Execution API (REAPI)
+<https://github.com/bazelbuild/remote-apis>`_ to various degrees as these projects
+have different priorities.
+
+On the client side, the remote execution service to use can be
+specified in the :ref:`user configuration <user_config_remote_execution>`.
+
+
+BuildStream specific requirements
+---------------------------------
+In order for BuildStream to work correctly with a remote execution cluster, there
+are a couple of requirements that implementation needs to meet.
+
+
+Implementation of platform properties
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The remote execution service must properly implement `platform properties
+<https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/platform.md>`_.
+
+This is crucial because BuildStream needs to be guaranteed the correct operating
+system and ISA which it requests from the service.
+
+
+Staging the input root as the filesystem root
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+BuildStream requires that the *input root* given to the remote execution service
+be treated as the absolute *filesystem root*.
+
+This is because BuildStream provides guarantees that all build dependencies, including
+the base runtime and compilers, are defined by elements and run within a sandboxed and
+isolated build environment, but the `REAPI <https://github.com/bazelbuild/remote-apis>`_
+was originally developped without this determinism and control in mind. Instead, typically
+it is up to the user to configure a cluster to use a docker image to build payloads
+with, rather than allowing the REAPI client to control the entire sandbox.
+
+Unfortunately the ability to dictate that the *input root* be treated as the *filesystem root*
+in a container on remote workers in the cluster is not yet standardized in the REAPI protocol.
+
+.. note::
+
+   The *input root* is referred to as the ``input_root_digest`` member of the ``Action`` message
+   as defined in the `protocol <https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto>`_
+
+
+Example working configuration
+-----------------------------
+A simple configuration to spin up the `BuildGrid <https://buildgrid.build/>`_ service using
+`docker compose <https://docs.docker.com/compose/>`_ follows:
+
+.. literalinclude:: ../../.github/compose/ci.buildgrid.yml
+    :language: yaml