You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2021/12/29 07:49:35 UTC
[buildstream] branch tristan/document-remote-execution-configuration created (now 9680ac4)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch tristan/document-remote-execution-configuration
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 9680ac4 doc: Document setting up remote execution services.
This branch includes the following new commits:
new 101f616 _project.py: Don't consider `remote-execution` as a valid toplevel key.
new 9680ac4 doc: Document setting up remote execution services.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 02/02: doc: Document setting up remote execution services.
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/document-remote-execution-configuration
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9680ac4def18c9005ec0f9cb0af627e792eab344
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Dec 29 16:48:09 2021 +0900
doc: Document setting up remote execution services.
This fixes #1324
---
doc/source/main_using.rst | 1 +
doc/source/using_configuring_remote_execution.rst | 67 +++++++++++++++++++++++
2 files changed, 68 insertions(+)
diff --git a/doc/source/main_using.rst b/doc/source/main_using.rst
index ddda44a..c6cd385 100644
--- a/doc/source/main_using.rst
+++ b/doc/source/main_using.rst
@@ -18,4 +18,5 @@ guides and information on user preferences and configuration.
using_config
using_commands
using_configuring_cache_server
+ using_configuring_remote_execution
using_examples
diff --git a/doc/source/using_configuring_remote_execution.rst b/doc/source/using_configuring_remote_execution.rst
new file mode 100644
index 0000000..759e40f
--- /dev/null
+++ b/doc/source/using_configuring_remote_execution.rst
@@ -0,0 +1,67 @@
+
+
+.. _remote_execution_servers:
+
+Remote Execution Servers
+========================
+BuildStream supports building remotely using the
+`Google Remote Execution API (REAPI). <https://github.com/bazelbuild/remote-apis>`_, which
+has various known implementations.
+
+Some of these implementations include:
+
+* `BuildGrid <https://buildgrid.build/>`_
+* `BuildBarn <https://github.com/buildbarn>`_
+* `Buildfarm <https://github.com/bazelbuild/bazel-buildfarm>`_
+
+These various implementations implement the `Google Remote Execution API (REAPI)
+<https://github.com/bazelbuild/remote-apis>`_ to various degrees as these projects
+have different priorities.
+
+On the client side, the remote execution service to use can be
+specified in the :ref:`user configuration <user_config_remote_execution>`.
+
+
+BuildStream specific requirements
+---------------------------------
+In order for BuildStream to work correctly with a remote execution cluster, there
+are a couple of requirements that implementation needs to meet.
+
+
+Implementation of platform properties
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The remote execution service must properly implement `platform properties
+<https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/platform.md>`_.
+
+This is crucial because BuildStream needs to be guaranteed the correct operating
+system and ISA which it requests from the service.
+
+
+Staging the input root as the filesystem root
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+BuildStream requires that the *input root* given to the remote execution service
+be treated as the absolute *filesystem root*.
+
+This is because BuildStream provides guarantees that all build dependencies, including
+the base runtime and compilers, are defined by elements and run within a sandboxed and
+isolated build environment, but the `REAPI <https://github.com/bazelbuild/remote-apis>`_
+was originally developped without this determinism and control in mind. Instead, typically
+it is up to the user to configure a cluster to use a docker image to build payloads
+with, rather than allowing the REAPI client to control the entire sandbox.
+
+Unfortunately the ability to dictate that the *input root* be treated as the *filesystem root*
+in a container on remote workers in the cluster is not yet standardized in the REAPI protocol.
+
+.. note::
+
+ The *input root* is referred to as the ``input_root_digest`` member of the ``Action`` message
+ as defined in the `protocol <https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto>`_
+
+
+Example working configuration
+-----------------------------
+A simple configuration to spin up the `BuildGrid <https://buildgrid.build/>`_ service using
+`docker compose <https://docs.docker.com/compose/>`_ follows:
+
+.. literalinclude:: ../../.github/compose/ci.buildgrid.yml
+ :language: yaml
[buildstream] 01/02: _project.py: Don't consider `remote-execution` as a valid toplevel key.
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/document-remote-execution-configuration
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 101f6163d8e4cf85db2f36a6ce7505c7a582b00c
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Dec 29 15:45:10 2021 +0900
_project.py: Don't consider `remote-execution` as a valid toplevel key.
Since commit 497e1caf1b51773be4b1473519da0ab543148d72, we removed
the ability for a project to specify a preferred remote execution
configuration, but forgot to remove `remote-execution` from validation
of the toplevel project.conf YAML.
---
src/buildstream/_project.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/buildstream/_project.py b/src/buildstream/_project.py
index bca3a6a..e7e24b5 100644
--- a/src/buildstream/_project.py
+++ b/src/buildstream/_project.py
@@ -638,7 +638,6 @@ class Project:
"ref-storage",
"sandbox",
"mirrors",
- "remote-execution",
"sources",
"source-caches",
"junctions",