You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by no...@apache.org on 2020/12/29 12:51:58 UTC
[buildstream] 03/08: plugins/elements/stack.py: Require all
dependencies be build & run.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tristan/shell-artifacts
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1cb5d696b845b492ec6fbebf4b35071efe3151cb
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Tue Dec 8 16:52:44 2020 +0900
plugins/elements/stack.py: Require all dependencies be build & run.
Stack elements cannot be build-only dependencies, as this would defeat
the purpose of using stack elements in order to directly build-depend on
them.
Stack element dependencies must all be built in order to build depend
on them, and as such we gain no build parallelism by allowing runtime-only
dependencies on stack elements. Declaring a runtime-only dependency on
a stack element as a whole might still be useful, but still requires the
entire stack to be built at the time we need that stack.
Instead, it is more useful to ensure that a stack element is a logical
group of all dependencies, including runtime dependencies, such that we
can guarantee cache key alignment with all stack dependencies.
This allows for stronger reliability in commands such as
`bst artifact checkout`, which can now reliably download and checkout
a fully built stack as a result, without any uncertainty about possible
runtime-only dependencies which might exist in the project where that
artifact was created.
This consequently closes #1075
This also fixes the following tests such that the no longer
require build-depends or runtime-depends to work in stack elements:
* tests/frontend/default_target.py: Was not necessary to check results of show,
these stacks were set to runtime-depends so that they would have the same
buildable state as their dependencies when shown.
* tests/format/dependencies.py: tests/frontend/pull.py, test/frontend/show.py,
tests/integration/compose.py:
These tests were using specific build/runtime dependencies in stacks, but
for no particular reason.
---
src/buildstream/plugins/elements/stack.py | 95 +++++++++++++++++++++-
.../dependencies1/elements/builddep-list.bst | 4 +-
.../format/dependencies1/elements/list-combine.bst | 2 +-
.../dependencies1/elements/runtimedep-list.bst | 4 +-
.../default-target/elements/dummy_stack.bst | 2 +-
tests/frontend/default_target.py | 44 +++++-----
tests/frontend/pull.py | 2 +-
.../strict-depends/elements/non-strict-depends.bst | 2 +-
.../strict-depends/elements/strict-depends.bst | 2 +-
.../project/elements/compose/test-integration.bst | 2 +-
10 files changed, 126 insertions(+), 33 deletions(-)
diff --git a/src/buildstream/plugins/elements/stack.py b/src/buildstream/plugins/elements/stack.py
index b15f670..bd914ed 100644
--- a/src/buildstream/plugins/elements/stack.py
+++ b/src/buildstream/plugins/elements/stack.py
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2016 Codethink Limited
+# Copyright (C) 2020 Codethink Limited
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -22,9 +22,85 @@ stack - Symbolic Element for dependency grouping
================================================
Stack elements are simply a symbolic element used for representing
a logical group of elements.
+
+All dependencies declared in stack elements must always be both
+:ref:`build and runtime dependencies <format_dependencies_types>`.
+
+**Example:**
+
+.. code:: yaml
+
+ kind: stack
+
+ # Declare all of your dependencies in the `depends` list.
+ depends:
+ - libc.bst
+ - coreutils.bst
+
+.. note::
+
+ Unlike other elements, whose cache keys are a unique identifier
+ of the contents of the artifacts they produce, stack elements do
+ not produce any artifact content. Instead, the cache key of an artifact
+ is a unique identifier for the assembly of its own dependencies.
+
+
+Using intermediate stacks
+-------------------------
+Using a stack element at intermediate levels of your build graph
+allows you to abstract away some parts of your project into logical
+subsystems which elements can more conveniently depend on as a whole.
+
+In addition to the added convenience, it will allow you to more
+easily change the implementation of a subsystem later on, without needing
+to update many reverse dependencies to depend on new elements, or even
+allow you to conditionally implement a subsystem with various implementations
+depending on what :ref:`project options <project_options>` were specified at
+build time.
+
+
+Using toplevel stacks
+---------------------
+Stack elements can also be useful as toplevel targets in your build graph
+to simply indicate all of the components which need to be built for a given
+system to be complete, or for your integration pipeline to be successful.
+
+
+Checking out and deploying toplevel stacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In case that your software is built remotely, it is possible to checkout
+the built content of a stack on your own machine for the purposes of
+inspection or further deployment.
+
+To accomplish this, you will need to know the cache key of the stack element
+which was built remotely, possibly by inspecting the remote build log or by
+deriving it with an equally configured BuildStream project, and you will
+need read access to the artifact cache server which the build was uploaded to,
+this should be configured in your :ref:`user configuration file <config_artifacts>`.
+
+You can then checkout the remotely built stack using the
+:ref:`bst artifact checkout <invoking_artifact_checkout>` command and providing
+it with the :ref:`artifact name <artifact_names>`:
+
+**Example:**
+
+.. code:: shell
+
+ bst artifact checkout --deps build --pull --integrate \\
+ --directory `pwd`/checkout \\
+ project/stack/788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27
+
+.. note::
+
+ It is possible to checkout other elements in the same way, however stack
+ elements are uniquely suited to this purpose, as they cannot have
+ :ref:`runtime only dependencies <format_dependencies_types>`, and consequently
+ their cache keys are always a unique representation of their collective
+ dependencies.
"""
-from buildstream import Element
+from buildstream import Element, ElementError
+from buildstream.types import _Scope
# Element implementation for the 'stack' kind.
@@ -46,7 +122,20 @@ class StackElement(Element):
pass
def preflight(self):
- pass
+
+ # Assert that all dependencies are both build and runtime dependencies.
+ #
+ all_deps = list(self._dependencies(_Scope.ALL, recurse=False))
+ run_deps = list(self._dependencies(_Scope.RUN, recurse=False))
+ build_deps = list(self._dependencies(_Scope.BUILD, recurse=False))
+ if any(dep not in run_deps for dep in all_deps) or any(dep not in build_deps for dep in all_deps):
+ # There is no need to specify the `self` provenance here in preflight() errors, as the base class
+ # will take care of prefixing these for plugin author convenience.
+ raise ElementError(
+ "All dependencies of 'stack' elements must be both build and runtime dependencies",
+ detail="Make sure you declare all dependencies in the `depends` list, without specifying any `type`.",
+ reason="stack-requires-build-and-run",
+ )
def get_unique_key(self):
# We do not add anything to the build, only our dependencies
diff --git a/tests/format/dependencies1/elements/builddep-list.bst b/tests/format/dependencies1/elements/builddep-list.bst
index a0cbcaf..eb216a1 100644
--- a/tests/format/dependencies1/elements/builddep-list.bst
+++ b/tests/format/dependencies1/elements/builddep-list.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
description: This element has a build-only dependency specified via build-depends
build-depends:
- - firstdep.bst
+- firstdep.bst
diff --git a/tests/format/dependencies1/elements/list-combine.bst b/tests/format/dependencies1/elements/list-combine.bst
index ed34522..d39ddd3 100644
--- a/tests/format/dependencies1/elements/list-combine.bst
+++ b/tests/format/dependencies1/elements/list-combine.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
description: This element depends on three elements in different ways
build-depends:
- firstdep.bst
diff --git a/tests/format/dependencies1/elements/runtimedep-list.bst b/tests/format/dependencies1/elements/runtimedep-list.bst
index 1207a49..eaa0cd2 100644
--- a/tests/format/dependencies1/elements/runtimedep-list.bst
+++ b/tests/format/dependencies1/elements/runtimedep-list.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
description: This element has a runtime-only dependency
runtime-depends:
- - firstdep.bst
+- firstdep.bst
diff --git a/tests/frontend/default-target/elements/dummy_stack.bst b/tests/frontend/default-target/elements/dummy_stack.bst
index 5f92166..3ea51b0 100644
--- a/tests/frontend/default-target/elements/dummy_stack.bst
+++ b/tests/frontend/default-target/elements/dummy_stack.bst
@@ -1,5 +1,5 @@
kind: stack
-runtime-depends:
+depends:
- dummy_1.bst
- dummy_2.bst
diff --git a/tests/frontend/default_target.py b/tests/frontend/default_target.py
index bb7a495..60578bb 100644
--- a/tests/frontend/default_target.py
+++ b/tests/frontend/default_target.py
@@ -13,27 +13,27 @@ from tests.testutils import create_artifact_share
DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "default-target",)
-###################################
-# build/show operations #
-###################################
-
-
+#
+# When no target is default, then expect all targets to be built
+#
@pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_no_default(cli, datafiles, operation, expected_state):
+def test_no_default(cli, datafiles):
project = str(datafiles)
all_targets = ["dummy_1.bst", "dummy_2.bst", "dummy_3.bst", "dummy_stack.bst"]
- result = cli.run(project=project, args=[operation])
+ result = cli.run(project=project, args=["build"])
result.assert_success()
states = cli.get_element_states(project, all_targets)
- assert all(states[e] == expected_state for e in all_targets)
+ assert all(states[e] == "cached" for e in all_targets)
+#
+# When the stack is specified as the default target, then
+# expect only it and it's dependencies to be built
+#
@pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_default_target(cli, datafiles, operation, expected_state):
+def test_default_target(cli, datafiles):
project = str(datafiles)
project_path = os.path.join(project, "project.conf")
@@ -49,19 +49,23 @@ def test_default_target(cli, datafiles, operation, expected_state):
# dummy_stack only depends on dummy_1 and dummy_2, but not dummy_3
all_targets = ["dummy_1.bst", "dummy_2.bst", "dummy_stack.bst"]
- result = cli.run(project=project, args=[operation])
+ result = cli.run(project=project, args=["build"])
result.assert_success()
states = cli.get_element_states(project, all_targets)
- assert all(states[e] == expected_state for e in all_targets)
+ assert all(states[e] == "cached" for e in all_targets)
# assert that dummy_3 isn't included in the output
assert "dummy_3.bst" not in states
+#
+# Even when there is a junction, expect that the elements in the
+# subproject referred to by the toplevel project are built when
+# calling `bst build` and no default is specified.
+#
@pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_no_default_with_junction(cli, datafiles, operation, expected_state):
+def test_no_default_with_junction(cli, datafiles):
project = str(datafiles)
junction_path = os.path.join(project, "elements", "junction.bst")
target_path = os.path.join(project, "elements", "junction-target.bst")
@@ -71,16 +75,16 @@ def test_no_default_with_junction(cli, datafiles, operation, expected_state):
_yaml.roundtrip_dump(junction_config, junction_path)
# Then, create a stack element with dependency on cross junction element
- target_config = {"kind": "stack", "runtime-depends": ["junction.bst:dummy_subproject.bst"]}
+ target_config = {"kind": "stack", "depends": ["junction.bst:dummy_subproject.bst"]}
_yaml.roundtrip_dump(target_config, target_path)
- # Now try to perform the specified operation.
+ # Now try to perform a build
# This should automatically fetch the junction at load time.
- result = cli.run(project=project, args=[operation])
+ result = cli.run(project=project, args=["build"])
result.assert_success()
- assert cli.get_element_state(project, "junction.bst:dummy_subproject.bst") == expected_state
- assert cli.get_element_state(project, "junction-target.bst") == expected_state
+ assert cli.get_element_state(project, "junction.bst:dummy_subproject.bst") == "cached"
+ assert cli.get_element_state(project, "junction-target.bst") == "cached"
###################################
diff --git a/tests/frontend/pull.py b/tests/frontend/pull.py
index c1fa76a..f873ad9 100644
--- a/tests/frontend/pull.py
+++ b/tests/frontend/pull.py
@@ -338,7 +338,7 @@ def test_pull_missing_local_blob(cli, tmpdir, datafiles):
_yaml.roundtrip_dump(input_config, input_file)
depends_name = "depends.bst"
- depends_config = {"kind": "stack", "depends": [{"filename": input_name, "type": "build"}]}
+ depends_config = {"kind": "stack", "depends": [input_name]}
depends_file = os.path.join(element_dir, depends_name)
_yaml.roundtrip_dump(depends_config, depends_file)
diff --git a/tests/frontend/strict-depends/elements/non-strict-depends.bst b/tests/frontend/strict-depends/elements/non-strict-depends.bst
index 9ab119b..e0ed21e 100644
--- a/tests/frontend/strict-depends/elements/non-strict-depends.bst
+++ b/tests/frontend/strict-depends/elements/non-strict-depends.bst
@@ -1,4 +1,4 @@
kind: stack
-build-depends:
+depends:
- base.bst
diff --git a/tests/frontend/strict-depends/elements/strict-depends.bst b/tests/frontend/strict-depends/elements/strict-depends.bst
index 1e4e294..10f3b94 100644
--- a/tests/frontend/strict-depends/elements/strict-depends.bst
+++ b/tests/frontend/strict-depends/elements/strict-depends.bst
@@ -1,5 +1,5 @@
kind: stack
-build-depends:
+depends:
- filename: base.bst
strict: true
diff --git a/tests/integration/project/elements/compose/test-integration.bst b/tests/integration/project/elements/compose/test-integration.bst
index 2f9faf1..ef47b3b 100644
--- a/tests/integration/project/elements/compose/test-integration.bst
+++ b/tests/integration/project/elements/compose/test-integration.bst
@@ -1,6 +1,6 @@
kind: stack
-runtime-depends:
+depends:
- base.bst
public: