You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by ro...@apache.org on 2020/12/29 13:38:06 UTC
[buildstream] branch variants-slow-functional-loopy created (now
25abdcd)
This is an automated email from the ASF dual-hosted git repository.
root pushed a change to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 25abdcd Slow but functioning algorithm for resolving variants
This branch includes the following new commits:
new fea302e local.py source plugin: Fixed to support symlinks
new 5732ed9 _loader.py / _metaelement.py: Pass along the selected variant
new c51d6c9 element.py: Added public variant attribute
new 8a05ad5 widget.py: Support %{variant} formatting in show_pipeline()
new 1d9fdf0 userconfig.yaml: Add %{variant} to default formatting
new f876d30 main.py: Use context.log_element_format as default format for `bst show`
new 9ce417d exceptions.py: Added new LoadErrorReason INVALID_VARIANT
new 97e66e2 _loader.py: Raise LoadError() for invalid variant requests.
new ce389ed variants.py tests: Added test cases to ensure LoadError() is raised for invalid variant requests
new 498e24d _loader.py: Fixed variant priority in variant resolution
new 92e8604 variant tests: Added tests to ensure default variant is chosen
new 0e4efd7 Added .coveragerc and use that in setup.cfg
new f7c6062 Committing crazy non-functional alternative with cached code paths.
new 25abdcd Slow but functioning algorithm for resolving variants
The 14 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 01/14: local.py source plugin: Fixed to support
symlinks
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit fea302e65836bf66edc0b780f6ddb8fe3ffd8378
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 14:59:25 2017 +0900
local.py source plugin: Fixed to support symlinks
This was otherwise throwing an exception when trying
the create the checksum for a symlink.
---
buildstream/plugins/sources/local.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/buildstream/plugins/sources/local.py b/buildstream/plugins/sources/local.py
index 4fdbbd0..a2f0992 100644
--- a/buildstream/plugins/sources/local.py
+++ b/buildstream/plugins/sources/local.py
@@ -97,6 +97,8 @@ def sha256sum(filename):
# If it's a directory, just return 0 string
if os.path.isdir(filename):
return "0"
+ elif os.path.islink(filename):
+ return "1"
h = hashlib.sha256()
with open(filename, "rb") as f:
[buildstream] 02/14: _loader.py / _metaelement.py: Pass along the
selected variant
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 5732ed9722b7c128c8ea632701d2fecf89a74f59
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 16:12:40 2017 +0900
_loader.py / _metaelement.py: Pass along the selected variant
Dont lose the selected variant of elements, pass them along to the metaelement at load time.
---
buildstream/_loader.py | 4 +++-
buildstream/_metaelement.py | 4 +++-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/buildstream/_loader.py b/buildstream/_loader.py
index fd2eef6..3c90e43 100644
--- a/buildstream/_loader.py
+++ b/buildstream/_loader.py
@@ -767,7 +767,9 @@ class Loader():
provenance.filename)
meta_sources.append(meta_source)
- meta_element = MetaElement(element_name, data.get('kind'), elt_provenance, meta_sources,
+ kind = _yaml.node_get(data, str, Symbol.KIND)
+ meta_element = MetaElement(element_name, kind, element.variant_name,
+ elt_provenance, meta_sources,
_yaml.node_get(data, Mapping, Symbol.CONFIG, default_value={}),
_yaml.node_get(data, Mapping, Symbol.VARIABLES, default_value={}),
_yaml.node_get(data, Mapping, Symbol.ENVIRONMENT, default_value={}),
diff --git a/buildstream/_metaelement.py b/buildstream/_metaelement.py
index bc8f36a..d7462cc 100644
--- a/buildstream/_metaelement.py
+++ b/buildstream/_metaelement.py
@@ -28,6 +28,7 @@ class MetaElement():
# Args:
# name: The resolved element name
# kind: The element kind
+ # variant: The element variant, or None
# provenance: The provenance of the element
# sources: An array of MetaSource objects
# config: The configuration data for the element
@@ -36,8 +37,9 @@ class MetaElement():
# env_nocache: List of environment vars which should not be considered in cache keys
# public: Public domain data dictionary
#
- def __init__(self, name, kind, provenance, sources, config, variables, environment, env_nocache, public):
+ def __init__(self, name, kind, variant, provenance, sources, config, variables, environment, env_nocache, public):
self.name = name
+ self.variant = variant
self.kind = kind
self.provenance = provenance
self.sources = sources
[buildstream] 11/14: variant tests: Added tests to ensure default
variant is chosen
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 92e8604fc2464a5717ecbdaa8eb76cc83dc5ed44
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 19:22:36 2017 +0900
variant tests: Added tests to ensure default variant is chosen
Both for when depending on a toplevel with variants, or when
ambivalently depending on an element with variants.
---
tests/loader/variants.py | 28 ++++++++++++++++++++++
.../elements/depends-on-element-with-variants.bst | 4 ++++
2 files changed, 32 insertions(+)
diff --git a/tests/loader/variants.py b/tests/loader/variants.py
index e0afe19..9a3b186 100644
--- a/tests/loader/variants.py
+++ b/tests/loader/variants.py
@@ -123,6 +123,34 @@ def test_variant_invalid_unused_dependency(datafiles):
##############################################################
+# Test Variant Default and Priority #
+##############################################################
+@pytest.mark.datafiles(DATA_DIR)
+def test_variant_default_target(datafiles):
+
+ # Assert that the default (first) variant is chosen for a toplevel target with variants.
+ basedir = os.path.join(datafiles.dirname, datafiles.basename)
+ loader = Loader(
+ basedir, 'elements/simple-variant-compositing.bst', None, None, None)
+
+ element = loader.load()
+ assert(element.variant == 'pink')
+
+
+@pytest.mark.datafiles(DATA_DIR)
+def test_variant_default_dependency(datafiles):
+
+ # Assert that the default (first) variant is chosen for a dependency of a toplevel
+ basedir = os.path.join(datafiles.dirname, datafiles.basename)
+ loader = Loader(
+ basedir, 'elements/depends-on-element-with-variants.bst', None, None, None)
+
+ element = loader.load()
+ dependency = element.dependencies[0]
+ assert(dependency.variant == 'pink')
+
+
+##############################################################
# Test Simple Variant Compositing #
##############################################################
@pytest.mark.datafiles(DATA_DIR)
diff --git a/tests/loader/variants/elements/depends-on-element-with-variants.bst b/tests/loader/variants/elements/depends-on-element-with-variants.bst
new file mode 100644
index 0000000..b58eb69
--- /dev/null
+++ b/tests/loader/variants/elements/depends-on-element-with-variants.bst
@@ -0,0 +1,4 @@
+kind: pony
+description: An element which depends on an element with variants
+depends:
+- elements/simple-dependency-variants.bst
[buildstream] 03/14: element.py: Added public variant attribute
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit c51d6c9007875803378b3201d204c0cfc3ef94c1
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 16:13:20 2017 +0900
element.py: Added public variant attribute
Reflects the selected variant at load time, otherwise None for
elements which do not declare any variants.
---
buildstream/element.py | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/buildstream/element.py b/buildstream/element.py
index 64d7204..0974388 100644
--- a/buildstream/element.py
+++ b/buildstream/element.py
@@ -99,6 +99,14 @@ class Element(Plugin):
and creating directory names and such.
"""
+ self.variant = meta.variant
+ """The effective variant for this element
+
+ If the element declares variants, then this will reflect the
+ variant which was chosen and resolved at load time, otherwise
+ this will be ``None``.
+ """
+
self.__runtime_dependencies = [] # Direct runtime dependency Elements
self.__build_dependencies = [] # Direct build dependency Elements
self.__sources = [] # List of Sources
[buildstream] 14/14: Slow but functioning algorithm for resolving
variants
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 25abdcd0a85bf4613f29f3938ed8b86ffe0279c6
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sun Jul 16 18:26:32 2017 +0900
Slow but functioning algorithm for resolving variants
---
buildstream/_loader.py | 90 ++++++++++----------------------------------------
1 file changed, 18 insertions(+), 72 deletions(-)
diff --git a/buildstream/_loader.py b/buildstream/_loader.py
index db31e21..6e1c70d 100644
--- a/buildstream/_loader.py
+++ b/buildstream/_loader.py
@@ -575,7 +575,7 @@ class Loader():
# })
#
- pools = self.configure_variants(toplevel_config, {})
+ pools = self.try_element_configuration(toplevel_config, {})
except VariantDisagreement as e:
raise LoadError(LoadErrorReason.VARIANT_DISAGREEMENT, str(e)) from e
@@ -586,7 +586,7 @@ class Loader():
element_config.element.apply_element_config(element_config)
#
- # configure_variants()
+ # try_element_configuration()
#
# Args:
# element_config (LoadElementConfig): the element to try
@@ -601,36 +601,31 @@ class Loader():
# the given configuration and the first valid configuration of its
# dependencies
#
- def configure_variants(self, element_config, pool, depth=0, visited=None, visit_count=None):
+ def try_element_configuration(self, element_config, pool, depth=0):
# print("{}TRY: {}".format(' ' * depth, element_config))
if element_config.filename in pool:
config = pool[element_config.filename]
- # The configuration pool can have only one selected configuration
- # for each element, handle intersections and conflicts.
- #
- if config.element is element_config.element:
-
- if config.variant_name != element_config.variant_name:
+ if config.variant_name != element_config.variant_name:
- # Two different variants of the same element should be reached
- # on a path of variant agreement.
- raise VariantDisagreement(element_config, config)
+ # Two different variants of the same element should be reached
+ # on a path of variant agreement.
+ raise VariantDisagreement(element_config, config)
- else:
- # A path converges on the same element configuration,
- # no need to recurse as we already have a result for this.
- return [pool]
+ else:
+ # A path converges on the same element configuration,
+ # no need to recurse as we already have a result for this.
+ return [pool]
# Now add ourselves to the pool and recurse into the dependency list
new_pool = dict(pool)
new_pool[element_config.filename] = element_config
- return self.configure_dependency_variants(element_config, element_config.deps, new_pool, depth=depth + 1, visited=visited, visit_count=visit_count)
+ return self.configure_dependency_variants(element_config, element_config.deps, new_pool, depth=depth + 1)
- def configure_dependency_variants(self, parent_config, deps, pool, depth=0, visited=None, visit_count=None):
+ def configure_dependency_variants(self, parent_config, deps, pool, depth=0):
# This is just the end of the list
if not deps:
@@ -661,54 +656,11 @@ class Loader():
valid_pools = []
for element_config in element_configs_to_try:
- # XXX DEBUG START
- if visited is None:
- visited = {}
- if visit_count is None:
- visit_count = {}
-
- iter_key = (
- parent_config.filename, parent_config.variant_name,
- element_config.filename, element_config.variant_name
- )
- count = visit_count.get(iter_key, 0)
- if count > 0:
- print("Visited path {} times: {}".format(count, element_config.make_path()))
-
- visit_count[iter_key] = count + 1
- # XXX DEBUG END
-
- iter_result = visited.get(iter_key)
-
- if iter_result is not None:
- valid_pools += iter_result['pools']
- if iter_result['error']:
- last_error = iter_result['error']
- print("Assigning {} valid pools with error {}".format(len(iter_result['pools']), iter_result['error']))
- continue
- else:
- print("Fresh iteration")
-
- iter_result = {
- 'pools': [],
- 'error': None
- }
-
- config_pools = []
- iter_error = None
-
# Recurse into this dependency for this config first
try:
- try_pools = self.configure_variants(element_config, pool, depth=depth, visited=visited, visit_count=visit_count)
+ try_pools = self.try_element_configuration(element_config, pool, depth=depth)
except VariantDisagreement as e:
- last_error = iter_error = e
-
- # XXX
-
- iter_result['pools'] = config_pools
- iter_result['error'] = iter_error
- visited[iter_key] = iter_result
-
+ last_error = e
continue
# For each valid configuration for this element
@@ -716,18 +668,12 @@ class Loader():
# Recurse into the siblings, siblings either pass of fail as a whole
try:
- config_pools += self.configure_dependency_variants(parent_config, deps[1:], try_pool,
- depth=depth + 1, visited=visited, visit_count=visit_count)
+ valid_pools += self.configure_dependency_variants(parent_config, deps[1:], try_pool,
+ depth=depth + 1)
except VariantDisagreement as e:
- last_error = iter_error = e
+ last_error = e
continue
- iter_result['pools'] = config_pools
- iter_result['error'] = iter_error
- visited[iter_key] = iter_result
-
- valid_pools += config_pools
-
# If unable to find any valid configuration, raise a VariantDisagreement
if not valid_pools:
raise last_error
[buildstream] 04/14: widget.py: Support %{variant} formatting in
show_pipeline()
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 8a05ad5228173c2ad9e08ac7253aa191b46dda69
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 16:14:09 2017 +0900
widget.py: Support %{variant} formatting in show_pipeline()
---
buildstream/_frontend/widget.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/buildstream/_frontend/widget.py b/buildstream/_frontend/widget.py
index 2c0cc18..67ff836 100644
--- a/buildstream/_frontend/widget.py
+++ b/buildstream/_frontend/widget.py
@@ -545,6 +545,9 @@ class LogLine(Widget):
else:
line = p.fmt_subst(line, 'state', "waiting", fg='blue')
+ # The variant
+ line = p.fmt_subst(line, 'variant', element.variant or "", fg='yellow', bold=True, dim=True)
+
# Element configuration
if "%{config" in format:
config = _yaml.node_sanitize(element._Element__config)
[buildstream] 09/14: variants.py tests: Added test cases to ensure
LoadError() is raised for invalid variant requests
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit ce389ed957045b300600786dd77e4d8a62ff157b
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 18:23:14 2017 +0900
variants.py tests: Added test cases to ensure LoadError() is raised for invalid variant requests
---
tests/loader/variants.py | 47 ++++++++++++++++++++++
.../elements/simple-unused-variant-invalid.bst | 14 +++++++
.../variants/elements/simple-variant-invalid.bst | 5 +++
3 files changed, 66 insertions(+)
diff --git a/tests/loader/variants.py b/tests/loader/variants.py
index 865ffa2..e0afe19 100644
--- a/tests/loader/variants.py
+++ b/tests/loader/variants.py
@@ -76,6 +76,53 @@ def test_variant_illegal_composite(datafiles):
##############################################################
+# Test Invalid Variant Requests #
+##############################################################
+@pytest.mark.datafiles(DATA_DIR)
+def test_variant_invalid_target(datafiles):
+
+ # Test that an invalid variant requested as the pipeline target raises the appropriate error
+ basedir = os.path.join(datafiles.dirname, datafiles.basename)
+ loader = Loader(
+ basedir, 'elements/simple-variant-compositing.bst', 'green', None, None)
+
+ with pytest.raises(LoadError) as exc:
+ element = loader.load()
+
+ assert (exc.value.reason == LoadErrorReason.INVALID_VARIANT)
+
+
+@pytest.mark.datafiles(DATA_DIR)
+def test_variant_invalid_dependency(datafiles):
+
+ # Test that an invalid variant requested as an element dependency raises the appropriate error
+ basedir = os.path.join(datafiles.dirname, datafiles.basename)
+ loader = Loader(
+ basedir, 'elements/simple-variant-invalid.bst', None, None, None)
+
+ with pytest.raises(LoadError) as exc:
+ element = loader.load()
+
+ assert (exc.value.reason == LoadErrorReason.INVALID_VARIANT)
+
+
+@pytest.mark.datafiles(DATA_DIR)
+def test_variant_invalid_unused_dependency(datafiles):
+
+ # Test that an invalid variant requested as an element dependency which is
+ # unused in the effective variant resolution still raises the appropriate
+ # error (ensure that errors occur even on unused variant paths)
+ basedir = os.path.join(datafiles.dirname, datafiles.basename)
+ loader = Loader(
+ basedir, 'elements/simple-unused-variant-invalid.bst', 'blue', None, None)
+
+ with pytest.raises(LoadError) as exc:
+ element = loader.load()
+
+ assert (exc.value.reason == LoadErrorReason.INVALID_VARIANT)
+
+
+##############################################################
# Test Simple Variant Compositing #
##############################################################
@pytest.mark.datafiles(DATA_DIR)
diff --git a/tests/loader/variants/elements/simple-unused-variant-invalid.bst b/tests/loader/variants/elements/simple-unused-variant-invalid.bst
new file mode 100644
index 0000000..610bfaf
--- /dev/null
+++ b/tests/loader/variants/elements/simple-unused-variant-invalid.bst
@@ -0,0 +1,14 @@
+kind: pony
+description: An element which depends on an invalid variant of a dependency
+variants:
+- variant: pink
+ config:
+ pony-color: pink
+ depends:
+ - filename: elements/simply-pink.bst
+ variant: green
+- variant: blue
+ config:
+ pony-color: blue
+ depends:
+ - elements/simply-blue.bst
diff --git a/tests/loader/variants/elements/simple-variant-invalid.bst b/tests/loader/variants/elements/simple-variant-invalid.bst
new file mode 100644
index 0000000..d8c2a40
--- /dev/null
+++ b/tests/loader/variants/elements/simple-variant-invalid.bst
@@ -0,0 +1,5 @@
+kind: pony
+description: An element which depends on an invalid variant of a dependency
+depends:
+- filename: elements/simple-variant-compositing.bst
+ variant: green
[buildstream] 13/14: Committing crazy non-functional alternative
with cached code paths.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit f7c6062b1b2fad4ea5cf5800b2c9a0553074a6fd
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sun Jul 16 17:45:41 2017 +0900
Committing crazy non-functional alternative with cached code paths.
o Single configuration in
o Multi configuration out
o Loop over configurations
---
buildstream/_loader.py | 167 ++++++++++++++++++++++++++++++++++---------------
1 file changed, 118 insertions(+), 49 deletions(-)
diff --git a/buildstream/_loader.py b/buildstream/_loader.py
index 0592c8a..db31e21 100644
--- a/buildstream/_loader.py
+++ b/buildstream/_loader.py
@@ -87,15 +87,12 @@ class Variant():
# depend on a given element in a way that conflicts
#
class VariantDisagreement(Exception):
- def __init__(self, element_config, dependency):
+ def __init__(self, element_config1, element_config2):
+ path1 = element_config1.make_path()
+ path2 = element_config2.make_path()
super(VariantDisagreement, self).__init__(
- "Variant disagreement occurred.\n"
- "Element '%s' requested element '%s (%s)'\n"
- "Element '%s' requested element '%s (%s)" %
- (element_config.dependency.owner, element_config.filename,
- element_config.dependency.variant_name,
- dependency.owner, element_config.filename,
- dependency.variant_name))
+ "Variant disagreement occurred.\n {}\n {}"
+ .format(path1, path2))
# VariantInvalid is raised to indicate that a nonexisting
@@ -119,13 +116,25 @@ class VariantInvalid(Exception):
# o The dependencies the element has when configured for the given variant
#
class LoadElementConfig():
- def __init__(self, dependency, element, variant_name=None):
- self.dependency = dependency
+ def __init__(self, parent_config, element, variant_name=None):
+ self.parent_config = parent_config
self.element = element
self.filename = element.filename
self.variant_name = variant_name
self.deps = element.deps_for_variant(variant_name)
+ def __str__(self):
+ if self.variant_name:
+ return "{} ({})".format(self.filename, self.variant_name)
+
+ return self.filename
+
+ def make_path(self):
+ path = str(self)
+ if self.parent_config:
+ path = self.parent_config.make_path() + ' -> ' + path
+ return path
+
# resolve_arch()
#
@@ -412,6 +421,9 @@ class Loader():
self.meta_elements = {} # Dict of resolved meta elements by name
self.elements = {} # Dict of elements
+
+ self.debug_count = 0 # XXX
+
########################################
# Main Entry Point #
########################################
@@ -555,13 +567,22 @@ class Loader():
#
toplevel_config = LoadElementConfig(None, target_element, target_variant)
try:
- pool = self.configure_variants(toplevel_config, [])
+
+ # XXX Start with self.configure_dependency_variants instead !!!
+ #
+ # pool = self.configure_dependency_variants(toplevel_config.deps, {
+ # toplevel_config.filename: toplevel_config
+ # })
+ #
+
+ pools = self.configure_variants(toplevel_config, {})
+
except VariantDisagreement as e:
raise LoadError(LoadErrorReason.VARIANT_DISAGREEMENT, str(e)) from e
# Now apply the chosen variant configurations
#
- for element_config in pool:
+ for _, element_config in pools[0].items():
element_config.element.apply_element_config(element_config)
#
@@ -569,7 +590,7 @@ class Loader():
#
# Args:
# element_config (LoadElementConfig): the element to try
- # pool (list): A list of LoadElementConfig objects
+ # pool (dict): A dict of LoadElementConfig objects
#
# Returns:
# A new configuration
@@ -580,35 +601,40 @@ class Loader():
# the given configuration and the first valid configuration of its
# dependencies
#
- def configure_variants(self, element_config, pool):
+ def configure_variants(self, element_config, pool, depth=0, visited=None, visit_count=None):
- # First, check the new element configuration to try against
- # the existing ones in the pool for conflicts.
- #
- for config in pool:
+ # print("{}TRY: {}".format(' ' * depth, element_config))
+
+ if element_config.filename in pool:
+ config = pool[element_config.filename]
# The configuration pool can have only one selected configuration
# for each element, handle intersections and conflicts.
#
if config.element is element_config.element:
- if config.variant_name == element_config.variant_name:
- # A path converges on the same element configuration,
- # this iteration can be safely discarded.
- return pool
- else:
+
+ if config.variant_name != element_config.variant_name:
+
# Two different variants of the same element should be reached
# on a path of variant agreement.
- raise VariantDisagreement(element_config, config.dependency)
+ raise VariantDisagreement(element_config, config)
+
+ else:
+ # A path converges on the same element configuration,
+ # no need to recurse as we already have a result for this.
+ return [pool]
# Now add ourselves to the pool and recurse into the dependency list
- new_pool = pool + [element_config]
- return self.configure_dependency_variants(element_config.deps, new_pool)
+ new_pool = dict(pool)
+ new_pool[element_config.filename] = element_config
- def configure_dependency_variants(self, deps, pool):
+ return self.configure_dependency_variants(element_config, element_config.deps, new_pool, depth=depth + 1, visited=visited, visit_count=visit_count)
+
+ def configure_dependency_variants(self, parent_config, deps, pool, depth=0, visited=None, visit_count=None):
# This is just the end of the list
if not deps:
- return pool
+ return [pool]
# Loop over the possible variants for this dependency
dependency = deps[0]
@@ -619,51 +645,94 @@ class Loader():
#
element_configs_to_try = []
if dependency.variant_name:
- config = LoadElementConfig(dependency, element, dependency.variant_name)
+ config = LoadElementConfig(parent_config, element, dependency.variant_name)
element_configs_to_try.append(config)
elif len(element.variants) == 0:
- config = LoadElementConfig(dependency, element, None)
+ config = LoadElementConfig(parent_config, element, None)
element_configs_to_try.append(config)
else:
for variant in element.variants:
- config = LoadElementConfig(dependency, element, variant.name)
+ config = LoadElementConfig(parent_config, element, variant.name)
element_configs_to_try.append(config)
# Loop over every possible element configuration for this dependency
#
- accum_pool = None
last_error = None
-
+ valid_pools = []
for element_config in element_configs_to_try:
- # Reset the attempted new pool for each try
- accum_pool = None
+ # XXX DEBUG START
+ if visited is None:
+ visited = {}
+ if visit_count is None:
+ visit_count = {}
+
+ iter_key = (
+ parent_config.filename, parent_config.variant_name,
+ element_config.filename, element_config.variant_name
+ )
+ count = visit_count.get(iter_key, 0)
+ if count > 0:
+ print("Visited path {} times: {}".format(count, element_config.make_path()))
+
+ visit_count[iter_key] = count + 1
+ # XXX DEBUG END
+
+ iter_result = visited.get(iter_key)
+
+ if iter_result is not None:
+ valid_pools += iter_result['pools']
+ if iter_result['error']:
+ last_error = iter_result['error']
+ print("Assigning {} valid pools with error {}".format(len(iter_result['pools']), iter_result['error']))
+ continue
+ else:
+ print("Fresh iteration")
- try:
- # If this configuration of the this element succeeds...
- try_pool = self.configure_variants(element_config, pool)
+ iter_result = {
+ 'pools': [],
+ 'error': None
+ }
- # ... Then recurse into sibling elements
- accum_pool = self.configure_dependency_variants(deps[1:], try_pool)
+ config_pools = []
+ iter_error = None
+ # Recurse into this dependency for this config first
+ try:
+ try_pools = self.configure_variants(element_config, pool, depth=depth, visited=visited, visit_count=visit_count)
except VariantDisagreement as e:
+ last_error = iter_error = e
+
+ # XXX
- # Hold onto the error
- last_error = e
+ iter_result['pools'] = config_pools
+ iter_result['error'] = iter_error
+ visited[iter_key] = iter_result
- # If this element configuration failed, then find more possible
- # element configurations
continue
- # If we've reached here without any disagreement, then we've found the
- # first valid configuration, which should take priority over any others.
- break
+ # For each valid configuration for this element
+ for try_pool in try_pools:
+
+ # Recurse into the siblings, siblings either pass of fail as a whole
+ try:
+ config_pools += self.configure_dependency_variants(parent_config, deps[1:], try_pool,
+ depth=depth + 1, visited=visited, visit_count=visit_count)
+ except VariantDisagreement as e:
+ last_error = iter_error = e
+ continue
+
+ iter_result['pools'] = config_pools
+ iter_result['error'] = iter_error
+ visited[iter_key] = iter_result
+
+ valid_pools += config_pools
# If unable to find any valid configuration, raise a VariantDisagreement
- if not accum_pool:
+ if not valid_pools:
raise last_error
- return accum_pool
+ return valid_pools
########################################
# Element Sorting #
[buildstream] 08/14: _loader.py: Raise LoadError() for invalid
variant requests.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 97e66e23b1d266f9ee1354a05e60247a4578c984
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 18:09:06 2017 +0900
_loader.py: Raise LoadError() for invalid variant requests.
o When the toplevel variant does not exist in the loaded target,
raise the LoadError
o When any element requests a nonexistent variant on any of
it's dependencies, raise the LoadError
o Internal Dependency objects now track the provenance from
whence they were declare, to allow more accurate error reporting
for invalid variants via dependencies.
Beyond this, we also remove the Loader's self.loaded_files attribute
and a bogus assertion that two elements can claim the same name, instead
just rely on a single map of loaded elements to protect against reentrancy.
This was a legacy leftover from an initial implementation which allowed
elements to be named, and as such we were treating loaded files and
loaded elements separately.
---
buildstream/_loader.py | 95 +++++++++++++++++++++++++++++++-------------------
1 file changed, 60 insertions(+), 35 deletions(-)
diff --git a/buildstream/_loader.py b/buildstream/_loader.py
index 3c90e43..17db17f 100644
--- a/buildstream/_loader.py
+++ b/buildstream/_loader.py
@@ -61,12 +61,14 @@ class Symbol():
# A simple dependency object
#
class Dependency():
- def __init__(self, owner_name, name, variant_name=None, filename=None, dep_type=None):
+ def __init__(self, owner_name, name, variant_name=None, filename=None,
+ dep_type=None, provenance=None):
self.owner = owner_name
self.name = name
self.variant_name = variant_name
self.filename = filename
self.dep_type = dep_type
+ self.provenance = provenance
# Holds a variant dictionary and normalized Dependency list
@@ -81,6 +83,34 @@ class Variant():
del self.data[Symbol.VARIANT]
+# VariantDisagreement is raised to indicate that 2 elements
+# depend on a given element in a way that conflicts
+#
+class VariantDisagreement(Exception):
+ def __init__(self, element_config, dependency):
+ super(VariantDisagreement, self).__init__(
+ "Variant disagreement occurred.\n"
+ "Element '%s' requested element '%s (%s)'\n"
+ "Element '%s' requested element '%s (%s)" %
+ (element_config.dependency.owner, element_config.filename,
+ element_config.dependency.variant_name,
+ dependency.owner, element_config.filename,
+ dependency.variant_name))
+
+
+# VariantInvalid is raised to indicate that a nonexisting
+# variant name on an element was requested
+#
+class VariantInvalid(Exception):
+ def __init__(self, dependency, element, variant_name):
+ message = "Variant '{}' is invalid for element {}" \
+ .format(variant_name, element.name)
+ if dependency:
+ message = "{}: {}".format(dependency.provenance, message)
+
+ super(VariantInvalid, self).__init__(message)
+
+
# A utility object wrapping the LoadElement, this represents
# a hypothetical configuration of an element, it describes:
#
@@ -97,21 +127,6 @@ class LoadElementConfig():
self.deps = element.deps_for_variant(variant_name)
-# VariantError is raised to indicate that 2 elements
-# depend on a given element in a way that conflicts
-#
-class VariantError(Exception):
- def __init__(self, element_config, dependency):
- super(VariantError, self).__init__(
- "Variant disagreement occurred.\n"
- "Element '%s' requested element '%s (%s)'\n"
- "Element '%s' requested element '%s (%s)" %
- (element_config.dependency.owner, element_config.filename,
- element_config.dependency.variant_name,
- dependency.owner, element_config.filename,
- dependency.variant_name))
-
-
# resolve_arch()
#
# Composites the data node with the active arch dict and discards
@@ -316,9 +331,10 @@ def extract_depends_from_node(owner, data):
output_deps = []
for dep in depends:
+ dep_provenance = _yaml.node_get_provenance(data, key=Symbol.DEPENDS, indices=[depends.index(dep)])
if isinstance(dep, str):
- dependency = Dependency(owner, dep, filename=dep)
+ dependency = Dependency(owner, dep, filename=dep, provenance=dep_provenance)
elif isinstance(dep, Mapping):
# Make variant optional, for this we set it to None after
@@ -337,7 +353,8 @@ def extract_depends_from_node(owner, data):
(str(provenance), dep_type))
filename = _yaml.node_get(dep, str, Symbol.FILENAME)
- dependency = Dependency(owner, filename, variant_name=variant, filename=filename, dep_type=dep_type)
+ dependency = Dependency(owner, filename, variant_name=variant, filename=filename,
+ dep_type=dep_type, provenance=dep_provenance)
else:
index = depends.index(dep)
@@ -392,7 +409,6 @@ class Loader():
self.host_arch = host_arch
self.target_arch = target_arch
- self.loaded_files = {} # Table of files we've already loaded
self.meta_elements = {} # Dict of resolved meta elements by name
self.elements = {} # Dict of elements
@@ -417,7 +433,14 @@ class Loader():
# First pass, recursively load files and populate our table of LoadElements
#
profile_start(Topics.LOAD_PROJECT, self.target_filename)
- self.load_file(self.target_filename, rewritable, ticker)
+
+ try:
+ target = self.load_file(self.target_filename, rewritable, ticker)
+ if self.target_variant and target.lookup_variant(self.target_variant) is None:
+ raise VariantInvalid(None, target, self.target_variant)
+ except VariantInvalid as e:
+ raise LoadError(LoadErrorReason.INVALID_VARIANT, str(e)) from e
+
profile_end(Topics.LOAD_PROJECT, self.target_filename)
#
@@ -454,16 +477,8 @@ class Loader():
def load_file(self, filename, rewritable, ticker):
# Silently ignore already loaded files
- if filename in self.loaded_files:
- return
- self.loaded_files[filename] = True
-
- # Raise error if two files claim the same name
if filename in self.elements:
- element = self.elements[filename]
- raise LoadError(LoadErrorReason.INVALID_DATA,
- "Tried to load file '%s' but existing file '%s' has the same name" %
- (filename, element.filename))
+ return self.elements[filename]
# Call the ticker
if ticker:
@@ -479,11 +494,21 @@ class Loader():
# Load all possible dependency files for the new LoadElement
for dep in element.base_deps:
- self.load_file(dep.filename, rewritable, ticker)
+ self.load_dependency_file(dep, rewritable, ticker)
for variant in element.variants:
for dep in variant.dependencies:
- self.load_file(dep.filename, rewritable, ticker)
+ self.load_dependency_file(dep, rewritable, ticker)
+
+ return element
+
+ def load_dependency_file(self, dependency, rewritable, ticker):
+
+ element = self.load_file(dependency.filename, rewritable, ticker)
+
+ # Check for invalid variant dependencies
+ if dependency.variant_name and element.lookup_variant(dependency.variant_name) is None:
+ raise VariantInvalid(dependency, element, dependency.variant_name)
########################################
# Resolving Variants #
@@ -531,7 +556,7 @@ class Loader():
toplevel_config = LoadElementConfig(None, target_element, target_variant)
try:
pool = self.configure_variants(toplevel_config, [])
- except VariantError as e:
+ except VariantDisagreement as e:
raise LoadError(LoadErrorReason.VARIANT_DISAGREEMENT, str(e)) from e
# Now apply the chosen variant configurations
@@ -573,7 +598,7 @@ class Loader():
else:
# Two different variants of the same element should be reached
# on a path of variant agreement.
- raise VariantError(element_config, config.dependency)
+ raise VariantDisagreement(element_config, config.dependency)
# Now add ourselves to the pool and recurse into the dependency list
new_pool = pool + [element_config]
@@ -621,7 +646,7 @@ class Loader():
# ... Then recurse into sibling elements
accum_pool = self.configure_dependency_variants(deps[1:], try_pool)
- except VariantError as e:
+ except VariantDisagreement as e:
# Hold onto the error
last_error = e
@@ -630,7 +655,7 @@ class Loader():
# element configurations
continue
- # If unable to find any valid configuration, raise a VariantError
+ # If unable to find any valid configuration, raise a VariantDisagreement
if not accum_pool:
raise last_error
[buildstream] 06/14: main.py: Use context.log_element_format as
default format for `bst show`
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit f876d30a2130b8373f25d07b0df7d74c3badc392
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 16:20:33 2017 +0900
main.py: Use context.log_element_format as default format for `bst show`
---
buildstream/_frontend/main.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/buildstream/_frontend/main.py b/buildstream/_frontend/main.py
index 9c06411..d57b93c 100644
--- a/buildstream/_frontend/main.py
+++ b/buildstream/_frontend/main.py
@@ -218,7 +218,7 @@ def track(app, target, variant, deps, except_):
@click.option('--order', default="stage",
type=click.Choice(['stage', 'alpha']),
help='Staging or alphabetic ordering of dependencies')
-@click.option('--format', '-f', metavar='FORMAT', default="%{state: >12} %{key} %{name}",
+@click.option('--format', '-f', metavar='FORMAT', default=None,
type=click.STRING,
help='Format string for each element')
@click.option('--variant',
@@ -248,6 +248,7 @@ def show(app, target, variant, deps, except_, order, format):
\b
%{name} The element name
+ %{variant} The selected element variant
%{key} The abbreviated cache key (if all sources are consistent)
%{full-key} The full cache key (if all sources are consistent)
%{state} cached, buildable, waiting or inconsistent
@@ -280,6 +281,9 @@ def show(app, target, variant, deps, except_, order, format):
if order == "alpha":
dependencies = sorted(dependencies)
+ if not format:
+ format = app.context.log_element_format
+
report = app.logger.show_pipeline(dependencies, format)
click.echo(report, color=app.colors)
[buildstream] 05/14: userconfig.yaml: Add %{variant} to default
formatting
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1d9fdf00fbf7e31f584f971465ddc5147eedd2fe
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 16:19:41 2017 +0900
userconfig.yaml: Add %{variant} to default formatting
And touch up comment string to mention that it is the default
also used for `bst show`.
---
buildstream/data/userconfig.yaml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/buildstream/data/userconfig.yaml b/buildstream/data/userconfig.yaml
index f3f794b..5525d7d 100644
--- a/buildstream/data/userconfig.yaml
+++ b/buildstream/data/userconfig.yaml
@@ -87,8 +87,8 @@ logging:
# Whether to enable debugging messages
debug: False
- # Format string for printing the pipeline at startup,
- # this uses the same formatting as the `bst show` command.
+ # Format string for printing the pipeline at startup, this
+ # also determines the default display format for `bst show`
element-format: |
- %{state: >12} %{key} %{name}
+ %{state: >12} %{key} %{name} %{variant}
[buildstream] 10/14: _loader.py: Fixed variant priority in variant
resolution
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 498e24db85bfb9c83fd8b184b8228284a0c1eaf8
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 19:21:04 2017 +0900
_loader.py: Fixed variant priority in variant resolution
Previously, the loader was incorrectly settling on the *last*
valid variant instead of the *first* valid configuration.
This is incorrect and violates the rule that the first variant
of an element is chosen when depended on ambivalently, and that
the first variant without disagreement will always be chosen.
---
buildstream/_loader.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/buildstream/_loader.py b/buildstream/_loader.py
index 17db17f..0592c8a 100644
--- a/buildstream/_loader.py
+++ b/buildstream/_loader.py
@@ -655,6 +655,10 @@ class Loader():
# element configurations
continue
+ # If we've reached here without any disagreement, then we've found the
+ # first valid configuration, which should take priority over any others.
+ break
+
# If unable to find any valid configuration, raise a VariantDisagreement
if not accum_pool:
raise last_error
[buildstream] 07/14: exceptions.py: Added new LoadErrorReason
INVALID_VARIANT
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9ce417d1db1934e9ae3de3fffb74a069feceb04a
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 18:05:41 2017 +0900
exceptions.py: Added new LoadErrorReason INVALID_VARIANT
---
buildstream/exceptions.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/buildstream/exceptions.py b/buildstream/exceptions.py
index 039551c..6d19a53 100644
--- a/buildstream/exceptions.py
+++ b/buildstream/exceptions.py
@@ -73,10 +73,14 @@ class LoadErrorReason(Enum):
when loading the project.
"""
- CIRCULAR_DEPENDENCY = 6
+ INVALID_VARIANT = 6
+ """A nonexistent variant was requested.
+ """
+
+ CIRCULAR_DEPENDENCY = 7
"""An circular dependency chain was detected"""
- UNRESOLVED_VARIABLE = 7
+ UNRESOLVED_VARIABLE = 8
"""A variable could not be resolved. This can happen if your project
has cyclic dependencies in variable declarations, or, when substituting
a string which refers to an undefined variable.
[buildstream] 12/14: Added .coveragerc and use that in setup.cfg
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
root pushed a commit to branch variants-slow-functional-loopy
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0e4efd70511c033c58fa88cde74cfbf79857fd33
Author: Tristan Van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jul 15 20:10:44 2017 +0900
Added .coveragerc and use that in setup.cfg
---
.coveragerc | 6 ++++++
setup.cfg | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/.coveragerc b/.coveragerc
new file mode 100644
index 0000000..8e1dd13
--- /dev/null
+++ b/.coveragerc
@@ -0,0 +1,6 @@
+[run]
+omit = buildstream/_profile.py
+branch = True
+
+[report]
+show_missing = True
diff --git a/setup.cfg b/setup.cfg
index 96213b6..8950a0c 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -2,7 +2,7 @@
test=pytest
[tool:pytest]
-addopts = --verbose --basetemp ./tmp --pep8 --cov=buildstream --cov-report term-missing
+addopts = --verbose --basetemp ./tmp --pep8 --cov=buildstream --cov-config .coveragerc
python_files = tests/*/*.py
pep8maxlinelength = 119
pep8ignore =