You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2022/03/30 05:40:30 UTC
[buildstream] branch tristan/remove-plugins created (now 9901b7f)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 9901b7f plugins/sources/git.py: Removing git source
This branch includes the following new commits:
new b386975 _testing/_cachekeys.py: Fix references to buildstream._testing
new 7dc1743 tests/conftest.py: Remove bzr from automated source tests
new d1a0748 tests/sources/bzr.py: Removing bzr specific tests
new 93d63c3 tests/cachekey: Removing bzr from cachekey tests
new 171989e plugins/sources/bzr.py: Removing bzr source
new f9bab3d tests/sources/pip.py: Removing pip source test
new 349a8fe tests/integration/pip_source.py: Removing pip source integration test
new 95810de tests/cachekey: Removing pip source from cachekey test
new 7b70120 plugins/sources/pip.py: Removing pip source
new 76cf305 tests/sources/patch.py: Removing patch source tests
new 7f9756d tests/cachekey: Removing patch source from cache key tests
new 4202056 tests/sourcecache/project: Adding patch source to the project so tests pass
new b23fc02 plugins/sources/patch.py: Removing patch source
new 33bf323 tests/format/project: Use tar source instead of git source for optional refs test
new c7020c3 tests/testutils/site.py: Move the pip_sample_plugins() check to a common site.py
new d178798 tests/plugins/sample-plugins: Adding git source to sample plugins
new 8252a2b tests/frontend/track.py: port tests to use internal git source
new 1248a12 tests/format/include.py: Use tar source instead of git source
new 976d63d tests/frontend/cross_junction_workspace.py: Use tar repo instead of git repo
new 4ecb6eb tests/frontend/logging.py: Use tar repo instead of git repo
new 8bd5a56 tests/frontend/order.py: Use tar repo instead of git repo
new 0ec4499 tests/frontend/workspace.py: Use tar repo instead of git repo
new 16a242e tests/frontend/mirror.py: Use internal git source for these tests
new 349ba87 tests/frontend/default_target.py: Use tar repo instead of git repo
new 10efdb4 tests/frontend/buildcheckout.py: Use tar repo instead of git repo
new a47d0f1 tests/frontend/pull.py: Use tar repo instead of git repo
new 429a40e tests/sourcecache/push.py: Use tar repo instead of git repo
new 13b3865 tests/sourcecache/fetch.py: Use tar repo instead of git repo
new 005310d tests/sources/no_fetch_cached.py: Use tar repo instead of git repo
new 885fad9 tests/elements/filter.py: Use tar repo instead of git repo
new 28add6d tests/testutils/junction.py: Port generate_junction() to use tar repo instead of git repo
new 9d0dce4 tests/testutils/element_generators.py: Port create_element_size() to use tar repo
new bb1bd54 tests/remoteexecution/junction.py: Use tar repo instead of git repo
new fda974b tests/sources/git.py: Remove git source tests
new 165ece8 tests/conftest.py: Remove git from generic source tests
new 6999d0c tests/cachekey: Removing git source from cache key tests
new 9901b7f plugins/sources/git.py: Removing git source
The 37 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 21/37: tests/frontend/order.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 8bd5a56e34ede06f9efe1bebd73d696cdc6b0251
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:37:11 2022 +0900
tests/frontend/order.py: Use tar repo instead of git repo
---
tests/frontend/order.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/frontend/order.py b/tests/frontend/order.py
index bac4976..30dd652 100644
--- a/tests/frontend/order.py
+++ b/tests/frontend/order.py
@@ -27,7 +27,7 @@ DATA_DIR = os.path.join(
def create_element(project, name, dependencies):
dev_files_path = os.path.join(project, "files", "dev-files")
element_path = os.path.join(project, "elements")
- repo = create_repo("git", project, "{}-repo".format(name))
+ repo = create_repo("tar", project, "{}-repo".format(name))
ref = repo.create(dev_files_path)
element = {"kind": "import", "sources": [repo.source_config(ref=ref)], "depends": dependencies}
[buildstream] 17/37: tests/frontend/track.py: port tests to use internal git source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 8252a2bfa6670c4d3f0506b27535921dde7db217
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Tue Mar 29 16:41:35 2022 +0900
tests/frontend/track.py: port tests to use internal git source
In order to remove the `git` plugin, we're porting some of our test cases
which we need to keep to use the private `git` plugin added to the test
sample-plugins package.
---
tests/frontend/project/project.conf | 6 ++++
.../track-cross-junction/subproject/project.conf | 6 ++++
tests/frontend/track-optional-inline/project.conf | 6 ++++
.../track-optional-project-refs/project.conf | 6 ++++
tests/frontend/track.py | 32 ++++++++++++++++++----
5 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/tests/frontend/project/project.conf b/tests/frontend/project/project.conf
index 5ba3168..7e690f5 100644
--- a/tests/frontend/project/project.conf
+++ b/tests/frontend/project/project.conf
@@ -2,3 +2,9 @@
name: test
min-version: 2.0
element-path: elements
+
+plugins:
+- origin: pip
+ package-name: sample-plugins
+ sources:
+ - git
diff --git a/tests/frontend/track-cross-junction/subproject/project.conf b/tests/frontend/track-cross-junction/subproject/project.conf
index 39a53e2..d0163cb 100644
--- a/tests/frontend/track-cross-junction/subproject/project.conf
+++ b/tests/frontend/track-cross-junction/subproject/project.conf
@@ -1,2 +1,8 @@
name: subtest
min-version: 2.0
+
+plugins:
+- origin: pip
+ package-name: sample-plugins
+ sources:
+ - git
diff --git a/tests/frontend/track-optional-inline/project.conf b/tests/frontend/track-optional-inline/project.conf
index 47a2552..5a77b96 100644
--- a/tests/frontend/track-optional-inline/project.conf
+++ b/tests/frontend/track-optional-inline/project.conf
@@ -5,3 +5,9 @@ options:
type: bool
description: Test boolean
default: False
+
+plugins:
+- origin: pip
+ package-name: sample-plugins
+ sources:
+ - git
diff --git a/tests/frontend/track-optional-project-refs/project.conf b/tests/frontend/track-optional-project-refs/project.conf
index 3cb9bdc..26ff42d 100644
--- a/tests/frontend/track-optional-project-refs/project.conf
+++ b/tests/frontend/track-optional-project-refs/project.conf
@@ -8,3 +8,9 @@ options:
type: bool
description: Test boolean
default: False
+
+plugins:
+- origin: pip
+ package-name: sample-plugins
+ sources:
+ - git
diff --git a/tests/frontend/track.py b/tests/frontend/track.py
index c870816..61bee73 100644
--- a/tests/frontend/track.py
+++ b/tests/frontend/track.py
@@ -12,6 +12,9 @@ from buildstream._testing._utils.site import have_subsecond_mtime
from buildstream.exceptions import ErrorDomain, LoadErrorReason
from buildstream import _yaml
from tests.testutils import generate_junction
+from tests.testutils.repo.git import Git
+from tests.testutils.site import pip_sample_packages # pylint: disable=unused-import
+from tests.testutils.site import SAMPLE_PACKAGES_SKIP_REASON
from . import configure_project
# Project directory
@@ -28,6 +31,7 @@ def generate_element(repo, element_path, dep_name=None):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_track_single(cli, tmpdir, datafiles):
project = str(datafiles)
dev_files_path = os.path.join(project, "files", "dev-files")
@@ -38,7 +42,7 @@ def test_track_single(cli, tmpdir, datafiles):
# Create our repo object of the given source type with
# the dev files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = Git(str(tmpdir))
repo.create(dev_files_path)
# Write out our test targets
@@ -70,6 +74,7 @@ def test_track_single(cli, tmpdir, datafiles):
@pytest.mark.datafiles(os.path.join(TOP_DIR))
@pytest.mark.parametrize("ref_storage", [("inline"), ("project-refs")])
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_track_optional(cli, tmpdir, datafiles, ref_storage):
project = os.path.join(datafiles.dirname, datafiles.basename, "track-optional-" + ref_storage)
dev_files_path = os.path.join(project, "files")
@@ -78,7 +83,7 @@ def test_track_optional(cli, tmpdir, datafiles, ref_storage):
# Create our repo object of the given source type with
# the dev files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = Git(str(tmpdir))
repo.create(dev_files_path)
# Now create an optional test branch and add a commit to that,
@@ -166,6 +171,7 @@ def test_track_deps(cli, datafiles, deps, expected_states):
@pytest.mark.datafiles(os.path.join(TOP_DIR, "track-cross-junction"))
@pytest.mark.parametrize("cross_junction", [("cross"), ("nocross")])
@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_track_cross_junction(cli, tmpdir, datafiles, cross_junction, ref_storage):
project = str(datafiles)
dev_files_path = os.path.join(project, "files")
@@ -175,7 +181,7 @@ def test_track_cross_junction(cli, tmpdir, datafiles, cross_junction, ref_storag
# Create our repo object of the given source type with
# the dev files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = Git(str(tmpdir))
repo.create(dev_files_path)
# Generate two elements using the git source, one in
@@ -185,7 +191,19 @@ def test_track_cross_junction(cli, tmpdir, datafiles, cross_junction, ref_storag
# Generate project.conf
#
- project_conf = {"name": "test", "min-version": "2.0", "ref-storage": ref_storage}
+ project_conf = {
+ "name": "test",
+ "min-version": "2.0",
+ "ref-storage": ref_storage,
+ "plugins": [
+ {
+ "origin": "pip",
+ "package-name": "sample-plugins",
+ "sources": ["git"],
+ }
+ ],
+ }
+
_yaml.roundtrip_dump(project_conf, os.path.join(project, "project.conf"))
#
@@ -333,6 +351,7 @@ def test_junction_element(cli, tmpdir, datafiles, ref_storage):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_track_error_cannot_write_file(cli, tmpdir, datafiles):
if os.geteuid() == 0:
pytest.skip("This is not testable with root permissions")
@@ -344,7 +363,7 @@ def test_track_error_cannot_write_file(cli, tmpdir, datafiles):
configure_project(project, {"ref-storage": "inline"})
- repo = create_repo("git", str(tmpdir))
+ repo = Git(str(tmpdir))
repo.create(dev_files_path)
element_full_path = os.path.join(element_path, element_name)
@@ -363,6 +382,7 @@ def test_track_error_cannot_write_file(cli, tmpdir, datafiles):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_no_needless_overwrite(cli, tmpdir, datafiles):
project = os.path.join(datafiles.dirname, datafiles.basename)
dev_files_path = os.path.join(project, "files", "dev-files")
@@ -377,7 +397,7 @@ def test_no_needless_overwrite(cli, tmpdir, datafiles):
# Create our repo object of the given source type with
# the dev files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = Git(str(tmpdir))
repo.create(dev_files_path)
# Write out our test target and assert it exists
[buildstream] 30/37: tests/elements/filter.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 885fad9f612c68b44dfc6bb12f6d88dd1a420790
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:31:43 2022 +0900
tests/elements/filter.py: Use tar repo instead of git repo
---
tests/elements/filter.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/tests/elements/filter.py b/tests/elements/filter.py
index 682fb42..8db488a 100644
--- a/tests/elements/filter.py
+++ b/tests/elements/filter.py
@@ -192,7 +192,7 @@ def test_filter_workspace_reset(datafiles, cli, tmpdir):
@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
def test_filter_track(datafiles, cli, tmpdir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(str(datafiles), "files"))
elements_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
@@ -238,7 +238,7 @@ def test_filter_track(datafiles, cli, tmpdir):
@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
def test_filter_track_excepted(datafiles, cli, tmpdir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
repo.create(os.path.join(str(datafiles), "files"))
elements_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
@@ -283,7 +283,7 @@ def test_filter_track_excepted(datafiles, cli, tmpdir):
@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
def test_filter_track_multi_to_one(datafiles, cli, tmpdir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(str(datafiles), "files"))
elements_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
@@ -329,7 +329,7 @@ def test_filter_track_multi_to_one(datafiles, cli, tmpdir):
@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
def test_filter_track_multi(datafiles, cli, tmpdir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(str(datafiles), "files"))
elements_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
@@ -390,7 +390,7 @@ def test_filter_track_multi(datafiles, cli, tmpdir):
@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
def test_filter_track_multi_exclude(datafiles, cli, tmpdir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(str(datafiles), "files"))
elements_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
[buildstream] 27/37: tests/sourcecache/push.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 429a40e01bbaaa5a6c9ac3c7bd8e4ee576446b85
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:15:18 2022 +0900
tests/sourcecache/push.py: Use tar repo instead of git repo
---
tests/sourcecache/push.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/tests/sourcecache/push.py b/tests/sourcecache/push.py
index 3c8b624..0931f68 100644
--- a/tests/sourcecache/push.py
+++ b/tests/sourcecache/push.py
@@ -72,7 +72,7 @@ def test_source_push_split(cli, tmpdir, datafiles):
_yaml.roundtrip_dump(user_config, file=user_config_file)
cli.configure(user_config)
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
element_name = "push.bst"
@@ -129,7 +129,7 @@ def test_source_push(cli, tmpdir, datafiles):
_yaml.roundtrip_dump(user_config, file=user_config_file)
cli.configure(user_config)
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
element_name = "push.bst"
@@ -187,7 +187,7 @@ def test_push_pull(cli, datafiles, tmpdir):
cli.configure(user_config)
# create repo to pull from
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
element_name = "push.bst"
@@ -232,7 +232,7 @@ def test_push_fail(cli, tmpdir, datafiles):
cli.configure(user_config)
# create repo to pull from
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
element_name = "push.bst"
@@ -268,7 +268,7 @@ def test_source_push_build_fail(cli, tmpdir, datafiles):
}
cli.configure(user_config)
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
[buildstream] 31/37: tests/testutils/junction.py: Port generate_junction() to use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 28add6da553be70900e01f2e582124d2cbd87dbe
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:58:45 2022 +0900
tests/testutils/junction.py: Port generate_junction() to use tar repo instead of git repo
---
tests/format/include.py | 6 ++++--
tests/testutils/junction.py | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/tests/format/include.py b/tests/format/include.py
index 3443cec..42f85c1 100644
--- a/tests/format/include.py
+++ b/tests/format/include.py
@@ -317,9 +317,11 @@ def test_option_from_junction_element(cli, tmpdir, datafiles):
@pytest.mark.datafiles(DATA_DIR)
def test_option_from_deep_junction(cli, tmpdir, datafiles):
project = os.path.join(str(datafiles), "junction_options_deep")
+ junction_repo_a = os.path.join(tmpdir, "a")
+ junction_repo_b = os.path.join(tmpdir, "b")
generate_junction(
- tmpdir,
+ junction_repo_a,
os.path.join(project, "subproject-2"),
os.path.join(project, "subproject-1", "junction-2.bst"),
store_ref=True,
@@ -327,7 +329,7 @@ def test_option_from_deep_junction(cli, tmpdir, datafiles):
)
generate_junction(
- tmpdir,
+ junction_repo_b,
os.path.join(project, "subproject-1"),
os.path.join(project, "junction-1.bst"),
store_ref=True,
diff --git a/tests/testutils/junction.py b/tests/testutils/junction.py
index 0a94846..4fc7de5 100644
--- a/tests/testutils/junction.py
+++ b/tests/testutils/junction.py
@@ -19,7 +19,7 @@ def generate_junction(tmpdir, subproject_path, junction_path, *, store_ref=True,
# Create a repo to hold the subproject and generate
# a junction element for it
#
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
source_ref = ref = repo.create(subproject_path)
if not store_ref:
source_ref = None
[buildstream] 29/37: tests/sources/no_fetch_cached.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 005310da397994b9d601430df4dccf2672f51675
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:29:43 2022 +0900
tests/sources/no_fetch_cached.py: Use tar repo instead of git repo
---
tests/sources/no_fetch_cached.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/tests/sources/no_fetch_cached.py b/tests/sources/no_fetch_cached.py
index 0092d33..7819b90 100644
--- a/tests/sources/no_fetch_cached.py
+++ b/tests/sources/no_fetch_cached.py
@@ -16,13 +16,12 @@ DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "no-fetch-c
# Tests #
##################################################################
# Test that fetch() is not called for cached sources
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
@pytest.mark.datafiles(DATA_DIR)
def test_no_fetch_cached(cli, tmpdir, datafiles):
project = str(datafiles)
# Create the repo from 'files' subdir
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project, "files"))
# Write out test target with a cached and a non-cached source
[buildstream] 05/37: plugins/sources/bzr.py: Removing bzr source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 171989e452c72fc24e1d4a5b8b84c7e7087f94e1
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:13:26 2022 +0900
plugins/sources/bzr.py: Removing bzr source
And removing bzr from the docs plugins index.
---
doc/source/core_plugins.rst | 1 -
src/buildstream/plugins/sources/bzr.py | 215 ---------------------------------
2 files changed, 216 deletions(-)
diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst
index 1a12506..b6ecc87 100644
--- a/doc/source/core_plugins.rst
+++ b/doc/source/core_plugins.rst
@@ -52,7 +52,6 @@ information.
sources/remote
sources/tar
sources/git
- sources/bzr
sources/patch
sources/pip
diff --git a/src/buildstream/plugins/sources/bzr.py b/src/buildstream/plugins/sources/bzr.py
deleted file mode 100644
index 7bf5b79..0000000
--- a/src/buildstream/plugins/sources/bzr.py
+++ /dev/null
@@ -1,215 +0,0 @@
-# Copyright (C) 2017 Codethink Limited
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors:
-# Jonathan Maw <jo...@codethink.co.uk>
-
-"""
-bzr - stage files from a bazaar repository
-==========================================
-
-**Host dependencies:**
-
- * bzr
-
-**Usage:**
-
-.. code:: yaml
-
- # Specify the bzr source kind
- kind: bzr
-
- # Specify the bzr url. Bazaar URLs come in many forms, see
- # `bzr help urlspec` for more information. Using an alias defined
- # in your project configuration is encouraged.
- url: https://launchpad.net/bzr
-
- # Specify the tracking branch. This is mandatory, as bzr cannot identify
- # an individual revision outside its branch. bzr URLs that omit the branch
- # name implicitly specify the trunk branch, but bst requires this to be
- # explicit.
- track: trunk
-
- # Specify the ref. This is a revision number. This is usually a decimal,
- # but revisions on a branch are of the form
- # <revision-branched-from>.<branch-number>.<revision-since-branching>
- # e.g. 6622.1.6.
- # The ref must be specified to build, and 'bst source track' will update the
- # revision number to the one on the tip of the branch specified in 'track'.
- ref: 6622
-
-See :ref:`built-in functionality doumentation <core_source_builtins>` for
-details on common configuration options for sources.
-"""
-
-import os
-import shutil
-import fcntl
-from contextlib import contextmanager
-
-from buildstream import Source, SourceError
-from buildstream import utils
-
-
-class BzrSource(Source):
- # pylint: disable=attribute-defined-outside-init
-
- BST_MIN_VERSION = "2.0"
-
- def configure(self, node):
- node.validate_keys(["url", "track", "ref", *Source.COMMON_CONFIG_KEYS])
-
- self.original_url = node.get_str("url")
- self.tracking = node.get_str("track")
- self.ref = node.get_str("ref", None)
- self.url = self.translate_url(self.original_url)
-
- def preflight(self):
- # Check if bzr is installed, get the binary at the same time.
- self.host_bzr = utils.get_host_tool("bzr")
-
- def get_unique_key(self):
- return [self.original_url, self.tracking, self.ref]
-
- def is_cached(self):
- with self._locked():
- return self._check_ref()
-
- def load_ref(self, node):
- self.ref = node.get_str("ref", None)
-
- def get_ref(self):
- return self.ref
-
- def set_ref(self, ref, node):
- node["ref"] = self.ref = ref
-
- def track(self): # pylint: disable=arguments-differ
- with self.timed_activity("Tracking {}".format(self.url), silent_nested=True), self._locked():
- self._ensure_mirror(skip_ref_check=True)
- ret, out = self.check_output(
- [self.host_bzr, "version-info", "--custom", "--template={revno}", self._get_branch_dir()],
- fail="Failed to read the revision number at '{}'".format(self._get_branch_dir()),
- )
- if ret != 0:
- raise SourceError("{}: Failed to get ref for tracking {}".format(self, self.tracking))
-
- return out
-
- def fetch(self): # pylint: disable=arguments-differ
- with self.timed_activity("Fetching {}".format(self.url), silent_nested=True), self._locked():
- self._ensure_mirror()
-
- def stage(self, directory):
- self.call(
- [
- self.host_bzr,
- "checkout",
- "--lightweight",
- "--revision=revno:{}".format(self.ref),
- self._get_branch_dir(),
- directory,
- ],
- fail="Failed to checkout revision {} from branch {} to {}".format(
- self.ref, self._get_branch_dir(), directory
- ),
- )
- # Remove .bzr dir
- shutil.rmtree(os.path.join(directory, ".bzr"))
-
- def init_workspace(self, directory):
- url = os.path.join(self.url, self.tracking)
- with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
- # Checkout from the cache
- self.call(
- [
- self.host_bzr,
- "branch",
- "--use-existing-dir",
- "--revision=revno:{}".format(self.ref),
- self._get_branch_dir(),
- directory,
- ],
- fail="Failed to branch revision {} from branch {} to {}".format(
- self.ref, self._get_branch_dir(), directory
- ),
- )
- # Switch the parent branch to the source's origin
- self.call(
- [self.host_bzr, "switch", "--directory={}".format(directory), url],
- fail="Failed to switch workspace's parent branch to {}".format(url),
- )
-
- # _locked()
- #
- # This context manager ensures exclusive access to the
- # bzr repository.
- #
- @contextmanager
- def _locked(self):
- lockdir = os.path.join(self.get_mirror_directory(), "locks")
- lockfile = os.path.join(lockdir, utils.url_directory_name(self.original_url) + ".lock")
- os.makedirs(lockdir, exist_ok=True)
- with open(lockfile, "w") as lock:
- fcntl.flock(lock, fcntl.LOCK_EX)
- try:
- yield
- finally:
- fcntl.flock(lock, fcntl.LOCK_UN)
-
- def _check_ref(self):
- # If the mirror doesnt exist yet, then we dont have the ref
- if not os.path.exists(self._get_branch_dir()):
- return False
-
- return self.call([self.host_bzr, "revno", "--revision=revno:{}".format(self.ref), self._get_branch_dir()]) == 0
-
- def _get_branch_dir(self):
- return os.path.join(self._get_mirror_dir(), self.tracking)
-
- def _get_mirror_dir(self):
- return os.path.join(self.get_mirror_directory(), utils.url_directory_name(self.original_url))
-
- def _ensure_mirror(self, skip_ref_check=False):
- mirror_dir = self._get_mirror_dir()
- bzr_metadata_dir = os.path.join(mirror_dir, ".bzr")
- if not os.path.exists(bzr_metadata_dir):
- self.call(
- [self.host_bzr, "init-repo", "--no-trees", mirror_dir], fail="Failed to initialize bzr repository"
- )
-
- branch_dir = os.path.join(mirror_dir, self.tracking)
- branch_url = self.url + "/" + self.tracking
- if not os.path.exists(branch_dir):
- # `bzr branch` the branch if it doesn't exist
- # to get the upstream code
- self.call(
- [self.host_bzr, "branch", branch_url, branch_dir],
- fail="Failed to branch from {} to {}".format(branch_url, branch_dir),
- )
-
- else:
- # `bzr pull` the branch if it does exist
- # to get any changes to the upstream code
- self.call(
- [self.host_bzr, "pull", "--directory={}".format(branch_dir), branch_url],
- fail="Failed to pull new changes for {}".format(branch_dir),
- )
-
- if not skip_ref_check and not self._check_ref():
- raise SourceError("Failed to ensure ref '{}' was mirrored".format(self.ref), reason="ref-not-mirrored")
-
-
-def setup():
- return BzrSource
[buildstream] 13/37: plugins/sources/patch.py: Removing patch source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit b23fc028afed085a029add9d0b52c97165e69e65
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 17:03:46 2022 +0900
plugins/sources/patch.py: Removing patch source
And removing patch source from documentation plugins index.
---
doc/source/core_plugins.rst | 1 -
src/buildstream/plugins/sources/patch.py | 106 -------------------------------
2 files changed, 107 deletions(-)
diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst
index f7eabde..15cd701 100644
--- a/doc/source/core_plugins.rst
+++ b/doc/source/core_plugins.rst
@@ -52,7 +52,6 @@ information.
sources/remote
sources/tar
sources/git
- sources/patch
.. _plugins_external:
diff --git a/src/buildstream/plugins/sources/patch.py b/src/buildstream/plugins/sources/patch.py
deleted file mode 100644
index 6392db8..0000000
--- a/src/buildstream/plugins/sources/patch.py
+++ /dev/null
@@ -1,106 +0,0 @@
-#
-# Copyright Bloomberg Finance LP
-# Copyright (C) 2018 Codethink Limited
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors:
-# Chandan Singh <cs...@bloomberg.net>
-# Tiago Gomes <ti...@codethink.co.uk>
-
-"""
-patch - apply locally stored patches
-====================================
-
-**Host dependencies:**
-
- * patch
-
-**Usage:**
-
-.. code:: yaml
-
- # Specify the local source kind
- kind: patch
-
- # Specify the project relative path to a patch file
- path: files/somefile.diff
-
- # Optionally specify the strip level, defaults to 1
- strip-level: 1
-
-See :ref:`built-in functionality doumentation <core_source_builtins>` for
-details on common configuration options for sources.
-"""
-
-import os
-from buildstream import Source, SourceError
-from buildstream import utils
-
-
-class PatchSource(Source):
- # pylint: disable=attribute-defined-outside-init
-
- BST_MIN_VERSION = "2.0"
-
- BST_REQUIRES_PREVIOUS_SOURCES_STAGE = True
-
- def configure(self, node):
- node.validate_keys(["path", "strip-level", *Source.COMMON_CONFIG_KEYS])
- self.path = self.node_get_project_path(node.get_scalar("path"), check_is_file=True)
- self.strip_level = node.get_int("strip-level", default=1)
- self.fullpath = os.path.join(self.get_project_directory(), self.path)
-
- def preflight(self):
- # Check if patch is installed, get the binary at the same time
- self.host_patch = utils.get_host_tool("patch")
-
- def get_unique_key(self):
- return [self.path, utils.sha256sum(self.fullpath), self.strip_level]
-
- def is_resolved(self):
- return True
-
- def is_cached(self):
- return True
-
- def load_ref(self, node):
- pass
-
- def get_ref(self):
- return None # pragma: nocover
-
- def set_ref(self, ref, node):
- pass # pragma: nocover
-
- def fetch(self): # pylint: disable=arguments-differ
- # Nothing to do here for a local source
- pass # pragma: nocover
-
- def stage(self, directory):
- with self.timed_activity("Applying local patch: {}".format(self.path)):
-
- # Bail out with a comprehensive message if the target directory is empty
- if not os.listdir(directory):
- raise SourceError("Nothing to patch in directory '{}'".format(directory), reason="patch-no-files")
-
- strip_level_option = "-p{}".format(self.strip_level)
- self.call(
- [self.host_patch, strip_level_option, "-i", self.fullpath, "-d", directory],
- fail="Failed to apply patch {}".format(self.path),
- )
-
-
-# Plugin entry point
-def setup():
- return PatchSource
[buildstream] 33/37: tests/remoteexecution/junction.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit bb1bd5493111567be04ed7fd3cd27db7b7094e07
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:35:09 2022 +0900
tests/remoteexecution/junction.py: Use tar repo instead of git repo
---
tests/remoteexecution/junction.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/remoteexecution/junction.py b/tests/remoteexecution/junction.py
index 2d51240..943ea7f 100644
--- a/tests/remoteexecution/junction.py
+++ b/tests/remoteexecution/junction.py
@@ -54,7 +54,7 @@ def test_junction_build_remote(cli, tmpdir, datafiles):
junction_path = os.path.join(element_path, "junction.bst")
# We need a repo for real trackable elements
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(amhello_files_path)
# ensure that the correct project directory is also listed in the junction
[buildstream] 12/37: tests/sourcecache/project: Adding patch source to the project so tests pass
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 42020561e708fe3363241da3f9e2b632af67bda8
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 17:47:33 2022 +0900
tests/sourcecache/project: Adding patch source to the project so tests pass
---
tests/sourcecache/project/plugins/sources/patch.py | 118 +++++++++++++++++++++
tests/sourcecache/project/project.conf | 4 +
2 files changed, 122 insertions(+)
diff --git a/tests/sourcecache/project/plugins/sources/patch.py b/tests/sourcecache/project/plugins/sources/patch.py
new file mode 100644
index 0000000..e664005
--- /dev/null
+++ b/tests/sourcecache/project/plugins/sources/patch.py
@@ -0,0 +1,118 @@
+#
+# Copyright Bloomberg Finance LP
+# Copyright (C) 2018 Codethink Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# Chandan Singh <cs...@bloomberg.net>
+# Tiago Gomes <ti...@codethink.co.uk>
+
+"""
+patch - apply locally stored patches
+====================================
+
+**Host dependencies:**
+
+ * patch
+
+**Usage:**
+
+.. code:: yaml
+
+ # Specify the local source kind
+ kind: patch
+
+ # Specify the project relative path to a patch file
+ path: files/somefile.diff
+
+ # Optionally specify the strip level, defaults to 1
+ strip-level: 1
+
+
+See `built-in functionality doumentation
+<https://docs.buildstream.build/master/buildstream.source.html#core-source-builtins>`_ for
+details on common configuration options for sources.
+"""
+
+import os
+from buildstream import Source, SourceError
+from buildstream import utils
+
+
+class PatchSource(Source):
+ # pylint: disable=attribute-defined-outside-init
+
+ BST_MIN_VERSION = "2.0"
+
+ BST_REQUIRES_PREVIOUS_SOURCES_STAGE = True
+
+ def configure(self, node):
+ node.validate_keys(["path", "strip-level", *Source.COMMON_CONFIG_KEYS])
+ self.path = self.node_get_project_path(node.get_scalar("path"), check_is_file=True)
+ self.strip_level = node.get_int("strip-level", default=1)
+ self.fullpath = os.path.join(self.get_project_directory(), self.path)
+
+ def preflight(self):
+ # Check if patch is installed, get the binary at the same time
+ self.host_patch = utils.get_host_tool("patch")
+
+ def get_unique_key(self):
+ return [self.path, utils.sha256sum(self.fullpath), self.strip_level]
+
+ def is_resolved(self):
+ return True
+
+ def is_cached(self):
+ return True
+
+ def load_ref(self, node):
+ pass
+
+ def get_ref(self):
+ return None # pragma: nocover
+
+ def set_ref(self, ref, node):
+ pass # pragma: nocover
+
+ def fetch(self): # pylint: disable=arguments-differ
+ # Nothing to do here for a local source
+ pass # pragma: nocover
+
+ def stage(self, directory):
+ with self.timed_activity("Applying local patch: {}".format(self.path)):
+
+ # Bail out with a comprehensive message if the target directory is empty
+ if not os.listdir(directory):
+ raise SourceError(
+ "Nothing to patch in directory '{}'".format(directory),
+ reason="patch-no-files",
+ )
+
+ strip_level_option = "-p{}".format(self.strip_level)
+ self.call(
+ [
+ self.host_patch,
+ strip_level_option,
+ "-i",
+ self.fullpath,
+ "-d",
+ directory,
+ ],
+ fail="Failed to apply patch {}".format(self.path),
+ )
+
+
+# Plugin entry point
+def setup():
+ return PatchSource
diff --git a/tests/sourcecache/project/project.conf b/tests/sourcecache/project/project.conf
index e3635e8..681fa1b 100644
--- a/tests/sourcecache/project/project.conf
+++ b/tests/sourcecache/project/project.conf
@@ -10,3 +10,7 @@ plugins:
path: plugins/elements
elements:
- always_fail
+- origin: local
+ path: plugins/sources
+ sources:
+ - patch
[buildstream] 06/37: tests/sources/pip.py: Removing pip source test
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit f9bab3d23a655f9859f465dc5b5c0fda876a80cd
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:32:11 2022 +0900
tests/sources/pip.py: Removing pip source test
---
tests/sources/pip.py | 61 ---------------------------
tests/sources/pip/first-source-pip/target.bst | 6 ---
tests/sources/pip/no-packages/file | 1 -
tests/sources/pip/no-packages/target.bst | 6 ---
tests/sources/pip/no-ref/file | 1 -
tests/sources/pip/no-ref/target.bst | 8 ----
6 files changed, 83 deletions(-)
diff --git a/tests/sources/pip.py b/tests/sources/pip.py
deleted file mode 100644
index 08483e4..0000000
--- a/tests/sources/pip.py
+++ /dev/null
@@ -1,61 +0,0 @@
-# Pylint doesn't play well with fixtures and dependency injection from pytest
-# pylint: disable=redefined-outer-name
-
-import os
-import pytest
-
-from buildstream.exceptions import ErrorDomain
-from buildstream.plugins.sources.pip import _match_package_name
-from buildstream._testing import cli, generate_project # pylint: disable=unused-import
-
-DATA_DIR = os.path.join(
- os.path.dirname(os.path.realpath(__file__)),
- "pip",
-)
-
-
-# Test that without ref, consistency is set appropriately.
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "no-ref"))
-def test_no_ref(cli, datafiles):
- project = str(datafiles)
- generate_project(project)
- assert cli.get_element_state(project, "target.bst") == "no reference"
-
-
-# Test that pip is not allowed to be the first source
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "first-source-pip"))
-def test_first_source(cli, datafiles):
- project = str(datafiles)
- generate_project(project)
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_main_error(ErrorDomain.ELEMENT, None)
-
-
-# Test that error is raised when neither packges nor requirements files
-# have been specified
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "no-packages"))
-def test_no_packages(cli, datafiles):
- project = str(datafiles)
- generate_project(project)
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_main_error(ErrorDomain.SOURCE, None)
-
-
-# Test that pip source parses tar ball names correctly for the ref
-@pytest.mark.parametrize(
- "tarball, expected_name, expected_version",
- [
- ("dotted.package-0.9.8.tar.gz", "dotted.package", "0.9.8"),
- ("hyphenated-package-2.6.0.tar.gz", "hyphenated-package", "2.6.0"),
- ("underscore_pkg-3.1.0.tar.gz", "underscore_pkg", "3.1.0"),
- ("numbers2and5-1.0.1.tar.gz", "numbers2and5", "1.0.1"),
- ("multiple.dots.package-5.6.7.tar.gz", "multiple.dots.package", "5.6.7"),
- ("multiple-hyphens-package-1.2.3.tar.gz", "multiple-hyphens-package", "1.2.3"),
- ("multiple_underscore_pkg-3.4.5.tar.gz", "multiple_underscore_pkg", "3.4.5"),
- ("shortversion-1.0.tar.gz", "shortversion", "1.0"),
- ("longversion-1.2.3.4.tar.gz", "longversion", "1.2.3.4"),
- ],
-)
-def test_match_package_name(tarball, expected_name, expected_version):
- name, version = _match_package_name(tarball)
- assert (expected_name, expected_version) == (name, version)
diff --git a/tests/sources/pip/first-source-pip/target.bst b/tests/sources/pip/first-source-pip/target.bst
deleted file mode 100644
index e5f20ab..0000000
--- a/tests/sources/pip/first-source-pip/target.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-description: pip should not be allowed to be the first source
-sources:
-- kind: pip
- packages:
- - flake8
diff --git a/tests/sources/pip/no-packages/file b/tests/sources/pip/no-packages/file
deleted file mode 100644
index 980a0d5..0000000
--- a/tests/sources/pip/no-packages/file
+++ /dev/null
@@ -1 +0,0 @@
-Hello World!
diff --git a/tests/sources/pip/no-packages/target.bst b/tests/sources/pip/no-packages/target.bst
deleted file mode 100644
index 0d8b948..0000000
--- a/tests/sources/pip/no-packages/target.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-description: The kind of this element is irrelevant.
-sources:
-- kind: local
- path: file
-- kind: pip
diff --git a/tests/sources/pip/no-ref/file b/tests/sources/pip/no-ref/file
deleted file mode 100644
index 980a0d5..0000000
--- a/tests/sources/pip/no-ref/file
+++ /dev/null
@@ -1 +0,0 @@
-Hello World!
diff --git a/tests/sources/pip/no-ref/target.bst b/tests/sources/pip/no-ref/target.bst
deleted file mode 100644
index ec450b7..0000000
--- a/tests/sources/pip/no-ref/target.bst
+++ /dev/null
@@ -1,8 +0,0 @@
-kind: import
-description: The kind of this element is irrelevant.
-sources:
-- kind: local
- path: file
-- kind: pip
- packages:
- - flake8
[buildstream] 18/37: tests/format/include.py: Use tar source instead of git source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1248a122c6026753f7ec8de50b87890948baa13e
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Tue Mar 29 19:14:51 2022 +0900
tests/format/include.py: Use tar source instead of git source
This test doesn't require git for testing, use tar since tar is not
going to be removed.
---
tests/format/include.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/format/include.py b/tests/format/include.py
index afa0d22..3443cec 100644
--- a/tests/format/include.py
+++ b/tests/format/include.py
@@ -109,7 +109,7 @@ def test_junction_element_partial_project_project(cli, tmpdir, datafiles):
subproject_path = os.path.join(project, "subproject")
junction_path = os.path.join(project, "junction.bst")
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(subproject_path)
@@ -133,7 +133,7 @@ def test_junction_element_not_partial_project_file(cli, tmpdir, datafiles):
subproject_path = os.path.join(project, "subproject")
junction_path = os.path.join(project, "junction.bst")
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(subproject_path)
[buildstream] 20/37: tests/frontend/logging.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 4ecb6ebdf6bb0444e4f4aa203732767f78be5786
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:32:00 2022 +0900
tests/frontend/logging.py: Use tar repo instead of git repo
---
tests/frontend/logging.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/frontend/logging.py b/tests/frontend/logging.py
index fd49a54..315d20b 100644
--- a/tests/frontend/logging.py
+++ b/tests/frontend/logging.py
@@ -26,7 +26,7 @@ def test_default_logging(cli, tmpdir, datafiles):
# Create our repo object of the given source type with
# the bin files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(bin_files_path)
# Write out our test target
@@ -55,7 +55,7 @@ def test_custom_logging(cli, tmpdir, datafiles):
# Create our repo object of the given source type with
# the bin files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(bin_files_path)
# Write out our test target
[buildstream] 37/37: plugins/sources/git.py: Removing git source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9901b7fd6afb8c20148888a80d03425fb7148c3b
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:39:38 2022 +0900
plugins/sources/git.py: Removing git source
And removing the git source from the documentation plugins index
---
doc/source/core_plugins.rst | 1 -
src/buildstream/_gitsourcebase.py | 831 ---------------------------------
src/buildstream/plugins/sources/git.py | 164 -------
3 files changed, 996 deletions(-)
diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst
index 15cd701..79e4d56 100644
--- a/doc/source/core_plugins.rst
+++ b/doc/source/core_plugins.rst
@@ -51,7 +51,6 @@ information.
sources/local
sources/remote
sources/tar
- sources/git
.. _plugins_external:
diff --git a/src/buildstream/_gitsourcebase.py b/src/buildstream/_gitsourcebase.py
deleted file mode 100644
index 901069b..0000000
--- a/src/buildstream/_gitsourcebase.py
+++ /dev/null
@@ -1,831 +0,0 @@
-#
-# Copyright (C) 2016 Codethink Limited
-# Copyright (C) 2018 Bloomberg Finance LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors:
-# Tristan Van Berkom <tr...@codethink.co.uk>
-# Chandan Singh <cs...@bloomberg.net>
-# Tom Mewett <to...@codethink.co.uk>
-
-"""Abstract base class for source implementations that work with a Git repository"""
-
-import os
-import re
-import shutil
-from io import StringIO
-from tempfile import TemporaryFile
-
-from configparser import RawConfigParser
-
-from .source import Source, SourceError, SourceFetcher
-from .types import CoreWarnings
-from . import utils
-from .types import FastEnum
-from .utils import move_atomic, DirectoryExistsError
-
-GIT_MODULES = ".gitmodules"
-EXACT_TAG_PATTERN = r"(?P<tag>.*)-0-g(?P<commit>.*)"
-
-# Warnings
-WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
-WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
-WARN_INVALID_SUBMODULE = "invalid-submodule"
-
-
-class _RefFormat(FastEnum):
- SHA1 = "sha1"
- GIT_DESCRIBE = "git-describe"
-
-
-def _strip_tag(rev):
- return rev.split("-g")[-1]
-
-
-# This class represents a single Git repository. The Git source needs to account for
-# submodules, but we don't want to cache them all under the umbrella of the
-# superproject - so we use this class which caches them independently, according
-# to their URL. Instances keep reference to their "parent" GitSourceBase,
-# and if applicable, where in the superproject they are found.
-#
-# Args:
-# source (_GitSourceBase or subclass): The parent source
-# path (str): The relative location of the submodule in the superproject;
-# the empty string for the superproject itself
-# url (str): Where to clone the repo from
-# ref (str): Specified 'ref' from the source configuration
-# primary (bool): Whether this is the primary URL for the source
-# tags (list): Tag configuration; see _GitSourceBase._load_tags
-#
-class _GitMirror(SourceFetcher):
- def __init__(self, source, path, url, ref, *, primary=False, tags=None):
-
- super().__init__()
- self.source = source
- self.path = path
- self.url = url
- self.ref = ref
- self.tags = tags or []
- self.primary = primary
- self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
-
- # _ensure_repo():
- #
- # Ensures that the Git repository exists at the mirror location and is configured
- # to fetch from the given URL
- #
- def _ensure_repo(self):
- if not os.path.exists(self.mirror):
- with self.source.tempdir() as tmpdir:
- self.source.call(
- [self.source.host_git, "init", "--bare", tmpdir],
- fail="Failed to initialise repository",
- )
-
- try:
- move_atomic(tmpdir, self.mirror)
- except DirectoryExistsError:
- # Another process was quicker to download this repository.
- # Let's discard our own
- self.source.status("{}: Discarding duplicate repository".format(self.source))
- except OSError as e:
- raise SourceError(
- "{}: Failed to move created repository from '{}' to '{}': {}".format(
- self.source, tmpdir, self.mirror, e
- )
- ) from e
-
- def _fetch(self, url, fetch_all=False):
- self._ensure_repo()
-
- # Work out whether we can fetch a specific tag: are we given a ref which
- # 1. is in git-describe format
- # 2. refers to an exact tag (is "...-0-g...")
- # 3. is available on the remote and tags the specified commit?
- # And lastly: are we on a new-enough Git which allows cloning from our potentially shallow cache?
- if fetch_all:
- pass
- # Fetching from a shallow-cloned repo was first supported in v1.9.0
- elif not self.ref or self.source.host_git_version is not None and self.source.host_git_version < (1, 9, 0):
- fetch_all = True
- else:
- m = re.match(EXACT_TAG_PATTERN, self.ref)
- if m is None:
- fetch_all = True
- else:
- tag = m.group("tag")
- commit = m.group("commit")
-
- if not self.remote_has_tag(url, tag, commit):
- self.source.status(
- "{}: {} is not advertised on {}. Fetching all Git refs".format(self.source, self.ref, url)
- )
- fetch_all = True
- else:
- exit_code = self.source.call(
- [
- self.source.host_git,
- "fetch",
- "--depth=1",
- url,
- "+refs/tags/{tag}:refs/tags/{tag}".format(tag=tag),
- ],
- cwd=self.mirror,
- )
- if exit_code != 0:
- self.source.status(
- "{}: Failed to fetch tag '{}' from {}. Fetching all Git refs".format(self.source, tag, url)
- )
- fetch_all = True
-
- if fetch_all:
- self.source.call(
- [
- self.source.host_git,
- "fetch",
- "--prune",
- url,
- "+refs/heads/*:refs/heads/*",
- "+refs/tags/*:refs/tags/*",
- ],
- fail="Failed to fetch from remote git repository: {}".format(url),
- fail_temporarily=True,
- cwd=self.mirror,
- )
-
- def fetch(self, alias_override=None): # pylint: disable=arguments-differ
- resolved_url = self.source.translate_url(self.url, alias_override=alias_override, primary=self.primary)
-
- with self.source.timed_activity("Fetching from {}".format(resolved_url), silent_nested=True):
- if not self.has_ref():
- self._fetch(resolved_url)
- self.assert_ref()
-
- def has_ref(self):
- if not self.ref:
- return False
-
- # If the mirror doesnt exist, we also dont have the ref
- if not os.path.exists(self.mirror):
- return False
-
- # Check if the ref is really there
- rc = self.source.call([self.source.host_git, "cat-file", "-t", self.ref], cwd=self.mirror)
- return rc == 0
-
- def assert_ref(self):
- if not self.has_ref():
- raise SourceError(
- "{}: expected ref '{}' was not found in git repository: '{}'".format(self.source, self.ref, self.url)
- )
-
- # remote_has_tag():
- #
- # Args:
- # url (str)
- # tag (str)
- # commit (str)
- #
- # Returns:
- # (bool): Whether the remote at `url` has the tag `tag` attached to `commit`
- #
- def remote_has_tag(self, url, tag, commit):
- _, ls_remote = self.source.check_output(
- [self.source.host_git, "ls-remote", url],
- cwd=self.mirror,
- fail="Failed to list advertised remote refs from git repository {}".format(url),
- )
-
- line = "{commit}\trefs/tags/{tag}".format(commit=commit, tag=tag)
- return line in ls_remote or line + "^{}" in ls_remote
-
- # to_commit():
- #
- # Args:
- # rev (str): A Git "commit-ish" rev
- #
- # Returns:
- # (str): The full revision ID of the commit
- #
- def to_commit(self, rev):
- _, output = self.source.check_output(
- [self.source.host_git, "rev-list", "-n", "1", rev],
- fail="Unable to find revision {}".format(rev),
- cwd=self.mirror,
- )
-
- return output.strip()
-
- # describe():
- #
- # Args:
- # rev (str): A Git "commit-ish" rev
- #
- # Returns:
- # (str): The full revision ID of the commit given by rev, prepended
- # with tag information as given by git-describe (where available)
- #
- def describe(self, rev):
- _, output = self.source.check_output(
- [self.source.host_git, "describe", "--tags", "--abbrev=40", "--long", "--always", rev],
- fail="Unable to find revision {}".format(rev),
- cwd=self.mirror,
- )
-
- return output.strip()
-
- # reachable_tags():
- #
- # Args:
- # rev (str): A Git "commit-ish" rev
- #
- # Returns:
- # (list): A list of tags in the ancestry of rev. Each entry is a triple of the form
- # (tag name (str), commit ref (str), annotated (bool)) describing a tag,
- # its tagged commit and whether it's annotated
- #
- def reachable_tags(self, rev):
- tags = set()
- for options in [[], ["--first-parent"], ["--tags"], ["--tags", "--first-parent"]]:
- exit_code, output = self.source.check_output(
- [self.source.host_git, "describe", "--abbrev=0", rev, *options], cwd=self.mirror
- )
- if exit_code == 0:
- tag = output.strip()
- _, commit_ref = self.source.check_output(
- [self.source.host_git, "rev-parse", tag + "^{commit}"],
- fail="Unable to resolve tag '{}'".format(tag),
- cwd=self.mirror,
- )
- exit_code = self.source.call([self.source.host_git, "cat-file", "tag", tag], cwd=self.mirror)
- annotated = exit_code == 0
-
- tags.add((tag, commit_ref.strip(), annotated))
-
- return list(tags)
-
- def stage(self, directory):
- fullpath = os.path.join(directory, self.path)
-
- # Using --shared here avoids copying the objects into the checkout, in any
- # case we're just checking out a specific commit and then removing the .git/
- # directory.
- self.source.call(
- [self.source.host_git, "clone", "--no-checkout", "--shared", self.mirror, fullpath],
- fail="Failed to create git mirror {} in directory: {}".format(self.mirror, fullpath),
- fail_temporarily=True,
- )
-
- self.source.call(
- [self.source.host_git, "checkout", "--force", self.ref],
- fail="Failed to checkout git ref {}".format(self.ref),
- cwd=fullpath,
- )
-
- # Remove .git dir
- shutil.rmtree(os.path.join(fullpath, ".git"))
-
- self._rebuild_git(fullpath)
-
- def init_workspace(self, directory):
- fullpath = os.path.join(directory, self.path)
- url = self.source.translate_url(self.url)
-
- self.source.call(
- [self.source.host_git, "clone", "--no-checkout", self.mirror, fullpath],
- fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
- fail_temporarily=True,
- )
-
- self.source.call(
- [self.source.host_git, "remote", "set-url", "origin", url],
- fail='Failed to add remote origin "{}"'.format(url),
- cwd=fullpath,
- )
-
- self.source.call(
- [self.source.host_git, "checkout", "--force", self.ref],
- fail="Failed to checkout git ref {}".format(self.ref),
- cwd=fullpath,
- )
-
- # get_submodule_mirrors():
- #
- # Returns:
- # An iterator through new instances of this class, one of each submodule
- # in the repo
- #
- def get_submodule_mirrors(self):
- for path, url in self.submodule_list():
- ref = self.submodule_ref(path)
- if ref is not None:
- mirror = self.__class__(self.source, os.path.join(self.path, path), url, ref)
- yield mirror
-
- # List the submodules (path/url tuples) present at the given ref of this repo
- def submodule_list(self):
- modules = "{}:{}".format(self.ref, GIT_MODULES)
- exit_code, output = self.source.check_output([self.source.host_git, "show", modules], cwd=self.mirror)
-
- # If git show reports error code 128 here, we take it to mean there is
- # no .gitmodules file to display for the given revision.
- if exit_code == 128:
- return
- elif exit_code != 0:
- raise SourceError("{plugin}: Failed to show gitmodules at ref {ref}".format(plugin=self, ref=self.ref))
-
- content = "\n".join([l.strip() for l in output.splitlines()])
-
- io = StringIO(content)
- parser = RawConfigParser()
- parser.read_file(io)
-
- for section in parser.sections():
- # validate section name against the 'submodule "foo"' pattern
- if re.match(r'submodule "(.*)"', section):
- path = parser.get(section, "path")
- url = parser.get(section, "url")
-
- yield (path, url)
-
- # Fetch the ref which this mirror requires its submodule to have,
- # at the given ref of this mirror.
- def submodule_ref(self, submodule, ref=None):
- if not ref:
- ref = self.ref
-
- # list objects in the parent repo tree to find the commit
- # object that corresponds to the submodule
- _, output = self.source.check_output(
- [self.source.host_git, "ls-tree", ref, submodule],
- fail="ls-tree failed for commit {} and submodule: {}".format(ref, submodule),
- cwd=self.mirror,
- )
-
- # read the commit hash from the output
- fields = output.split()
- if len(fields) >= 2 and fields[1] == "commit":
- submodule_commit = output.split()[2]
-
- # fail if the commit hash is invalid
- if len(submodule_commit) != 40:
- raise SourceError(
- "{}: Error reading commit information for submodule '{}'".format(self.source, submodule)
- )
-
- return submodule_commit
-
- else:
- detail = (
- "The submodule '{}' is defined either in the BuildStream source\n".format(submodule)
- + "definition, or in a .gitmodules file. But the submodule was never added to the\n"
- + "underlying git repository with `git submodule add`."
- )
-
- self.source.warn(
- "{}: Ignoring inconsistent submodule '{}'".format(self.source, submodule),
- detail=detail,
- warning_token=WARN_INCONSISTENT_SUBMODULE,
- )
-
- return None
-
- def _rebuild_git(self, fullpath):
- if not self.tags:
- return
-
- with self.source.tempdir() as tmpdir:
- included = set()
- shallow = set()
- for _, commit_ref, _ in self.tags:
-
- if commit_ref == self.ref:
- # rev-list does not work in case of same rev
- shallow.add(self.ref)
- else:
- _, out = self.source.check_output(
- [
- self.source.host_git,
- "rev-list",
- "--ancestry-path",
- "--boundary",
- "{}..{}".format(commit_ref, self.ref),
- ],
- fail="Failed to get git history {}..{} in directory: {}".format(
- commit_ref, self.ref, fullpath
- ),
- fail_temporarily=True,
- cwd=self.mirror,
- )
- self.source.warn("refs {}..{}: {}".format(commit_ref, self.ref, out.splitlines()))
- for line in out.splitlines():
- rev = line.lstrip("-")
- if line[0] == "-":
- shallow.add(rev)
- else:
- included.add(rev)
-
- shallow -= included
- included |= shallow
-
- self.source.call(
- [self.source.host_git, "init"],
- fail="Cannot initialize git repository: {}".format(fullpath),
- cwd=fullpath,
- )
-
- for rev in included:
- with TemporaryFile(dir=tmpdir) as commit_file:
- self.source.call(
- [self.source.host_git, "cat-file", "commit", rev],
- stdout=commit_file,
- fail="Failed to get commit {}".format(rev),
- cwd=self.mirror,
- )
- commit_file.seek(0, 0)
- self.source.call(
- [self.source.host_git, "hash-object", "-w", "-t", "commit", "--stdin"],
- stdin=commit_file,
- fail="Failed to add commit object {}".format(rev),
- cwd=fullpath,
- )
-
- with open(os.path.join(fullpath, ".git", "shallow"), "w", encoding="utf-8") as shallow_file:
- for rev in shallow:
- shallow_file.write("{}\n".format(rev))
-
- for tag, commit_ref, annotated in self.tags:
- if annotated:
- with TemporaryFile(dir=tmpdir) as tag_file:
- tag_data = "object {}\ntype commit\ntag {}\n".format(commit_ref, tag)
- tag_file.write(tag_data.encode("ascii"))
- tag_file.seek(0, 0)
- _, tag_ref = self.source.check_output(
- [self.source.host_git, "hash-object", "-w", "-t", "tag", "--stdin"],
- stdin=tag_file,
- fail="Failed to add tag object {}".format(tag),
- cwd=fullpath,
- )
-
- self.source.call(
- [self.source.host_git, "tag", tag, tag_ref.strip()],
- fail="Failed to tag: {}".format(tag),
- cwd=fullpath,
- )
- else:
- self.source.call(
- [self.source.host_git, "tag", tag, commit_ref],
- fail="Failed to tag: {}".format(tag),
- cwd=fullpath,
- )
-
- with open(os.path.join(fullpath, ".git", "HEAD"), "w", encoding="utf-8") as head:
- self.source.call(
- [self.source.host_git, "rev-parse", self.ref],
- stdout=head,
- fail="Failed to parse commit {}".format(self.ref),
- cwd=self.mirror,
- )
-
-
-class _GitSourceBase(Source):
- # pylint: disable=attribute-defined-outside-init
-
- # The GitMirror class which this plugin uses. This may be
- # overridden in derived plugins as long as the replacement class
- # follows the same interface used by the _GitMirror class
- BST_MIRROR_CLASS = _GitMirror
-
- def configure(self, node):
- ref = node.get_str("ref", None)
-
- config_keys = ["url", "track", "ref", "submodules", "checkout-submodules", "ref-format", "track-tags", "tags"]
- node.validate_keys(config_keys + Source.COMMON_CONFIG_KEYS)
-
- tags_node = node.get_sequence("tags", [])
- for tag_node in tags_node:
- tag_node.validate_keys(["tag", "commit", "annotated"])
-
- tags = self._load_tags(node)
- self.track_tags = node.get_bool("track-tags", default=False)
-
- self.original_url = node.get_str("url")
- self.mirror = self.BST_MIRROR_CLASS(self, "", self.original_url, ref, tags=tags, primary=True)
- self.tracking = node.get_str("track", None)
-
- self.ref_format = node.get_enum("ref-format", _RefFormat, _RefFormat.SHA1)
-
- # At this point we now know if the source has a ref and/or a track.
- # If it is missing both then we will be unable to track or build.
- if self.mirror.ref is None and self.tracking is None:
- raise SourceError(
- "{}: Git sources require a ref and/or track".format(self), reason="missing-track-and-ref"
- )
-
- self.checkout_submodules = node.get_bool("checkout-submodules", default=True)
-
- # Parse a dict of submodule overrides, stored in the submodule_overrides
- # and submodule_checkout_overrides dictionaries.
- self.submodule_overrides = {}
- self.submodule_checkout_overrides = {}
- modules = node.get_mapping("submodules", {})
- for path in modules.keys():
- submodule = modules.get_mapping(path)
- url = submodule.get_str("url", None)
-
- # Make sure to mark all URLs that are specified in the configuration
- if url:
- self.mark_download_url(url, primary=False)
-
- self.submodule_overrides[path] = url
- if "checkout" in submodule:
- checkout = submodule.get_bool("checkout")
- self.submodule_checkout_overrides[path] = checkout
-
- self.mark_download_url(self.original_url)
-
- def preflight(self):
- # Check if git is installed, get the binary at the same time
- self.host_git = utils.get_host_tool("git")
-
- rc, version_str = self.check_output([self.host_git, "--version"])
- # e.g. on Git for Windows we get "git version 2.21.0.windows.1".
- # e.g. on Mac via Homebrew we get "git version 2.19.0".
- if rc == 0:
- self.host_git_version = tuple(int(x) for x in version_str.split(" ")[2].split(".")[:3])
- else:
- self.host_git_version = None
-
- def get_unique_key(self):
- ref = self.mirror.ref
- if ref is not None:
- # Strip any (arbitary) tag information, leaving just the commit ID
- ref = _strip_tag(ref)
-
- # Here we want to encode the local name of the repository and
- # the ref, if the user changes the alias to fetch the same sources
- # from another location, it should not affect the cache key.
- key = [self.original_url, ref]
- if self.mirror.tags:
- tags = {tag: (commit, annotated) for tag, commit, annotated in self.mirror.tags}
- key.append({"tags": tags})
-
- # Only modify the cache key with checkout_submodules if it's something
- # other than the default behaviour.
- if self.checkout_submodules is False:
- key.append({"checkout_submodules": self.checkout_submodules})
-
- # We want the cache key to change if the source was
- # configured differently, and submodules count.
- if self.submodule_overrides:
- key.append(self.submodule_overrides)
-
- if self.submodule_checkout_overrides:
- key.append({"submodule_checkout_overrides": self.submodule_checkout_overrides})
-
- return key
-
- def is_resolved(self):
- return self.mirror.ref is not None
-
- def is_cached(self):
- return self._have_all_refs()
-
- def load_ref(self, node):
- self.mirror.ref = node.get_str("ref", None)
- self.mirror.tags = self._load_tags(node)
-
- def get_ref(self):
- if self.mirror.ref is None:
- return None
- return self.mirror.ref, self.mirror.tags
-
- def set_ref(self, ref, node):
- if not ref:
- self.mirror.ref = None
- if "ref" in node:
- del node["ref"]
- self.mirror.tags = []
- if "tags" in node:
- del node["tags"]
- else:
- actual_ref, tags = ref
- node["ref"] = self.mirror.ref = actual_ref
- self.mirror.tags = tags
- if tags:
- node["tags"] = []
- for tag, commit_ref, annotated in tags:
- data = {"tag": tag, "commit": commit_ref, "annotated": annotated}
- node["tags"].append(data)
- else:
- if "tags" in node:
- del node["tags"]
-
- def track(self): # pylint: disable=arguments-differ
-
- # If self.tracking is not specified it's not an error, just silently return
- if not self.tracking:
- # Is there a better way to check if a ref is given.
- if self.mirror.ref is None:
- detail = "Without a tracking branch ref can not be updated. Please " + "provide a ref or a track."
- raise SourceError("{}: No track or ref".format(self), detail=detail, reason="track-attempt-no-track")
- return None
-
- # Resolve the URL for the message
- resolved_url = self.translate_url(self.mirror.url)
- with self.timed_activity("Tracking {} from {}".format(self.tracking, resolved_url), silent_nested=True):
- self.mirror._fetch(resolved_url, fetch_all=True)
-
- ref = self.mirror.to_commit(self.tracking)
- tags = self.mirror.reachable_tags(ref) if self.track_tags else []
-
- if self.ref_format == _RefFormat.GIT_DESCRIBE:
- ref = self.mirror.describe(ref)
-
- return ref, tags
-
- def init_workspace(self, directory):
- with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
- self.mirror.init_workspace(directory)
- for mirror in self._recurse_submodules(configure=True):
- mirror.init_workspace(directory)
-
- def stage(self, directory):
- # Stage the main repo in the specified directory
- #
- with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
- self.mirror.stage(directory)
- for mirror in self._recurse_submodules(configure=True):
- mirror.stage(directory)
-
- def get_source_fetchers(self):
- self.mirror.mark_download_url(self.mirror.url)
- yield self.mirror
- # _recurse_submodules only iterates those which are known at the current
- # cached state - but fetch is called on each result as we go, so this will
- # yield all configured submodules
- for submodule in self._recurse_submodules(configure=True):
- submodule.mark_download_url(submodule.url)
- yield submodule
-
- def validate_cache(self):
- discovered_submodules = {}
- unlisted_submodules = []
- invalid_submodules = []
-
- for submodule in self._recurse_submodules(configure=False):
- discovered_submodules[submodule.path] = submodule.url
- if self._ignoring_submodule(submodule.path):
- continue
-
- if submodule.path not in self.submodule_overrides:
- unlisted_submodules.append((submodule.path, submodule.url))
-
- # Warn about submodules which are explicitly configured but do not exist
- for path, url in self.submodule_overrides.items():
- if path not in discovered_submodules:
- invalid_submodules.append((path, url))
-
- if invalid_submodules:
- detail = []
- for path, url in invalid_submodules:
- detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
-
- self.warn(
- "{}: Invalid submodules specified".format(self),
- warning_token=WARN_INVALID_SUBMODULE,
- detail="The following submodules are specified in the source "
- "description but do not exist according to the repository\n\n" + "\n".join(detail),
- )
-
- # Warn about submodules which exist but have not been explicitly configured
- if unlisted_submodules:
- detail = []
- for path, url in unlisted_submodules:
- detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
-
- self.warn(
- "{}: Unlisted submodules exist".format(self),
- warning_token=WARN_UNLISTED_SUBMODULE,
- detail="The following submodules exist but are not specified "
- + "in the source description\n\n"
- + "\n".join(detail),
- )
-
- # Assert that the ref exists in the track tag/branch, if track has been specified.
- # Also don't do this check if an exact tag ref is given, as we probably didn't fetch
- # any branch refs
- ref_in_track = False
- if not re.match(EXACT_TAG_PATTERN, self.mirror.ref) and self.tracking:
- _, branch = self.check_output(
- [self.host_git, "branch", "--list", self.tracking, "--contains", self.mirror.ref],
- cwd=self.mirror.mirror,
- )
- if branch:
- ref_in_track = True
- else:
- _, tag = self.check_output(
- [self.host_git, "tag", "--list", self.tracking, "--contains", self.mirror.ref],
- cwd=self.mirror.mirror,
- )
- if tag:
- ref_in_track = True
-
- if not ref_in_track:
- detail = (
- "The ref provided for the element does not exist locally "
- + "in the provided track branch / tag '{}'.\n".format(self.tracking)
- + "You may wish to track the element to update the ref from '{}' ".format(self.tracking)
- + "with `bst source track`,\n"
- + "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
- )
-
- self.warn(
- "{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n".format(
- self, self.mirror.ref, self.tracking, self.mirror.url
- ),
- detail=detail,
- warning_token=CoreWarnings.REF_NOT_IN_TRACK,
- )
-
- ###########################################################
- # Local Functions #
- ###########################################################
-
- def _have_all_refs(self):
- return self.mirror.has_ref() and all(
- submodule.has_ref() for submodule in self._recurse_submodules(configure=True)
- )
-
- # _configure_submodules():
- #
- # Args:
- # submodules: An iterator of _GitMirror (or similar) objects for submodules
- #
- # Returns:
- # An iterator through `submodules` but filtered of any ignored submodules
- # and modified to use any custom URLs configured in the source
- #
- def _configure_submodules(self, submodules):
- for submodule in submodules:
- if self._ignoring_submodule(submodule.path):
- continue
- # Allow configuration to override the upstream location of the submodules.
- submodule.url = self.submodule_overrides.get(submodule.path, submodule.url)
- yield submodule
-
- # _recurse_submodules():
- #
- # Recursively iterates through GitMirrors for submodules of the main repo. Only
- # submodules that are cached are recursed into - but this is decided at
- # iteration time, so you can fetch in a for loop over this function to fetch
- # all submodules.
- #
- # Args:
- # configure (bool): Whether to apply the 'submodule' config while recursing
- # (URL changing and 'checkout' overrides)
- #
- def _recurse_submodules(self, configure):
- def recurse(mirror):
- submodules = mirror.get_submodule_mirrors()
- if configure:
- submodules = self._configure_submodules(submodules)
-
- for submodule in submodules:
- yield submodule
- if submodule.has_ref():
- yield from recurse(submodule)
-
- yield from recurse(self.mirror)
-
- def _load_tags(self, node):
- tags = []
- tags_node = node.get_sequence("tags", [])
- for tag_node in tags_node:
- tag = tag_node.get_str("tag")
- commit_ref = tag_node.get_str("commit")
- annotated = tag_node.get_bool("annotated")
- tags.append((tag, commit_ref, annotated))
- return tags
-
- # _ignoring_submodule():
- #
- # Args:
- # path (str): The path of a submodule in the superproject
- #
- # Returns:
- # (bool): Whether to not clone/checkout this submodule
- #
- def _ignoring_submodule(self, path):
- return not self.submodule_checkout_overrides.get(path, self.checkout_submodules)
diff --git a/src/buildstream/plugins/sources/git.py b/src/buildstream/plugins/sources/git.py
deleted file mode 100644
index 0bdcea5..0000000
--- a/src/buildstream/plugins/sources/git.py
+++ /dev/null
@@ -1,164 +0,0 @@
-#
-# Copyright (C) 2016 Codethink Limited
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors:
-# Tristan Van Berkom <tr...@codethink.co.uk>
-
-"""
-git - stage files from a git repository
-=======================================
-
-**Host dependencies:**
-
- * git
-
-.. attention::
-
- Note that this plugin **will checkout git submodules by default**; even if
- they are not specified in the `.bst` file.
-
-**Usage:**
-
-.. code:: yaml
-
- # Specify the git source kind
- kind: git
-
- # Specify the repository url, using an alias defined
- # in your project configuration is recommended.
- url: upstream:foo.git
-
- # Optionally specify a symbolic tracking branch or tag, this
- # will be used to update the 'ref' when refreshing the pipeline.
- track: master
-
- # Optionally specify the ref format used for tracking.
- # The default is 'sha1' for the raw commit hash.
- # If you specify 'git-describe', the commit hash will be prefixed
- # with the closest tag.
- ref-format: sha1
-
- # Specify the commit ref, this must be specified in order to
- # checkout sources and build, but can be automatically updated
- # if the 'track' attribute was specified.
- ref: d63cbb6fdc0bbdadc4a1b92284826a6d63a7ebcd
-
- # Optionally specify whether submodules should be checked-out.
- # This is done recursively, as with `git clone --recurse-submodules`.
- # If not set, this will default to 'True'
- checkout-submodules: True
-
- # If your repository has submodules, explicitly specifying the
- # url from which they are to be fetched allows you to easily
- # rebuild the same sources from a different location. This is
- # especially handy when used with project defined aliases which
- # can be redefined at a later time.
- # You may also explicitly specify whether to check out this
- # submodule. If 'checkout' is set, it will control whether to
- # checkout that submodule and recurse into it. It defaults to the
- # value of 'checkout-submodules'.
- submodules:
- plugins/bar:
- url: upstream:bar.git
- checkout: True
- plugins/bar/quux:
- checkout: False
- plugins/baz:
- url: upstream:baz.git
- checkout: False
-
- # Enable tag tracking.
- #
- # This causes the `tags` metadata to be populated automatically
- # as a result of tracking the git source.
- #
- # By default this is 'False'.
- #
- track-tags: True
-
- # If the list of tags below is set, then a lightweight dummy
- # git repository will be staged along with the content at
- # build time.
- #
- # This is useful for a growing number of modules which use
- # `git describe` at build time in order to determine the version
- # which will be encoded into the built software.
- #
- # The 'tags' below is considered as a part of the git source
- # reference and will be stored in the 'project.refs' file if
- # that has been selected as your project's ref-storage.
- #
- # Migration notes:
- #
- # If you are upgrading from BuildStream 1.2, which used to
- # stage the entire repository by default, you will notice that
- # some modules which use `git describe` are broken, and will
- # need to enable this feature in order to fix them.
- #
- # If you need to enable this feature without changing the
- # the specific commit that you are building, then we recommend
- # the following migration steps for any git sources where
- # `git describe` is required:
- #
- # o Enable `track-tags` feature
- # o Set the `track` parameter to the desired commit sha which
- # the current `ref` points to
- # o Run `bst source track` for these elements, this will result in
- # populating the `tags` portion of the refs without changing
- # the refs
- # o Restore the `track` parameter to the branches which you have
- # previously been tracking afterwards.
- #
- tags:
- - tag: lightweight-example
- commit: 04ad0dc656cb7cc6feb781aa13bdbf1d67d0af78
- annotated: false
- - tag: annotated-example
- commit: 10abe77fe8d77385d86f225b503d9185f4ef7f3a
- annotated: true
-
-See :ref:`built-in functionality doumentation <core_source_builtins>` for
-details on common configuration options for sources.
-
-**Configurable Warnings:**
-
-This plugin provides the following :ref:`configurable warnings <configurable_warnings>`:
-
-- ``git:inconsistent-submodule`` - A submodule present in the git repository's .gitmodules was never
- added with `git submodule add`.
-
-- ``git:unlisted-submodule`` - A submodule is present in the git repository but was not specified in
- the source configuration and was not disabled for checkout.
-
-- ``git:invalid-submodule`` - A submodule is specified in the source configuration but does not exist
- in the repository.
-
-This plugin also utilises the following configurable :class:`core warnings <buildstream.types.CoreWarnings>`:
-
-- :attr:`ref-not-in-track <buildstream.types.CoreWarnings.REF_NOT_IN_TRACK>` - The provided ref was not
- found in the provided track in the element's git repository.
-"""
-
-from buildstream import _GitSourceBase
-
-
-class GitSource(_GitSourceBase):
-
- BST_MIN_VERSION = "2.0"
-
-
-# Plugin entry point
-def setup():
- return GitSource
[buildstream] 04/37: tests/cachekey: Removing bzr from cachekey tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 93d63c3343b2b562f8100edd0c66ddc54b40cb0b
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:12:23 2022 +0900
tests/cachekey: Removing bzr from cachekey tests
---
tests/cachekey/project/sources/bzr1.bst | 6 ------
tests/cachekey/project/sources/bzr1.expected | 1 -
tests/cachekey/project/target.bst | 1 -
tests/cachekey/project/target.expected | 2 +-
4 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/tests/cachekey/project/sources/bzr1.bst b/tests/cachekey/project/sources/bzr1.bst
deleted file mode 100644
index 4798d8a..0000000
--- a/tests/cachekey/project/sources/bzr1.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-sources:
-- kind: bzr
- url: https://launchpad.net/bzr
- track: trunk
- ref: 6622
diff --git a/tests/cachekey/project/sources/bzr1.expected b/tests/cachekey/project/sources/bzr1.expected
deleted file mode 100644
index dcad774..0000000
--- a/tests/cachekey/project/sources/bzr1.expected
+++ /dev/null
@@ -1 +0,0 @@
-a8ca1b3d852de71c0d97fbdddfe86f815a8edc62acf5aebbdcca1972a1627b12
\ No newline at end of file
diff --git a/tests/cachekey/project/target.bst b/tests/cachekey/project/target.bst
index 8ec6284..efef5ec 100644
--- a/tests/cachekey/project/target.bst
+++ b/tests/cachekey/project/target.bst
@@ -4,7 +4,6 @@ description: |
This is the main entry point including cases in the cache key test.
depends:
-- sources/bzr1.bst
- sources/git1.bst
- sources/git2.bst
- sources/git3.bst
diff --git a/tests/cachekey/project/target.expected b/tests/cachekey/project/target.expected
index 9734d67..94a1051 100644
--- a/tests/cachekey/project/target.expected
+++ b/tests/cachekey/project/target.expected
@@ -1 +1 @@
-f6351b90338635b277a832e8ceaa366ecdef231aae5d07c0da1cb01890837b97
\ No newline at end of file
+a83f2988088ce2a98abfbf7f3df8b04702a89d7e9d0628d7e922ce6f6d16b324
\ No newline at end of file
[buildstream] 09/37: plugins/sources/pip.py: Removing pip source
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7b70120e2310238683d05120d59848961e426aeb
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:39:51 2022 +0900
plugins/sources/pip.py: Removing pip source
And removing pip from the docs plugins index.
---
doc/source/core_plugins.rst | 1 -
src/buildstream/plugins/sources/pip.py | 263 ---------------------------------
2 files changed, 264 deletions(-)
diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst
index b6ecc87..f7eabde 100644
--- a/doc/source/core_plugins.rst
+++ b/doc/source/core_plugins.rst
@@ -53,7 +53,6 @@ information.
sources/tar
sources/git
sources/patch
- sources/pip
.. _plugins_external:
diff --git a/src/buildstream/plugins/sources/pip.py b/src/buildstream/plugins/sources/pip.py
deleted file mode 100644
index bb81647..0000000
--- a/src/buildstream/plugins/sources/pip.py
+++ /dev/null
@@ -1,263 +0,0 @@
-#
-# Copyright 2018 Bloomberg Finance LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors:
-# Chandan Singh <cs...@bloomberg.net>
-
-"""
-pip - stage python packages using pip
-=====================================
-
-**Host depndencies:**
-
- * ``pip`` python module
-
-This plugin will download source distributions for specified packages using
-``pip`` but will not install them. It is expected that the elements using this
-source will install the downloaded packages.
-
-Downloaded tarballs will be stored in a directory called ".bst_pip_downloads".
-
-**Usage:**
-
-.. code:: yaml
-
- # Specify the pip source kind
- kind: pip
-
- # Optionally specify index url, defaults to PyPi
- # This url is used to discover new versions of packages and download them
- # Projects intending to mirror their sources to a permanent location should
- # use an aliased url, and declare the alias in the project configuration
- url: https://mypypi.example.com/simple
-
- # Optionally specify the path to requirements files
- # Note that either 'requirements-files' or 'packages' must be defined
- requirements-files:
- - requirements.txt
-
- # Optionally specify a list of additional packages
- # Note that either 'requirements-files' or 'packages' must be defined
- packages:
- - flake8
-
- # Specify the ref. It is a list of strings of format
- # "<package-name>==<version>", separated by "\\n".
- # Usually this will be contents of a requirements.txt file where all
- # package versions have been frozen.
- ref: "flake8==3.5.0\\nmccabe==0.6.1\\npkg-resources==0.0.0\\npycodestyle==2.3.1\\npyflakes==1.6.0"
-
-See :ref:`built-in functionality doumentation <core_source_builtins>` for
-details on common configuration options for sources.
-"""
-
-import hashlib
-import os
-import re
-
-from buildstream import Source, SourceError, utils
-
-_OUTPUT_DIRNAME = ".bst_pip_downloads"
-_PYPI_INDEX_URL = "https://pypi.org/simple/"
-
-# Used only for finding pip command
-_PYTHON_VERSIONS = [
- "python", # when running in a venv, we might not have the exact version
- "python2.7",
- "python3.0",
- "python3.1",
- "python3.2",
- "python3.3",
- "python3.4",
- "python3.5",
- "python3.6",
- "python3.7",
- "python3.8",
- "python3.9",
-]
-
-# List of allowed extensions taken from
-# https://docs.python.org/3/distutils/sourcedist.html.
-# Names of source distribution archives must be of the form
-# '%{package-name}-%{version}.%{extension}'.
-_SDIST_RE = re.compile(r"^([\w.-]+?)-((?:[\d.]+){2,})\.(?:tar|tar.bz2|tar.gz|tar.xz|tar.Z|zip)$", re.IGNORECASE)
-
-
-class PipSource(Source):
- # pylint: disable=attribute-defined-outside-init
-
- BST_MIN_VERSION = "2.0"
-
- # We need access to previous sources at track time to use requirements.txt
- # but not at fetch time as self.ref should contain sufficient information
- # for this plugin
- BST_REQUIRES_PREVIOUS_SOURCES_TRACK = True
-
- def configure(self, node):
- node.validate_keys(["url", "packages", "ref", "requirements-files"] + Source.COMMON_CONFIG_KEYS)
- self.ref = node.get_str("ref", None)
- self.original_url = node.get_str("url", _PYPI_INDEX_URL)
- self.index_url = self.translate_url(self.original_url)
- self.packages = node.get_str_list("packages", [])
- self.requirements_files = node.get_str_list("requirements-files", [])
-
- if not (self.packages or self.requirements_files):
- raise SourceError("{}: Either 'packages' or 'requirements-files' must be specified".format(self))
-
- def preflight(self):
- # Try to find a pip version that spports download command
- self.host_pip = None
- for python in reversed(_PYTHON_VERSIONS):
- try:
- host_python = utils.get_host_tool(python)
- rc = self.call([host_python, "-m", "pip", "download", "--help"])
- if rc == 0:
- self.host_pip = [host_python, "-m", "pip"]
- break
- except utils.ProgramNotFoundError:
- pass
-
- if self.host_pip is None:
- raise SourceError("{}: Unable to find a suitable pip command".format(self))
-
- def get_unique_key(self):
- return [self.original_url, self.ref]
-
- def is_cached(self):
- return os.path.exists(self._mirror) and os.listdir(self._mirror)
-
- def get_ref(self):
- return self.ref
-
- def load_ref(self, node):
- self.ref = node.get_str("ref", None)
-
- def set_ref(self, ref, node):
- node["ref"] = self.ref = ref
-
- def track(self, previous_sources_dir): # pylint: disable=arguments-differ
- # XXX pip does not offer any public API other than the CLI tool so it
- # is not feasible to correctly parse the requirements file or to check
- # which package versions pip is going to install.
- # See https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program
- # for details.
- # As a result, we have to wastefully install the packages during track.
- with self.tempdir() as tmpdir:
- install_args = self.host_pip + [
- "download",
- "--no-binary",
- ":all:",
- "--index-url",
- self.index_url,
- "--dest",
- tmpdir,
- ]
- for requirement_file in self.requirements_files:
- fpath = os.path.join(previous_sources_dir, requirement_file)
- install_args += ["-r", fpath]
- install_args += self.packages
-
- self.call(install_args, fail="Failed to install python packages")
- reqs = self._parse_sdist_names(tmpdir)
-
- return "\n".join(["{}=={}".format(pkg, ver) for pkg, ver in reqs])
-
- def fetch(self): # pylint: disable=arguments-differ
- with self.tempdir() as tmpdir:
- packages = self.ref.strip().split("\n")
- package_dir = os.path.join(tmpdir, "packages")
- os.makedirs(package_dir)
- self.call(
- [
- *self.host_pip,
- "download",
- "--no-binary",
- ":all:",
- "--index-url",
- self.index_url,
- "--dest",
- package_dir,
- *packages,
- ],
- fail="Failed to install python packages: {}".format(packages),
- )
-
- # If the mirror directory already exists, assume that some other
- # process has fetched the sources before us and ensure that we do
- # not raise an error in that case.
- try:
- utils.move_atomic(package_dir, self._mirror)
- except utils.DirectoryExistsError:
- # Another process has beaten us and has fetched the sources
- # before us.
- pass
- except OSError as e:
- raise SourceError(
- "{}: Failed to move downloaded pip packages from '{}' to '{}': {}".format(
- self, package_dir, self._mirror, e
- )
- ) from e
-
- def stage(self, directory):
- with self.timed_activity("Staging Python packages", silent_nested=True):
- utils.copy_files(self._mirror, os.path.join(directory, _OUTPUT_DIRNAME))
-
- # Directory where this source should stage its files
- #
- @property
- def _mirror(self):
- if not self.ref:
- return None
- return os.path.join(
- self.get_mirror_directory(),
- utils.url_directory_name(self.original_url),
- hashlib.sha256(self.ref.encode()).hexdigest(),
- )
-
- # Parse names of downloaded source distributions
- #
- # Args:
- # basedir (str): Directory containing source distribution archives
- #
- # Returns:
- # (list): List of (package_name, version) tuples in sorted order
- #
- def _parse_sdist_names(self, basedir):
- reqs = []
- for f in os.listdir(basedir):
- pkg = _match_package_name(f)
- if pkg is not None:
- reqs.append(pkg)
-
- return sorted(reqs)
-
-
-# Extract the package name and version of a source distribution
-#
-# Args:
-# filename (str): Filename of the source distribution
-#
-# Returns:
-# (tuple): A tuple of (package_name, version)
-#
-def _match_package_name(filename):
- pkg_match = _SDIST_RE.match(filename)
- if pkg_match is None:
- return None
- return pkg_match.groups()
-
-
-def setup():
- return PipSource
[buildstream] 10/37: tests/sources/patch.py: Removing patch source tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 76cf305da7b174e9ee5dbd32a407a6e6309c53d6
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:59:03 2022 +0900
tests/sources/patch.py: Removing patch source tests
---
tests/sources/patch.py | 147 ---------------------
tests/sources/patch/basic/failure-empty-dir.bst | 5 -
.../patch/basic/failure-nonexistent-dir.bst | 6 -
tests/sources/patch/basic/file.txt | 1 -
tests/sources/patch/basic/file_1.patch | 7 -
tests/sources/patch/basic/irregular.bst | 7 -
tests/sources/patch/basic/project.conf | 3 -
tests/sources/patch/basic/target.bst | 7 -
tests/sources/patch/different-strip-level/file.txt | 1 -
.../patch/different-strip-level/file_1.patch | 7 -
.../patch/different-strip-level/project.conf | 3 -
.../sources/patch/different-strip-level/target.bst | 8 --
.../patch/invalid-relative-path/file_1.patch | 7 -
.../patch/invalid-relative-path/irregular.bst | 5 -
.../patch/invalid-relative-path/project.conf | 3 -
tests/sources/patch/multiple-patches/file.txt | 1 -
tests/sources/patch/multiple-patches/file_1.patch | 7 -
tests/sources/patch/multiple-patches/file_2.patch | 7 -
tests/sources/patch/multiple-patches/project.conf | 3 -
tests/sources/patch/multiple-patches/target.bst | 9 --
.../sources/patch/separate-patch-dir/file_1.patch | 7 -
.../separate-patch-dir/files/test-dir/file.txt | 1 -
.../sources/patch/separate-patch-dir/project.conf | 3 -
tests/sources/patch/separate-patch-dir/target.bst | 8 --
24 files changed, 263 deletions(-)
diff --git a/tests/sources/patch.py b/tests/sources/patch.py
deleted file mode 100644
index c34f5b3..0000000
--- a/tests/sources/patch.py
+++ /dev/null
@@ -1,147 +0,0 @@
-# Pylint doesn't play well with fixtures and dependency injection from pytest
-# pylint: disable=redefined-outer-name
-
-import os
-import pytest
-
-from buildstream.exceptions import ErrorDomain, LoadErrorReason
-from buildstream._testing import cli # pylint: disable=unused-import
-from tests.testutils import filetypegenerator
-
-DATA_DIR = os.path.join(
- os.path.dirname(os.path.realpath(__file__)),
- "patch",
-)
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_missing_patch(cli, datafiles):
- project = str(datafiles)
-
- # Removing the local file causes preflight to fail
- localfile = os.path.join(project, "file_1.patch")
- os.remove(localfile)
-
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.MISSING_FILE)
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_non_regular_file_patch(cli, datafiles):
- project = str(datafiles)
-
- patch_path = os.path.join(project, "irregular_file.patch")
- for _file_type in filetypegenerator.generate_file_types(patch_path):
- result = cli.run(project=project, args=["show", "irregular.bst"])
- if os.path.isfile(patch_path) and not os.path.islink(patch_path):
- result.assert_success()
- else:
- result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.PROJ_PATH_INVALID_KIND)
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_invalid_absolute_path(cli, datafiles):
- project = str(datafiles)
-
- with open(os.path.join(project, "target.bst"), "r", encoding="utf-8") as f:
- old_yaml = f.read()
- new_yaml = old_yaml.replace("file_1.patch", os.path.join(project, "file_1.patch"))
- assert old_yaml != new_yaml
-
- with open(os.path.join(project, "target.bst"), "w", encoding="utf-8") as f:
- f.write(new_yaml)
-
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.PROJ_PATH_INVALID)
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "invalid-relative-path"))
-def test_invalid_relative_path(cli, datafiles):
- project = str(datafiles)
-
- result = cli.run(project=project, args=["show", "irregular.bst"])
- result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.PROJ_PATH_INVALID)
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_stage_and_patch(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Build, checkout
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Test the file.txt was patched and changed
- with open(os.path.join(checkoutdir, "file.txt"), encoding="utf-8") as f:
- assert f.read() == "This is text file with superpowers\n"
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_stage_file_nonexistent_dir(cli, datafiles):
- project = str(datafiles)
-
- # Fails at build time because it tries to patch into a non-existing directory
- result = cli.run(project=project, args=["build", "failure-nonexistent-dir.bst"])
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.SOURCE, "patch-no-files")
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "basic"))
-def test_stage_file_empty_dir(cli, datafiles):
- project = str(datafiles)
-
- # Fails at build time because it tries to patch with nothing else staged
- result = cli.run(project=project, args=["build", "failure-empty-dir.bst"])
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.SOURCE, "patch-no-files")
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "separate-patch-dir"))
-def test_stage_separate_patch_dir(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Track, fetch, build, checkout
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Test the file.txt was patched and changed
- with open(os.path.join(checkoutdir, "test-dir", "file.txt"), encoding="utf-8") as f:
- assert f.read() == "This is text file in a directory with superpowers\n"
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "multiple-patches"))
-def test_stage_multiple_patches(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Track, fetch, build, checkout
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Test the file.txt was patched and changed
- with open(os.path.join(checkoutdir, "file.txt"), encoding="utf-8") as f:
- assert f.read() == "This is text file with more superpowers\n"
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "different-strip-level"))
-def test_patch_strip_level(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Track, fetch, build, checkout
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Test the file.txt was patched and changed
- with open(os.path.join(checkoutdir, "file.txt"), encoding="utf-8") as f:
- assert f.read() == "This is text file with superpowers\n"
diff --git a/tests/sources/patch/basic/failure-empty-dir.bst b/tests/sources/patch/basic/failure-empty-dir.bst
deleted file mode 100644
index b22af27..0000000
--- a/tests/sources/patch/basic/failure-empty-dir.bst
+++ /dev/null
@@ -1,5 +0,0 @@
-kind: import
-description: This is also the pony
-sources:
-- kind: patch
- path: file_1.patch
diff --git a/tests/sources/patch/basic/failure-nonexistent-dir.bst b/tests/sources/patch/basic/failure-nonexistent-dir.bst
deleted file mode 100644
index 8fd593d..0000000
--- a/tests/sources/patch/basic/failure-nonexistent-dir.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-description: This is also the pony
-sources:
-- kind: patch
- path: file_1.patch
- directory: /idontexist
diff --git a/tests/sources/patch/basic/file.txt b/tests/sources/patch/basic/file.txt
deleted file mode 100644
index a496efe..0000000
--- a/tests/sources/patch/basic/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-This is a text file
diff --git a/tests/sources/patch/basic/file_1.patch b/tests/sources/patch/basic/file_1.patch
deleted file mode 100644
index 424a486..0000000
--- a/tests/sources/patch/basic/file_1.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git a/file.txt b/file.txt
-index a496efe..341ef26 100644
---- a/file.txt
-+++ b/file.txt
-@@ -1 +1 @@
--This is a text file
-+This is text file with superpowers
diff --git a/tests/sources/patch/basic/irregular.bst b/tests/sources/patch/basic/irregular.bst
deleted file mode 100644
index 425cbcc..0000000
--- a/tests/sources/patch/basic/irregular.bst
+++ /dev/null
@@ -1,7 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: local
- path: file.txt
-- kind: patch
- path: irregular_file.patch
diff --git a/tests/sources/patch/basic/project.conf b/tests/sources/patch/basic/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/patch/basic/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/patch/basic/target.bst b/tests/sources/patch/basic/target.bst
deleted file mode 100644
index 913371d..0000000
--- a/tests/sources/patch/basic/target.bst
+++ /dev/null
@@ -1,7 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: local
- path: file.txt
-- kind: patch
- path: file_1.patch
diff --git a/tests/sources/patch/different-strip-level/file.txt b/tests/sources/patch/different-strip-level/file.txt
deleted file mode 100644
index a496efe..0000000
--- a/tests/sources/patch/different-strip-level/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-This is a text file
diff --git a/tests/sources/patch/different-strip-level/file_1.patch b/tests/sources/patch/different-strip-level/file_1.patch
deleted file mode 100644
index ff7f7fe..0000000
--- a/tests/sources/patch/different-strip-level/file_1.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git foo/a/file.txt foo/b/file.txt
-index a496efe..341ef26 100644
---- foo/a/file.txt
-+++ foo/b/file.txt
-@@ -1 +1 @@
--This is a text file
-+This is text file with superpowers
diff --git a/tests/sources/patch/different-strip-level/project.conf b/tests/sources/patch/different-strip-level/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/patch/different-strip-level/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/patch/different-strip-level/target.bst b/tests/sources/patch/different-strip-level/target.bst
deleted file mode 100644
index c8ea19a..0000000
--- a/tests/sources/patch/different-strip-level/target.bst
+++ /dev/null
@@ -1,8 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: local
- path: file.txt
-- kind: patch
- path: file_1.patch
- strip-level: 2
diff --git a/tests/sources/patch/invalid-relative-path/file_1.patch b/tests/sources/patch/invalid-relative-path/file_1.patch
deleted file mode 100644
index 424a486..0000000
--- a/tests/sources/patch/invalid-relative-path/file_1.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git a/file.txt b/file.txt
-index a496efe..341ef26 100644
---- a/file.txt
-+++ b/file.txt
-@@ -1 +1 @@
--This is a text file
-+This is text file with superpowers
diff --git a/tests/sources/patch/invalid-relative-path/irregular.bst b/tests/sources/patch/invalid-relative-path/irregular.bst
deleted file mode 100644
index 6b63a4e..0000000
--- a/tests/sources/patch/invalid-relative-path/irregular.bst
+++ /dev/null
@@ -1,5 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: patch
- path: ../invalid-relative-path/irregular_file.patch
diff --git a/tests/sources/patch/invalid-relative-path/project.conf b/tests/sources/patch/invalid-relative-path/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/patch/invalid-relative-path/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/patch/multiple-patches/file.txt b/tests/sources/patch/multiple-patches/file.txt
deleted file mode 100644
index a496efe..0000000
--- a/tests/sources/patch/multiple-patches/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-This is a text file
diff --git a/tests/sources/patch/multiple-patches/file_1.patch b/tests/sources/patch/multiple-patches/file_1.patch
deleted file mode 100644
index 424a486..0000000
--- a/tests/sources/patch/multiple-patches/file_1.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git a/file.txt b/file.txt
-index a496efe..341ef26 100644
---- a/file.txt
-+++ b/file.txt
-@@ -1 +1 @@
--This is a text file
-+This is text file with superpowers
diff --git a/tests/sources/patch/multiple-patches/file_2.patch b/tests/sources/patch/multiple-patches/file_2.patch
deleted file mode 100644
index f56614b..0000000
--- a/tests/sources/patch/multiple-patches/file_2.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git a/file.txt b/file.txt
-index a496efe..341ef26 100644
---- a/file.txt
-+++ b/file.txt
-@@ -1 +1 @@
--This is text file with superpowers
-+This is text file with more superpowers
diff --git a/tests/sources/patch/multiple-patches/project.conf b/tests/sources/patch/multiple-patches/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/patch/multiple-patches/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/patch/multiple-patches/target.bst b/tests/sources/patch/multiple-patches/target.bst
deleted file mode 100644
index 4665e7d..0000000
--- a/tests/sources/patch/multiple-patches/target.bst
+++ /dev/null
@@ -1,9 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: local
- path: file.txt
-- kind: patch
- path: file_1.patch
-- kind: patch
- path: file_2.patch
diff --git a/tests/sources/patch/separate-patch-dir/file_1.patch b/tests/sources/patch/separate-patch-dir/file_1.patch
deleted file mode 100644
index ae8bc33..0000000
--- a/tests/sources/patch/separate-patch-dir/file_1.patch
+++ /dev/null
@@ -1,7 +0,0 @@
-diff --git a/file.txt b/file.txt
-index a496efe..341ef26 100644
---- a/file.txt
-+++ b/file.txt
-@@ -1 +1 @@
--This is a text file in a directory
-+This is text file in a directory with superpowers
diff --git a/tests/sources/patch/separate-patch-dir/files/test-dir/file.txt b/tests/sources/patch/separate-patch-dir/files/test-dir/file.txt
deleted file mode 100644
index 425911a..0000000
--- a/tests/sources/patch/separate-patch-dir/files/test-dir/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-This is a text file in a directory
diff --git a/tests/sources/patch/separate-patch-dir/project.conf b/tests/sources/patch/separate-patch-dir/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/patch/separate-patch-dir/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/patch/separate-patch-dir/target.bst b/tests/sources/patch/separate-patch-dir/target.bst
deleted file mode 100644
index 796c131..0000000
--- a/tests/sources/patch/separate-patch-dir/target.bst
+++ /dev/null
@@ -1,8 +0,0 @@
-kind: import
-description: This is the pony
-sources:
-- kind: local
- path: files
-- kind: patch
- path: file_1.patch
- directory: test-dir
[buildstream] 11/37: tests/cachekey: Removing patch source from cache key tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7f9756dc61cdfcd0552b6c3cf1fda3497b7ab8ea
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 17:02:13 2022 +0900
tests/cachekey: Removing patch source from cache key tests
---
tests/cachekey/project/sources/patch1.bst | 4 ----
tests/cachekey/project/sources/patch1.expected | 1 -
tests/cachekey/project/sources/patch2.bst | 6 ------
tests/cachekey/project/sources/patch2.expected | 1 -
tests/cachekey/project/sources/patch3.bst | 6 ------
tests/cachekey/project/sources/patch3.expected | 1 -
tests/cachekey/project/target.bst | 3 ---
tests/cachekey/project/target.expected | 2 +-
8 files changed, 1 insertion(+), 23 deletions(-)
diff --git a/tests/cachekey/project/sources/patch1.bst b/tests/cachekey/project/sources/patch1.bst
deleted file mode 100644
index 255eeb2..0000000
--- a/tests/cachekey/project/sources/patch1.bst
+++ /dev/null
@@ -1,4 +0,0 @@
-kind: import
-sources:
-- kind: patch
- path: files/patches/patch.diff
diff --git a/tests/cachekey/project/sources/patch1.expected b/tests/cachekey/project/sources/patch1.expected
deleted file mode 100644
index 4105aa7..0000000
--- a/tests/cachekey/project/sources/patch1.expected
+++ /dev/null
@@ -1 +0,0 @@
-ebe10de8ffc95d9d0e3472db032a4eda5e21b6dd36fade5e97fca08745a70552
\ No newline at end of file
diff --git a/tests/cachekey/project/sources/patch2.bst b/tests/cachekey/project/sources/patch2.bst
deleted file mode 100644
index faab3a2..0000000
--- a/tests/cachekey/project/sources/patch2.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-sources:
-- kind: patch
- path: files/patches/patch.diff
- directory: usr/bin
- strip-level: 1
diff --git a/tests/cachekey/project/sources/patch2.expected b/tests/cachekey/project/sources/patch2.expected
deleted file mode 100644
index 8710f42..0000000
--- a/tests/cachekey/project/sources/patch2.expected
+++ /dev/null
@@ -1 +0,0 @@
-db89febc96247d6b0730c7d21041cfef074256866e20aff8372207b962aacd14
\ No newline at end of file
diff --git a/tests/cachekey/project/sources/patch3.bst b/tests/cachekey/project/sources/patch3.bst
deleted file mode 100644
index cd22c6b..0000000
--- a/tests/cachekey/project/sources/patch3.bst
+++ /dev/null
@@ -1,6 +0,0 @@
-kind: import
-sources:
-- kind: patch
- path: files/patches/patch.diff
- directory: usr/bin
- strip-level: 3
diff --git a/tests/cachekey/project/sources/patch3.expected b/tests/cachekey/project/sources/patch3.expected
deleted file mode 100644
index 19b94ad..0000000
--- a/tests/cachekey/project/sources/patch3.expected
+++ /dev/null
@@ -1 +0,0 @@
-381a0bab35e9e91a9746bd1b6611d773fa6c77aff0317764b9a0379762be16c9
\ No newline at end of file
diff --git a/tests/cachekey/project/target.bst b/tests/cachekey/project/target.bst
index 4e253a6..cd5cc31 100644
--- a/tests/cachekey/project/target.bst
+++ b/tests/cachekey/project/target.bst
@@ -9,9 +9,6 @@ depends:
- sources/git3.bst
- sources/local1.bst
- sources/local2.bst
-- sources/patch1.bst
-- sources/patch2.bst
-- sources/patch3.bst
- sources/remote1.bst
- sources/remote2.bst
- sources/tar1.bst
diff --git a/tests/cachekey/project/target.expected b/tests/cachekey/project/target.expected
index 6f310dd..bccc890 100644
--- a/tests/cachekey/project/target.expected
+++ b/tests/cachekey/project/target.expected
@@ -1 +1 @@
-16a729fbaad947673ad95468fcf6d00b49fa76b3d6a80fc0866abb8bd9a8a371
\ No newline at end of file
+66c2423caaae90d058d6156ee82832ac817c9f44fac766acaceb6b2d54fabb9c
\ No newline at end of file
[buildstream] 08/37: tests/cachekey: Removing pip source from cachekey test
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 95810de20d145b616b0b0dbfc25354aac04826e8
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:38:42 2022 +0900
tests/cachekey: Removing pip source from cachekey test
---
tests/cachekey/project/sources/pip1.bst | 12 ------------
tests/cachekey/project/sources/pip1.expected | 1 -
tests/cachekey/project/target.bst | 1 -
tests/cachekey/project/target.expected | 2 +-
4 files changed, 1 insertion(+), 15 deletions(-)
diff --git a/tests/cachekey/project/sources/pip1.bst b/tests/cachekey/project/sources/pip1.bst
deleted file mode 100644
index ee69efa..0000000
--- a/tests/cachekey/project/sources/pip1.bst
+++ /dev/null
@@ -1,12 +0,0 @@
-kind: import
-
-sources:
-- kind: git
- url: https://example.com/foo/foobar.git
- ref: b99955530263172ed1beae52aed7a33885ef781f
-- kind: pip
- url: https://pypi.example.com/simple
- packages:
- - horses
- - ponies
- ref: 'horses==0.0.1\nponies==0.0.2'
diff --git a/tests/cachekey/project/sources/pip1.expected b/tests/cachekey/project/sources/pip1.expected
deleted file mode 100644
index ee2f814..0000000
--- a/tests/cachekey/project/sources/pip1.expected
+++ /dev/null
@@ -1 +0,0 @@
-99c354c584dddbebe32f36005211dafce09847ea4b4c5003191eb585a1c81576
\ No newline at end of file
diff --git a/tests/cachekey/project/target.bst b/tests/cachekey/project/target.bst
index efef5ec..4e253a6 100644
--- a/tests/cachekey/project/target.bst
+++ b/tests/cachekey/project/target.bst
@@ -12,7 +12,6 @@ depends:
- sources/patch1.bst
- sources/patch2.bst
- sources/patch3.bst
-- sources/pip1.bst
- sources/remote1.bst
- sources/remote2.bst
- sources/tar1.bst
diff --git a/tests/cachekey/project/target.expected b/tests/cachekey/project/target.expected
index 94a1051..6f310dd 100644
--- a/tests/cachekey/project/target.expected
+++ b/tests/cachekey/project/target.expected
@@ -1 +1 @@
-a83f2988088ce2a98abfbf7f3df8b04702a89d7e9d0628d7e922ce6f6d16b324
\ No newline at end of file
+16a729fbaad947673ad95468fcf6d00b49fa76b3d6a80fc0866abb8bd9a8a371
\ No newline at end of file
[buildstream] 16/37: tests/plugins/sample-plugins: Adding git source to sample plugins
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d178798c4189a5a735183fafc514065165c17155
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Tue Mar 29 15:49:07 2022 +0900
tests/plugins/sample-plugins: Adding git source to sample plugins
---
tests/plugins/sample-plugins/setup.py | 1 +
.../src/sample_plugins/sources/git.py | 1059 ++++++++++++++++++++
2 files changed, 1060 insertions(+)
diff --git a/tests/plugins/sample-plugins/setup.py b/tests/plugins/sample-plugins/setup.py
index 070f8c7..0db7481 100755
--- a/tests/plugins/sample-plugins/setup.py
+++ b/tests/plugins/sample-plugins/setup.py
@@ -32,6 +32,7 @@ setup(
],
"buildstream.plugins.sources": [
"sample = sample_plugins.sources.sample",
+ "git = sample_plugins.sources.git",
],
},
zip_safe=False,
diff --git a/tests/plugins/sample-plugins/src/sample_plugins/sources/git.py b/tests/plugins/sample-plugins/src/sample_plugins/sources/git.py
new file mode 100644
index 0000000..decd2ec
--- /dev/null
+++ b/tests/plugins/sample-plugins/src/sample_plugins/sources/git.py
@@ -0,0 +1,1059 @@
+#
+# Copyright (C) 2016 Codethink Limited
+# Copyright (C) 2018 Bloomberg Finance LP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# Tristan Van Berkom <tr...@codethink.co.uk>
+# Chandan Singh <cs...@bloomberg.net>
+# Tom Mewett <to...@codethink.co.uk>
+
+"""
+git - stage files from a git repository
+=======================================
+
+**Host dependencies:**
+
+ * git
+
+.. attention::
+
+ Note that this plugin **will checkout git submodules by default**; even if
+ they are not specified in the `.bst` file.
+
+**Usage:**
+
+.. code:: yaml
+
+ # Specify the git source kind
+ kind: git
+
+ # Specify the repository url, using an alias defined
+ # in your project configuration is recommended.
+ url: upstream:foo.git
+
+ # Optionally specify a symbolic tracking branch or tag, this
+ # will be used to update the 'ref' when refreshing the pipeline.
+ track: master
+
+ # Optionally specify the ref format used for tracking.
+ # The default is 'sha1' for the raw commit hash.
+ # If you specify 'git-describe', the commit hash will be prefixed
+ # with the closest tag.
+ ref-format: sha1
+
+ # Specify the commit ref, this must be specified in order to
+ # checkout sources and build, but can be automatically updated
+ # if the 'track' attribute was specified.
+ ref: d63cbb6fdc0bbdadc4a1b92284826a6d63a7ebcd
+
+ # Optionally specify whether submodules should be checked-out.
+ # This is done recursively, as with `git clone --recurse-submodules`.
+ # If not set, this will default to 'True'
+ checkout-submodules: True
+
+ # If your repository has submodules, explicitly specifying the
+ # url from which they are to be fetched allows you to easily
+ # rebuild the same sources from a different location. This is
+ # especially handy when used with project defined aliases which
+ # can be redefined at a later time.
+ # You may also explicitly specify whether to check out this
+ # submodule. If 'checkout' is set, it will control whether to
+ # checkout that submodule and recurse into it. It defaults to the
+ # value of 'checkout-submodules'.
+ submodules:
+ plugins/bar:
+ url: upstream:bar.git
+ checkout: True
+ plugins/bar/quux:
+ checkout: False
+ plugins/baz:
+ url: upstream:baz.git
+ checkout: False
+
+ # Enable tag tracking.
+ #
+ # This causes the `tags` metadata to be populated automatically
+ # as a result of tracking the git source.
+ #
+ # By default this is 'False'.
+ #
+ track-tags: True
+
+ # If the list of tags below is set, then a lightweight dummy
+ # git repository will be staged along with the content at
+ # build time.
+ #
+ # This is useful for a growing number of modules which use
+ # `git describe` at build time in order to determine the version
+ # which will be encoded into the built software.
+ #
+ # The 'tags' below is considered as a part of the git source
+ # reference and will be stored in the 'project.refs' file if
+ # that has been selected as your project's ref-storage.
+ #
+ # Migration notes:
+ #
+ # If you are upgrading from BuildStream 1.2, which used to
+ # stage the entire repository by default, you will notice that
+ # some modules which use `git describe` are broken, and will
+ # need to enable this feature in order to fix them.
+ #
+ # If you need to enable this feature without changing the
+ # the specific commit that you are building, then we recommend
+ # the following migration steps for any git sources where
+ # `git describe` is required:
+ #
+ # o Enable `track-tags` feature
+ # o Set the `track` parameter to the desired commit sha which
+ # the current `ref` points to
+ # o Run `bst source track` for these elements, this will result in
+ # populating the `tags` portion of the refs without changing
+ # the refs
+ # o Restore the `track` parameter to the branches which you have
+ # previously been tracking afterwards.
+ #
+ tags:
+ - tag: lightweight-example
+ commit: 04ad0dc656cb7cc6feb781aa13bdbf1d67d0af78
+ annotated: false
+ - tag: annotated-example
+ commit: 10abe77fe8d77385d86f225b503d9185f4ef7f3a
+ annotated: true
+
+See `built-in functionality doumentation
+<https://docs.buildstream.build/master/buildstream.source.html#core-source-builtins>`_ for
+details on common configuration options for sources.
+
+
+**Configurable Warnings:**
+
+This plugin provides the following
+`configurable warnings <https://docs.buildstream.build/master/format_project.html#configurable-warnings>`_:
+
+- ``git:inconsistent-submodule`` - A submodule present in the git repository's .gitmodules was never
+ added with `git submodule add`.
+
+- ``git:unlisted-submodule`` - A submodule is present in the git repository but was not specified in
+ the source configuration and was not disabled for checkout.
+
+- ``git:invalid-submodule`` - A submodule is specified in the source configuration but does not exist
+ in the repository.
+
+This plugin also utilises the following configurable
+`core warnings <https://docs.buildstream.build/master/buildstream.types.html#buildstream.types.CoreWarnings>`_:
+
+- `ref-not-in-track <https://docs.buildstream.build/master/buildstream.types.html#buildstream.types.CoreWarnings.REF_NOT_IN_TRACK>`_ -
+ The provided ref was not found in the provided track in the element's git repository.
+"""
+
+
+import os
+import re
+import shutil
+from io import StringIO
+from tempfile import TemporaryFile
+
+from configparser import RawConfigParser
+
+from buildstream import Source, SourceError, SourceFetcher
+from buildstream import CoreWarnings, FastEnum
+from buildstream import utils
+from buildstream.utils import DirectoryExistsError
+
+GIT_MODULES = ".gitmodules"
+EXACT_TAG_PATTERN = r"(?P<tag>.*)-0-g(?P<commit>.*)"
+
+# Warnings
+WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
+WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
+WARN_INVALID_SUBMODULE = "invalid-submodule"
+
+
+class _RefFormat(FastEnum):
+ SHA1 = "sha1"
+ GIT_DESCRIBE = "git-describe"
+
+
+def _strip_tag(rev):
+ return rev.split("-g")[-1]
+
+
+# This class represents a single Git repository. The Git source needs to account for
+# submodules, but we don't want to cache them all under the umbrella of the
+# superproject - so we use this class which caches them independently, according
+# to their URL. Instances keep reference to their "parent" GitSource,
+# and if applicable, where in the superproject they are found.
+#
+# Args:
+# source (GitSource): The parent source
+# path (str): The relative location of the submodule in the superproject;
+# the empty string for the superproject itself
+# url (str): Where to clone the repo from
+# ref (str): Specified 'ref' from the source configuration
+# primary (bool): Whether this is the primary URL for the source
+# tags (list): Tag configuration; see GitSource._load_tags
+#
+class GitMirror(SourceFetcher):
+ def __init__(self, source, path, url, ref, *, primary=False, tags=None):
+
+ super().__init__()
+ self.source = source
+ self.path = path
+ self.url = url
+ self.ref = ref
+ self.tags = tags or []
+ self.primary = primary
+ self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
+
+ # _ensure_repo():
+ #
+ # Ensures that the Git repository exists at the mirror location and is configured
+ # to fetch from the given URL
+ #
+ def _ensure_repo(self):
+ if not os.path.exists(self.mirror):
+ with self.source.tempdir() as tmpdir:
+ self.source.call(
+ [self.source.host_git, "init", "--bare", tmpdir],
+ fail="Failed to initialise repository",
+ )
+
+ try:
+ utils.move_atomic(tmpdir, self.mirror)
+ except DirectoryExistsError:
+ # Another process was quicker to download this repository.
+ # Let's discard our own
+ self.source.status("{}: Discarding duplicate repository".format(self.source))
+ except OSError as e:
+ raise SourceError(
+ "{}: Failed to move created repository from '{}' to '{}': {}".format(
+ self.source, tmpdir, self.mirror, e
+ )
+ ) from e
+
+ def _fetch(self, url, fetch_all=False):
+ self._ensure_repo()
+
+ # Work out whether we can fetch a specific tag: are we given a ref which
+ # 1. is in git-describe format
+ # 2. refers to an exact tag (is "...-0-g...")
+ # 3. is available on the remote and tags the specified commit?
+ # And lastly: are we on a new-enough Git which allows cloning from our potentially shallow cache?
+ if fetch_all:
+ pass
+ # Fetching from a shallow-cloned repo was first supported in v1.9.0
+ elif not self.ref or self.source.host_git_version is not None and self.source.host_git_version < (1, 9, 0):
+ fetch_all = True
+ else:
+ m = re.match(EXACT_TAG_PATTERN, self.ref)
+ if m is None:
+ fetch_all = True
+ else:
+ tag = m.group("tag")
+ commit = m.group("commit")
+
+ if not self.remote_has_tag(url, tag, commit):
+ self.source.status(
+ "{}: {} is not advertised on {}. Fetching all Git refs".format(self.source, self.ref, url)
+ )
+ fetch_all = True
+ else:
+ exit_code = self.source.call(
+ [
+ self.source.host_git,
+ "fetch",
+ "--depth=1",
+ url,
+ "+refs/tags/{tag}:refs/tags/{tag}".format(tag=tag),
+ ],
+ cwd=self.mirror,
+ )
+ if exit_code != 0:
+ self.source.status(
+ "{}: Failed to fetch tag '{}' from {}. Fetching all Git refs".format(self.source, tag, url)
+ )
+ fetch_all = True
+
+ if fetch_all:
+ self.source.call(
+ [
+ self.source.host_git,
+ "fetch",
+ "--prune",
+ url,
+ "+refs/heads/*:refs/heads/*",
+ "+refs/tags/*:refs/tags/*",
+ ],
+ fail="Failed to fetch from remote git repository: {}".format(url),
+ fail_temporarily=True,
+ cwd=self.mirror,
+ )
+
+ def fetch(self, alias_override=None): # pylint: disable=arguments-differ
+ resolved_url = self.source.translate_url(self.url, alias_override=alias_override, primary=self.primary)
+
+ with self.source.timed_activity("Fetching from {}".format(resolved_url), silent_nested=True):
+ if not self.has_ref():
+ self._fetch(resolved_url)
+ self.assert_ref()
+
+ def has_ref(self):
+ if not self.ref:
+ return False
+
+ # If the mirror doesnt exist, we also dont have the ref
+ if not os.path.exists(self.mirror):
+ return False
+
+ # Check if the ref is really there
+ rc = self.source.call([self.source.host_git, "cat-file", "-t", self.ref], cwd=self.mirror)
+ return rc == 0
+
+ def assert_ref(self):
+ if not self.has_ref():
+ raise SourceError(
+ "{}: expected ref '{}' was not found in git repository: '{}'".format(self.source, self.ref, self.url)
+ )
+
+ # remote_has_tag():
+ #
+ # Args:
+ # url (str)
+ # tag (str)
+ # commit (str)
+ #
+ # Returns:
+ # (bool): Whether the remote at `url` has the tag `tag` attached to `commit`
+ #
+ def remote_has_tag(self, url, tag, commit):
+ _, ls_remote = self.source.check_output(
+ [self.source.host_git, "ls-remote", url],
+ cwd=self.mirror,
+ fail="Failed to list advertised remote refs from git repository {}".format(url),
+ )
+
+ line = "{commit}\trefs/tags/{tag}".format(commit=commit, tag=tag)
+ return line in ls_remote or line + "^{}" in ls_remote
+
+ # to_commit():
+ #
+ # Args:
+ # rev (str): A Git "commit-ish" rev
+ #
+ # Returns:
+ # (str): The full revision ID of the commit
+ #
+ def to_commit(self, rev):
+ _, output = self.source.check_output(
+ [self.source.host_git, "rev-list", "-n", "1", rev],
+ fail="Unable to find revision {}".format(rev),
+ cwd=self.mirror,
+ )
+
+ return output.strip()
+
+ # describe():
+ #
+ # Args:
+ # rev (str): A Git "commit-ish" rev
+ #
+ # Returns:
+ # (str): The full revision ID of the commit given by rev, prepended
+ # with tag information as given by git-describe (where available)
+ #
+ def describe(self, rev):
+ _, output = self.source.check_output(
+ [
+ self.source.host_git,
+ "describe",
+ "--tags",
+ "--abbrev=40",
+ "--long",
+ "--always",
+ rev,
+ ],
+ fail="Unable to find revision {}".format(rev),
+ cwd=self.mirror,
+ )
+
+ return output.strip()
+
+ # reachable_tags():
+ #
+ # Args:
+ # rev (str): A Git "commit-ish" rev
+ #
+ # Returns:
+ # (list): A list of tags in the ancestry of rev. Each entry is a triple of the form
+ # (tag name (str), commit ref (str), annotated (bool)) describing a tag,
+ # its tagged commit and whether it's annotated
+ #
+ def reachable_tags(self, rev):
+ tags = set()
+ for options in [
+ [],
+ ["--first-parent"],
+ ["--tags"],
+ ["--tags", "--first-parent"],
+ ]:
+ exit_code, output = self.source.check_output(
+ [
+ self.source.host_git,
+ "describe",
+ "--abbrev=0",
+ rev,
+ *options,
+ ],
+ cwd=self.mirror,
+ )
+ if exit_code == 0:
+ tag = output.strip()
+ _, commit_ref = self.source.check_output(
+ [self.source.host_git, "rev-parse", tag + "^{commit}"],
+ fail="Unable to resolve tag '{}'".format(tag),
+ cwd=self.mirror,
+ )
+ exit_code = self.source.call(
+ [self.source.host_git, "cat-file", "tag", tag],
+ cwd=self.mirror,
+ )
+ annotated = exit_code == 0
+
+ tags.add((tag, commit_ref.strip(), annotated))
+
+ return list(tags)
+
+ def stage(self, directory):
+ fullpath = os.path.join(directory, self.path)
+
+ # Using --shared here avoids copying the objects into the checkout, in any
+ # case we're just checking out a specific commit and then removing the .git/
+ # directory.
+ self.source.call(
+ [
+ self.source.host_git,
+ "clone",
+ "--no-checkout",
+ "--shared",
+ self.mirror,
+ fullpath,
+ ],
+ fail="Failed to create git mirror {} in directory: {}".format(self.mirror, fullpath),
+ fail_temporarily=True,
+ )
+
+ self.source.call(
+ [self.source.host_git, "checkout", "--force", self.ref],
+ fail="Failed to checkout git ref {}".format(self.ref),
+ cwd=fullpath,
+ )
+
+ # Remove .git dir
+ shutil.rmtree(os.path.join(fullpath, ".git"))
+
+ self._rebuild_git(fullpath)
+
+ def init_workspace(self, directory):
+ fullpath = os.path.join(directory, self.path)
+ url = self.source.translate_url(self.url)
+
+ self.source.call(
+ [
+ self.source.host_git,
+ "clone",
+ "--no-checkout",
+ self.mirror,
+ fullpath,
+ ],
+ fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
+ fail_temporarily=True,
+ )
+
+ self.source.call(
+ [self.source.host_git, "remote", "set-url", "origin", url],
+ fail='Failed to add remote origin "{}"'.format(url),
+ cwd=fullpath,
+ )
+
+ self.source.call(
+ [self.source.host_git, "checkout", "--force", self.ref],
+ fail="Failed to checkout git ref {}".format(self.ref),
+ cwd=fullpath,
+ )
+
+ # get_submodule_mirrors():
+ #
+ # Returns:
+ # An iterator through new instances of this class, one of each submodule
+ # in the repo
+ #
+ def get_submodule_mirrors(self):
+ for path, url in self.submodule_list():
+ ref = self.submodule_ref(path)
+ if ref is not None:
+ mirror = self.__class__(self.source, os.path.join(self.path, path), url, ref)
+ yield mirror
+
+ # List the submodules (path/url tuples) present at the given ref of this repo
+ def submodule_list(self):
+ modules = "{}:{}".format(self.ref, GIT_MODULES)
+ exit_code, output = self.source.check_output([self.source.host_git, "show", modules], cwd=self.mirror)
+
+ # If git show reports error code 128 here, we take it to mean there is
+ # no .gitmodules file to display for the given revision.
+ if exit_code == 128:
+ return
+ elif exit_code != 0:
+ raise SourceError("{plugin}: Failed to show gitmodules at ref {ref}".format(plugin=self, ref=self.ref))
+
+ content = "\n".join([l.strip() for l in output.splitlines()])
+
+ io = StringIO(content)
+ parser = RawConfigParser()
+ parser.read_file(io)
+
+ for section in parser.sections():
+ # validate section name against the 'submodule "foo"' pattern
+ if re.match(r'submodule "(.*)"', section):
+ path = parser.get(section, "path")
+ url = parser.get(section, "url")
+
+ yield (path, url)
+
+ # Fetch the ref which this mirror requires its submodule to have,
+ # at the given ref of this mirror.
+ def submodule_ref(self, submodule, ref=None):
+ if not ref:
+ ref = self.ref
+
+ # list objects in the parent repo tree to find the commit
+ # object that corresponds to the submodule
+ _, output = self.source.check_output(
+ [self.source.host_git, "ls-tree", ref, submodule],
+ fail="ls-tree failed for commit {} and submodule: {}".format(ref, submodule),
+ cwd=self.mirror,
+ )
+
+ # read the commit hash from the output
+ fields = output.split()
+ if len(fields) >= 2 and fields[1] == "commit":
+ submodule_commit = output.split()[2]
+
+ # fail if the commit hash is invalid
+ if len(submodule_commit) != 40:
+ raise SourceError(
+ "{}: Error reading commit information for submodule '{}'".format(self.source, submodule)
+ )
+
+ return submodule_commit
+
+ else:
+ detail = (
+ "The submodule '{}' is defined either in the BuildStream source\n".format(submodule)
+ + "definition, or in a .gitmodules file. But the submodule was never added to the\n"
+ + "underlying git repository with `git submodule add`."
+ )
+
+ self.source.warn(
+ "{}: Ignoring inconsistent submodule '{}'".format(self.source, submodule),
+ detail=detail,
+ warning_token=WARN_INCONSISTENT_SUBMODULE,
+ )
+
+ return None
+
+ def _rebuild_git(self, fullpath):
+ if not self.tags:
+ return
+
+ with self.source.tempdir() as tmpdir:
+ included = set()
+ shallow = set()
+ for _, commit_ref, _ in self.tags:
+
+ if commit_ref == self.ref:
+ # rev-list does not work in case of same rev
+ shallow.add(self.ref)
+ else:
+ _, out = self.source.check_output(
+ [
+ self.source.host_git,
+ "rev-list",
+ "--ancestry-path",
+ "--boundary",
+ "{}..{}".format(commit_ref, self.ref),
+ ],
+ fail="Failed to get git history {}..{} in directory: {}".format(
+ commit_ref, self.ref, fullpath
+ ),
+ fail_temporarily=True,
+ cwd=self.mirror,
+ )
+ self.source.warn("refs {}..{}: {}".format(commit_ref, self.ref, out.splitlines()))
+ for line in out.splitlines():
+ rev = line.lstrip("-")
+ if line[0] == "-":
+ shallow.add(rev)
+ else:
+ included.add(rev)
+
+ shallow -= included
+ included |= shallow
+
+ self.source.call(
+ [self.source.host_git, "init"],
+ fail="Cannot initialize git repository: {}".format(fullpath),
+ cwd=fullpath,
+ )
+
+ for rev in included:
+ with TemporaryFile(dir=tmpdir) as commit_file:
+ self.source.call(
+ [self.source.host_git, "cat-file", "commit", rev],
+ stdout=commit_file,
+ fail="Failed to get commit {}".format(rev),
+ cwd=self.mirror,
+ )
+ commit_file.seek(0, 0)
+ self.source.call(
+ [
+ self.source.host_git,
+ "hash-object",
+ "-w",
+ "-t",
+ "commit",
+ "--stdin",
+ ],
+ stdin=commit_file,
+ fail="Failed to add commit object {}".format(rev),
+ cwd=fullpath,
+ )
+
+ with open(
+ os.path.join(fullpath, ".git", "shallow"),
+ "w",
+ encoding="utf-8",
+ ) as shallow_file:
+ for rev in shallow:
+ shallow_file.write("{}\n".format(rev))
+
+ for tag, commit_ref, annotated in self.tags:
+ if annotated:
+ with TemporaryFile(dir=tmpdir) as tag_file:
+ tag_data = "object {}\ntype commit\ntag {}\n".format(commit_ref, tag)
+ tag_file.write(tag_data.encode("ascii"))
+ tag_file.seek(0, 0)
+ _, tag_ref = self.source.check_output(
+ [
+ self.source.host_git,
+ "hash-object",
+ "-w",
+ "-t",
+ "tag",
+ "--stdin",
+ ],
+ stdin=tag_file,
+ fail="Failed to add tag object {}".format(tag),
+ cwd=fullpath,
+ )
+
+ self.source.call(
+ [self.source.host_git, "tag", tag, tag_ref.strip()],
+ fail="Failed to tag: {}".format(tag),
+ cwd=fullpath,
+ )
+ else:
+ self.source.call(
+ [self.source.host_git, "tag", tag, commit_ref],
+ fail="Failed to tag: {}".format(tag),
+ cwd=fullpath,
+ )
+
+ with open(os.path.join(fullpath, ".git", "HEAD"), "w", encoding="utf-8") as head:
+ self.source.call(
+ [self.source.host_git, "rev-parse", self.ref],
+ stdout=head,
+ fail="Failed to parse commit {}".format(self.ref),
+ cwd=self.mirror,
+ )
+
+
+class GitSource(Source):
+ # pylint: disable=attribute-defined-outside-init
+
+ BST_MIN_VERSION = "2.0"
+
+ def configure(self, node):
+ ref = node.get_str("ref", None)
+
+ config_keys = [
+ "url",
+ "track",
+ "ref",
+ "submodules",
+ "checkout-submodules",
+ "ref-format",
+ "track-tags",
+ "tags",
+ ]
+ node.validate_keys(config_keys + Source.COMMON_CONFIG_KEYS)
+
+ tags_node = node.get_sequence("tags", [])
+ for tag_node in tags_node:
+ tag_node.validate_keys(["tag", "commit", "annotated"])
+
+ tags = self._load_tags(node)
+ self.track_tags = node.get_bool("track-tags", default=False)
+
+ self.original_url = node.get_str("url")
+ self.mirror = GitMirror(self, "", self.original_url, ref, tags=tags, primary=True)
+ self.tracking = node.get_str("track", None)
+
+ self.ref_format = node.get_enum("ref-format", _RefFormat, _RefFormat.SHA1)
+
+ # At this point we now know if the source has a ref and/or a track.
+ # If it is missing both then we will be unable to track or build.
+ if self.mirror.ref is None and self.tracking is None:
+ raise SourceError(
+ "{}: Git sources require a ref and/or track".format(self),
+ reason="missing-track-and-ref",
+ )
+
+ self.checkout_submodules = node.get_bool("checkout-submodules", default=True)
+
+ # Parse a dict of submodule overrides, stored in the submodule_overrides
+ # and submodule_checkout_overrides dictionaries.
+ self.submodule_overrides = {}
+ self.submodule_checkout_overrides = {}
+ modules = node.get_mapping("submodules", {})
+ for path in modules.keys():
+ submodule = modules.get_mapping(path)
+ url = submodule.get_str("url", None)
+
+ # Make sure to mark all URLs that are specified in the configuration
+ if url:
+ self.mark_download_url(url, primary=False)
+
+ self.submodule_overrides[path] = url
+ if "checkout" in submodule:
+ checkout = submodule.get_bool("checkout")
+ self.submodule_checkout_overrides[path] = checkout
+
+ self.mark_download_url(self.original_url)
+
+ def preflight(self):
+ # Check if git is installed, get the binary at the same time
+ self.host_git = utils.get_host_tool("git")
+
+ rc, version_str = self.check_output([self.host_git, "--version"])
+ # e.g. on Git for Windows we get "git version 2.21.0.windows.1".
+ # e.g. on Mac via Homebrew we get "git version 2.19.0".
+ if rc == 0:
+ self.host_git_version = tuple(int(x) for x in version_str.split(" ")[2].split(".")[:3])
+ else:
+ self.host_git_version = None
+
+ def get_unique_key(self):
+ ref = self.mirror.ref
+ if ref is not None:
+ # Strip any (arbitary) tag information, leaving just the commit ID
+ ref = _strip_tag(ref)
+
+ # Here we want to encode the local name of the repository and
+ # the ref, if the user changes the alias to fetch the same sources
+ # from another location, it should not affect the cache key.
+ key = [self.original_url, ref]
+ if self.mirror.tags:
+ tags = {tag: (commit, annotated) for tag, commit, annotated in self.mirror.tags}
+ key.append({"tags": tags})
+
+ # Only modify the cache key with checkout_submodules if it's something
+ # other than the default behaviour.
+ if self.checkout_submodules is False:
+ key.append({"checkout_submodules": self.checkout_submodules})
+
+ # We want the cache key to change if the source was
+ # configured differently, and submodules count.
+ if self.submodule_overrides:
+ key.append(self.submodule_overrides)
+
+ if self.submodule_checkout_overrides:
+ key.append({"submodule_checkout_overrides": self.submodule_checkout_overrides})
+
+ return key
+
+ def is_resolved(self):
+ return self.mirror.ref is not None
+
+ def is_cached(self):
+ return self._have_all_refs()
+
+ def load_ref(self, node):
+ self.mirror.ref = node.get_str("ref", None)
+ self.mirror.tags = self._load_tags(node)
+
+ def get_ref(self):
+ if self.mirror.ref is None:
+ return None
+ return self.mirror.ref, self.mirror.tags
+
+ def set_ref(self, ref, node):
+ if not ref:
+ self.mirror.ref = None
+ if "ref" in node:
+ del node["ref"]
+ self.mirror.tags = []
+ if "tags" in node:
+ del node["tags"]
+ else:
+ actual_ref, tags = ref
+ node["ref"] = self.mirror.ref = actual_ref
+ self.mirror.tags = tags
+ if tags:
+ node["tags"] = []
+ for tag, commit_ref, annotated in tags:
+ data = {
+ "tag": tag,
+ "commit": commit_ref,
+ "annotated": annotated,
+ }
+ node["tags"].append(data)
+ else:
+ if "tags" in node:
+ del node["tags"]
+
+ def track(self): # pylint: disable=arguments-differ
+
+ # If self.tracking is not specified it's not an error, just silently return
+ if not self.tracking:
+ # Is there a better way to check if a ref is given.
+ if self.mirror.ref is None:
+ detail = "Without a tracking branch ref can not be updated. Please " + "provide a ref or a track."
+ raise SourceError(
+ "{}: No track or ref".format(self),
+ detail=detail,
+ reason="track-attempt-no-track",
+ )
+ return None
+
+ # Resolve the URL for the message
+ resolved_url = self.translate_url(self.mirror.url)
+ with self.timed_activity(
+ "Tracking {} from {}".format(self.tracking, resolved_url),
+ silent_nested=True,
+ ):
+ self.mirror._fetch(resolved_url, fetch_all=True)
+
+ ref = self.mirror.to_commit(self.tracking)
+ tags = self.mirror.reachable_tags(ref) if self.track_tags else []
+
+ if self.ref_format == _RefFormat.GIT_DESCRIBE:
+ ref = self.mirror.describe(ref)
+
+ return ref, tags
+
+ def init_workspace(self, directory):
+ with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
+ self.mirror.init_workspace(directory)
+ for mirror in self._recurse_submodules(configure=True):
+ mirror.init_workspace(directory)
+
+ def stage(self, directory):
+ # Stage the main repo in the specified directory
+ #
+ with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
+ self.mirror.stage(directory)
+ for mirror in self._recurse_submodules(configure=True):
+ mirror.stage(directory)
+
+ def get_source_fetchers(self):
+ self.mirror.mark_download_url(self.mirror.url)
+ yield self.mirror
+ # _recurse_submodules only iterates those which are known at the current
+ # cached state - but fetch is called on each result as we go, so this will
+ # yield all configured submodules
+ for submodule in self._recurse_submodules(configure=True):
+ submodule.mark_download_url(submodule.url)
+ yield submodule
+
+ def validate_cache(self):
+ discovered_submodules = {}
+ unlisted_submodules = []
+ invalid_submodules = []
+
+ for submodule in self._recurse_submodules(configure=False):
+ discovered_submodules[submodule.path] = submodule.url
+ if self._ignoring_submodule(submodule.path):
+ continue
+
+ if submodule.path not in self.submodule_overrides:
+ unlisted_submodules.append((submodule.path, submodule.url))
+
+ # Warn about submodules which are explicitly configured but do not exist
+ for path, url in self.submodule_overrides.items():
+ if path not in discovered_submodules:
+ invalid_submodules.append((path, url))
+
+ if invalid_submodules:
+ detail = []
+ for path, url in invalid_submodules:
+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
+
+ self.warn(
+ "{}: Invalid submodules specified".format(self),
+ warning_token=WARN_INVALID_SUBMODULE,
+ detail="The following submodules are specified in the source "
+ "description but do not exist according to the repository\n\n" + "\n".join(detail),
+ )
+
+ # Warn about submodules which exist but have not been explicitly configured
+ if unlisted_submodules:
+ detail = []
+ for path, url in unlisted_submodules:
+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
+
+ self.warn(
+ "{}: Unlisted submodules exist".format(self),
+ warning_token=WARN_UNLISTED_SUBMODULE,
+ detail="The following submodules exist but are not specified "
+ + "in the source description\n\n"
+ + "\n".join(detail),
+ )
+
+ # Assert that the ref exists in the track tag/branch, if track has been specified.
+ # Also don't do this check if an exact tag ref is given, as we probably didn't fetch
+ # any branch refs
+ ref_in_track = False
+ if not re.match(EXACT_TAG_PATTERN, self.mirror.ref) and self.tracking:
+ _, branch = self.check_output(
+ [
+ self.host_git,
+ "branch",
+ "--list",
+ self.tracking,
+ "--contains",
+ self.mirror.ref,
+ ],
+ cwd=self.mirror.mirror,
+ )
+ if branch:
+ ref_in_track = True
+ else:
+ _, tag = self.check_output(
+ [
+ self.host_git,
+ "tag",
+ "--list",
+ self.tracking,
+ "--contains",
+ self.mirror.ref,
+ ],
+ cwd=self.mirror.mirror,
+ )
+ if tag:
+ ref_in_track = True
+
+ if not ref_in_track:
+ detail = (
+ "The ref provided for the element does not exist locally "
+ + "in the provided track branch / tag '{}'.\n".format(self.tracking)
+ + "You may wish to track the element to update the ref from '{}' ".format(self.tracking)
+ + "with `bst source track`,\n"
+ + "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
+ )
+
+ self.warn(
+ "{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n".format(
+ self, self.mirror.ref, self.tracking, self.mirror.url
+ ),
+ detail=detail,
+ warning_token=CoreWarnings.REF_NOT_IN_TRACK,
+ )
+
+ ###########################################################
+ # Local Functions #
+ ###########################################################
+
+ def _have_all_refs(self):
+ return self.mirror.has_ref() and all(
+ submodule.has_ref() for submodule in self._recurse_submodules(configure=True)
+ )
+
+ # _configure_submodules():
+ #
+ # Args:
+ # submodules: An iterator of GitMirror (or similar) objects for submodules
+ #
+ # Returns:
+ # An iterator through `submodules` but filtered of any ignored submodules
+ # and modified to use any custom URLs configured in the source
+ #
+ def _configure_submodules(self, submodules):
+ for submodule in submodules:
+ if self._ignoring_submodule(submodule.path):
+ continue
+ # Allow configuration to override the upstream location of the submodules.
+ submodule.url = self.submodule_overrides.get(submodule.path, submodule.url)
+ yield submodule
+
+ # _recurse_submodules():
+ #
+ # Recursively iterates through GitMirrors for submodules of the main repo. Only
+ # submodules that are cached are recursed into - but this is decided at
+ # iteration time, so you can fetch in a for loop over this function to fetch
+ # all submodules.
+ #
+ # Args:
+ # configure (bool): Whether to apply the 'submodule' config while recursing
+ # (URL changing and 'checkout' overrides)
+ #
+ def _recurse_submodules(self, configure):
+ def recurse(mirror):
+ submodules = mirror.get_submodule_mirrors()
+ if configure:
+ submodules = self._configure_submodules(submodules)
+
+ for submodule in submodules:
+ yield submodule
+ if submodule.has_ref():
+ yield from recurse(submodule)
+
+ yield from recurse(self.mirror)
+
+ def _load_tags(self, node):
+ tags = []
+ tags_node = node.get_sequence("tags", [])
+ for tag_node in tags_node:
+ tag = tag_node.get_str("tag")
+ commit_ref = tag_node.get_str("commit")
+ annotated = tag_node.get_bool("annotated")
+ tags.append((tag, commit_ref, annotated))
+ return tags
+
+ # _ignoring_submodule():
+ #
+ # Args:
+ # path (str): The path of a submodule in the superproject
+ #
+ # Returns:
+ # (bool): Whether to not clone/checkout this submodule
+ #
+ def _ignoring_submodule(self, path):
+ return not self.submodule_checkout_overrides.get(path, self.checkout_submodules)
+
+
+# Plugin entry point
+def setup():
+ return GitSource
[buildstream] 34/37: tests/sources/git.py: Remove git source tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit fda974b39a9c08fd7c8dd03f94f17d07f6230d22
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:35:36 2022 +0900
tests/sources/git.py: Remove git source tests
---
tests/sources/git.py | 1145 --------------------
tests/sources/git/project-override/project.conf | 12 -
.../git/project-override/repofiles/file.txt | 1 -
.../git/project-override/subrepofiles/ponyfile.txt | 1 -
.../template/inconsistent-submodule/.gitmodules | 3 -
.../git/template/othersubrepofiles/unicornfile.txt | 1 -
tests/sources/git/template/project.conf | 3 -
tests/sources/git/template/repofiles/file.txt | 1 -
.../sources/git/template/subrepofiles/ponyfile.txt | 1 -
9 files changed, 1168 deletions(-)
diff --git a/tests/sources/git.py b/tests/sources/git.py
deleted file mode 100644
index 8cb5af4..0000000
--- a/tests/sources/git.py
+++ /dev/null
@@ -1,1145 +0,0 @@
-#
-# Copyright (C) 2018 Codethink Limited
-# Copyright (C) 2018 Bloomberg Finance LP
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Authors: Tristan Van Berkom <tr...@codethink.co.uk>
-# Jonathan Maw <jo...@codethink.co.uk>
-# William Salmon <wi...@codethink.co.uk>
-#
-
-# Pylint doesn't play well with fixtures and dependency injection from pytest
-# pylint: disable=redefined-outer-name
-
-import os
-import subprocess
-import shutil
-
-import pytest
-
-from buildstream import Node
-from buildstream.exceptions import ErrorDomain
-from buildstream.plugin import CoreWarnings
-from buildstream._testing import cli # pylint: disable=unused-import
-from buildstream._testing import generate_project, generate_element, load_yaml
-from buildstream._testing import create_repo
-from buildstream._testing._utils.site import HAVE_GIT, HAVE_OLD_GIT
-
-DATA_DIR = os.path.join(
- os.path.dirname(os.path.realpath(__file__)),
- "git",
-)
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_fetch_bad_ref(cli, tmpdir, datafiles):
- project = str(datafiles)
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Write out our test target with a bad ref
- element = {"kind": "import", "sources": [repo.source_config(ref="5")]}
- generate_element(project, "target.bst", element)
-
- # Assert that fetch raises an error here
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.SOURCE, None)
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.skipif(HAVE_OLD_GIT, reason="old git cannot clone a shallow repo to stage the source")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_fetch_shallow(cli, tmpdir, datafiles):
- project = str(datafiles)
- workspacedir = os.path.join(str(tmpdir), "workspace")
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
- first_commit = repo.latest_commit()
- repo.add_commit()
- repo.add_tag("tag")
-
- ref = "tag-0-g" + repo.latest_commit()
-
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["workspace", "open", "--directory", workspacedir, "target.bst"])
- result.assert_success()
-
- assert subprocess.call(["git", "show", "tag"], cwd=workspacedir) == 0
- assert subprocess.call(["git", "show", first_commit], cwd=workspacedir) != 0
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_checkout(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out both files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_recursive_submodule_fetch_checkout(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create a submodule from the 'othersubrepofiles' subdir
- subsubrepo = create_repo("git", str(tmpdir), "subsubrepo")
- subsubrepo.create(os.path.join(project, "othersubrepofiles"))
-
- # Create another submodule from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Configure submodules
- subrepo.add_submodule("subdir", "file://" + subsubrepo.repo)
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out all files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "subdir", "subdir", "unicornfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_source_enable_explicit(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config_extra(ref=ref, checkout_submodules=True)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out both files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_source_disable(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config_extra(ref=ref, checkout_submodules=False)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out both files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert not os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_submodule_does_override(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo, checkout=True)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config_extra(ref=ref, checkout_submodules=False)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out both files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_submodule_individual_checkout(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create another submodule from the 'othersubrepofiles' subdir
- other_subrepo = create_repo("git", str(tmpdir), "othersubrepo")
- other_subrepo.create(os.path.join(project, "othersubrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- repo.add_submodule("subdir", "file://" + subrepo.repo, checkout=False)
- ref = repo.add_submodule("othersubdir", "file://" + other_subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config_extra(ref=ref, checkout_submodules=True)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert not os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "othersubdir", "unicornfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_fetch_submodule_individual_checkout_explicit(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create another submodule from the 'othersubrepofiles' subdir
- other_subrepo = create_repo("git", str(tmpdir), "othersubrepo")
- other_subrepo.create(os.path.join(project, "othersubrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- repo.add_submodule("subdir", "file://" + subrepo.repo, checkout=False)
- ref = repo.add_submodule("othersubdir", "file://" + other_subrepo.repo, checkout=True)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config_extra(ref=ref, checkout_submodules=True)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert not os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
- assert os.path.exists(os.path.join(checkoutdir, "othersubdir", "unicornfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "project-override"))
-def test_submodule_fetch_project_override(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- result.assert_success()
-
- # Assert we checked out both files at their expected location
- assert os.path.exists(os.path.join(checkoutdir, "file.txt"))
- assert not os.path.exists(os.path.join(checkoutdir, "subdir", "ponyfile.txt"))
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_track_ignore_inconsistent(cli, tmpdir, datafiles):
- project = str(datafiles)
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- ref = repo.create(os.path.join(project, "repofiles"))
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- # Now add a .gitmodules file with an inconsistent submodule,
- # we are calling this inconsistent because the file was created
- # but `git submodule add` was never called, so there is no reference
- # associated to the submodule.
- #
- repo.add_file(os.path.join(project, "inconsistent-submodule", ".gitmodules"))
-
- # Fetch should work, we're not yet at the offending ref
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
-
- # Track to update to the offending commit
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
-
- # Fetch after track will encounter an inconsistent submodule without any ref
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
-
- # Assert that we are just fine without it, and emit a warning to the user.
- assert "Ignoring inconsistent submodule" in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
- project = str(datafiles)
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Write out our test target
- gitsource = repo.source_config(ref=None)
- gitsource.pop("track")
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- # Track will encounter an inconsistent submodule without any ref
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
- result.assert_task_error(None, None)
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("fail", ["warn", "error"])
-def test_ref_not_in_track(cli, tmpdir, datafiles, fail):
- project = str(datafiles)
-
- # Make the warning an error if we're testing errors
- if fail == "error":
- generate_project(project, config={"fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]})
-
- # Create the repo from 'repofiles', create a branch without latest commit
- repo = create_repo("git", str(tmpdir))
- ref = repo.create(os.path.join(project, "repofiles"))
-
- gitsource = repo.source_config(ref=ref)
-
- # Overwrite the track value to the added branch
- gitsource["track"] = "foo"
-
- # Write out our test target
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- result = cli.run(project=project, args=["build", "target.bst"])
-
- # Assert a warning or an error depending on what we're checking
- if fail == "error":
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
- else:
- result.assert_success()
- assert "ref-not-in-track" in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("fail", ["warn", "error"])
-def test_unlisted_submodule(cli, tmpdir, datafiles, fail):
- project = str(datafiles)
-
- # Make the warning an error if we're testing errors
- if fail == "error":
- generate_project(project, config={"fatal-warnings": ["git:unlisted-submodule"]})
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Create the source, and delete the explicit configuration
- # of the submodules.
- #
- # We expect this to cause an unlisted submodule warning
- # after the source has been fetched.
- #
- gitsource = repo.source_config(ref=ref)
- del gitsource["submodules"]
-
- # Write out our test target
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- # The warning or error is reported during fetch. There should be no
- # error with `bst show`.
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_success()
- assert "git:unlisted-submodule" not in result.stderr
-
- # We will notice this directly in fetch, as it will try to fetch
- # the submodules it discovers as a result of fetching the primary repo.
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
-
- # Assert a warning or an error depending on what we're checking
- if fail == "error":
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.PLUGIN, "git:unlisted-submodule")
- else:
- result.assert_success()
- assert "git:unlisted-submodule" in result.stderr
-
- # Verify that `bst show` will still not error out after fetching.
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_success()
- assert "git:unlisted-submodule" not in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("fail", ["warn", "error"])
-def test_track_unlisted_submodule(cli, tmpdir, datafiles, fail):
- project = str(datafiles)
-
- # Make the warning an error if we're testing errors
- if fail == "error":
- generate_project(project, config={"fatal-warnings": ["git:unlisted-submodule"]})
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- ref = repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created, but use
- # the original ref, let the submodules appear after tracking
- repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Create the source, and delete the explicit configuration
- # of the submodules.
- gitsource = repo.source_config(ref=ref)
- del gitsource["submodules"]
-
- # Write out our test target
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- # Fetch the repo, we will not see the warning because we
- # are still pointing to a ref which predates the submodules
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- assert "git:unlisted-submodule" not in result.stderr
-
- # We won't get a warning/error when tracking either, the source
- # has not become cached so the opportunity to check
- # for the warning has not yet arisen.
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
- assert "git:unlisted-submodule" not in result.stderr
-
- # Fetching the repo at the new ref will finally reveal the warning
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- if fail == "error":
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.PLUGIN, "git:unlisted-submodule")
- else:
- result.assert_success()
- assert "git:unlisted-submodule" in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("fail", ["warn", "error"])
-def test_invalid_submodule(cli, tmpdir, datafiles, fail):
- project = str(datafiles)
-
- # Make the warning an error if we're testing errors
- if fail == "error":
- generate_project(project, config={"fatal-warnings": ["git:invalid-submodule"]})
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- ref = repo.create(os.path.join(project, "repofiles"))
-
- # Create the source without any submodules, and add
- # an invalid submodule configuration to it.
- #
- # We expect this to cause an invalid submodule warning
- # after the source has been fetched and we know what
- # the real submodules actually are.
- #
- gitsource = repo.source_config(ref=ref)
- gitsource["submodules"] = {"subdir": {"url": "https://pony.org/repo.git"}}
-
- # Write out our test target
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- # The warning or error is reported during fetch. There should be no
- # error with `bst show`.
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_success()
- assert "git:invalid-submodule" not in result.stderr
-
- # We will notice this directly in fetch, as it will try to fetch
- # the submodules it discovers as a result of fetching the primary repo.
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
-
- # Assert a warning or an error depending on what we're checking
- if fail == "error":
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.PLUGIN, "git:invalid-submodule")
- else:
- result.assert_success()
- assert "git:invalid-submodule" in result.stderr
-
- # Verify that `bst show` will still not error out after fetching.
- result = cli.run(project=project, args=["show", "target.bst"])
- result.assert_success()
- assert "git:invalid-submodule" not in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.skipif(HAVE_OLD_GIT, reason="old git rm does not update .gitmodules")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("fail", ["warn", "error"])
-def test_track_invalid_submodule(cli, tmpdir, datafiles, fail):
- project = str(datafiles)
-
- # Make the warning an error if we're testing errors
- if fail == "error":
- generate_project(project, config={"fatal-warnings": ["git:invalid-submodule"]})
-
- # Create the submodule first from the 'subrepofiles' subdir
- subrepo = create_repo("git", str(tmpdir), "subrepo")
- subrepo.create(os.path.join(project, "subrepofiles"))
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
-
- # Add a submodule pointing to the one we created
- ref = repo.add_submodule("subdir", "file://" + subrepo.repo)
-
- # Add a commit beyond the ref which *removes* the submodule we've added
- repo.remove_path("subdir")
-
- # Create the source, this will keep the submodules so initially
- # the configuration is valid for the ref we're using
- gitsource = repo.source_config(ref=ref)
-
- # Write out our test target
- element = {"kind": "import", "sources": [gitsource]}
- generate_element(project, "target.bst", element)
-
- # Fetch the repo, we will not see the warning because we
- # are still pointing to a ref which predates the submodules
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
- assert "git:invalid-submodule" not in result.stderr
-
- # After tracking we're pointing to a ref, which would trigger an invalid
- # submodule warning. However, cache validation is only performed as part
- # of fetch.
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
-
- # Fetch to trigger cache validation
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- if fail == "error":
- result.assert_main_error(ErrorDomain.STREAM, None)
- result.assert_task_error(ErrorDomain.PLUGIN, "git:invalid-submodule")
- else:
- result.assert_success()
- assert "git:invalid-submodule" in result.stderr
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("ref_format", ["sha1", "git-describe"])
-@pytest.mark.parametrize("tag,extra_commit", [(False, False), (True, False), (True, True)])
-def test_track_fetch(cli, tmpdir, datafiles, ref_format, tag, extra_commit):
- project = str(datafiles)
-
- # Create the repo from 'repofiles' subdir
- repo = create_repo("git", str(tmpdir))
- repo.create(os.path.join(project, "repofiles"))
- if tag:
- repo.add_tag("tag")
- if extra_commit:
- repo.add_commit()
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config()]}
- element["sources"][0]["ref-format"] = ref_format
- generate_element(project, "target.bst", element)
- element_path = os.path.join(project, "target.bst")
-
- # Track it
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
-
- element = load_yaml(element_path)
- new_ref = element.get_sequence("sources").mapping_at(0).get_str("ref")
-
- if ref_format == "git-describe" and tag:
- # Check and strip prefix
- prefix = "tag-{}-g".format(0 if not extra_commit else 1)
- assert new_ref.startswith(prefix)
- new_ref = new_ref[len(prefix) :]
-
- # 40 chars for SHA-1
- assert len(new_ref) == 40
-
- # Fetch it
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- result.assert_success()
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.skipif(HAVE_OLD_GIT, reason="old git describe lacks --first-parent")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
-@pytest.mark.parametrize("tag_type", [("annotated"), ("lightweight")])
-def test_git_describe(cli, tmpdir, datafiles, ref_storage, tag_type):
- project = str(datafiles)
-
- project_config = load_yaml(os.path.join(project, "project.conf"))
- project_config["ref-storage"] = ref_storage
- generate_project(project, config=project_config)
-
- repofiles = os.path.join(str(tmpdir), "repofiles")
- os.makedirs(repofiles, exist_ok=True)
- file0 = os.path.join(repofiles, "file0")
- with open(file0, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- repo = create_repo("git", str(tmpdir))
-
- def tag(name):
- if tag_type == "annotated":
- repo.add_annotated_tag(name, name)
- else:
- repo.add_tag(name)
-
- repo.create(repofiles)
- tag("uselesstag")
-
- file1 = os.path.join(str(tmpdir), "file1")
- with open(file1, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file1)
- tag("tag1")
-
- file2 = os.path.join(str(tmpdir), "file2")
- with open(file2, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.branch("branch2")
- repo.add_file(file2)
- tag("tag2")
-
- repo.checkout("master")
- file3 = os.path.join(str(tmpdir), "file3")
- with open(file3, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file3)
-
- repo.merge("branch2")
-
- config = repo.source_config()
- config["track"] = repo.latest_commit()
- config["track-tags"] = True
-
- # Write out our test target
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
- element_path = os.path.join(project, "target.bst")
-
- if ref_storage == "inline":
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
- else:
- result = cli.run(project=project, args=["source", "track", "target.bst", "--deps", "all"])
- result.assert_success()
-
- if ref_storage == "inline":
- element = load_yaml(element_path)
- tags = element.get_sequence("sources").mapping_at(0).get_sequence("tags")
- assert len(tags) == 2
- for tag in tags:
- assert "tag" in tag
- assert "commit" in tag
- assert "annotated" in tag
- assert tag.get_bool("annotated") == (tag_type == "annotated")
-
- assert {(tag.get_str("tag"), tag.get_str("commit")) for tag in tags} == {
- ("tag1", repo.rev_parse("tag1^{commit}")),
- ("tag2", repo.rev_parse("tag2^{commit}")),
- }
-
- checkout = os.path.join(str(tmpdir), "checkout")
-
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkout])
- result.assert_success()
-
- if tag_type == "annotated":
- options = []
- else:
- options = ["--tags"]
- describe = subprocess.check_output(["git", "describe", *options], cwd=checkout, universal_newlines=True)
- assert describe.startswith("tag2-2-")
-
- describe_fp = subprocess.check_output(
- ["git", "describe", "--first-parent", *options], cwd=checkout, universal_newlines=True
- )
- assert describe_fp.startswith("tag1-2-")
-
- tags = subprocess.check_output(["git", "tag"], cwd=checkout, universal_newlines=True)
- tags = set(tags.splitlines())
- assert tags == set(["tag1", "tag2"])
-
- with pytest.raises(subprocess.CalledProcessError):
- subprocess.run(["git", "log", repo.rev_parse("uselesstag")], cwd=checkout, check=True)
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
-@pytest.mark.parametrize("tag_type", [("annotated"), ("lightweight")])
-def test_git_describe_head_is_tagged(cli, tmpdir, datafiles, ref_storage, tag_type):
- project = str(datafiles)
-
- project_config = load_yaml(os.path.join(project, "project.conf"))
- project_config["ref-storage"] = ref_storage
- generate_project(project, config=project_config)
-
- repofiles = os.path.join(str(tmpdir), "repofiles")
- os.makedirs(repofiles, exist_ok=True)
- file0 = os.path.join(repofiles, "file0")
- with open(file0, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- repo = create_repo("git", str(tmpdir))
-
- def tag(name):
- if tag_type == "annotated":
- repo.add_annotated_tag(name, name)
- else:
- repo.add_tag(name)
-
- repo.create(repofiles)
- tag("uselesstag")
-
- file1 = os.path.join(str(tmpdir), "file1")
- with open(file1, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file1)
-
- file2 = os.path.join(str(tmpdir), "file2")
- with open(file2, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.branch("branch2")
- repo.add_file(file2)
-
- repo.checkout("master")
- file3 = os.path.join(str(tmpdir), "file3")
- with open(file3, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file3)
-
- tagged_ref = repo.merge("branch2")
- tag("tag")
-
- config = repo.source_config()
- config["track"] = repo.latest_commit()
- config["track-tags"] = True
-
- # Write out our test target
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
- element_path = os.path.join(project, "target.bst")
-
- if ref_storage == "inline":
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
- else:
- result = cli.run(project=project, args=["source", "track", "target.bst", "--deps", "all"])
- result.assert_success()
-
- if ref_storage == "inline":
- element = load_yaml(element_path)
- source = element.get_sequence("sources").mapping_at(0)
- tags = source.get_sequence("tags")
- assert len(tags) == 1
-
- tag = source.get_sequence("tags").mapping_at(0)
- assert "tag" in tag
- assert "commit" in tag
- assert "annotated" in tag
- assert tag.get_bool("annotated") == (tag_type == "annotated")
-
- tag_name = tag.get_str("tag")
- commit = tag.get_str("commit")
- assert (tag_name, commit) == ("tag", repo.rev_parse("tag^{commit}"))
-
- checkout = os.path.join(str(tmpdir), "checkout")
-
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkout])
- result.assert_success()
-
- if tag_type == "annotated":
- options = []
- else:
- options = ["--tags"]
- describe = subprocess.check_output(["git", "describe", *options], cwd=checkout, universal_newlines=True)
- assert describe.startswith("tag")
-
- tags = subprocess.check_output(["git", "tag"], cwd=checkout, universal_newlines=True)
- tags = set(tags.splitlines())
- assert tags == set(["tag"])
-
- rev_list = subprocess.check_output(["git", "rev-list", "--all"], cwd=checkout, universal_newlines=True)
-
- assert set(rev_list.splitlines()) == set([tagged_ref])
-
- with pytest.raises(subprocess.CalledProcessError):
- subprocess.run(["git", "log", repo.rev_parse("uselesstag")], cwd=checkout, check=True)
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_git_describe_relevant_history(cli, tmpdir, datafiles):
- project = str(datafiles)
-
- project_config = load_yaml(os.path.join(project, "project.conf"))
- project_config["ref-storage"] = "project.refs"
- generate_project(project, config=project_config)
-
- repofiles = os.path.join(str(tmpdir), "repofiles")
- os.makedirs(repofiles, exist_ok=True)
- file0 = os.path.join(repofiles, "file0")
- with open(file0, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- repo = create_repo("git", str(tmpdir))
- repo.create(repofiles)
-
- file1 = os.path.join(str(tmpdir), "file1")
- with open(file1, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file1)
- repo.branch("branch")
- repo.checkout("master")
-
- file2 = os.path.join(str(tmpdir), "file2")
- with open(file2, "w", encoding="utf-8") as f:
- f.write("test\n")
- repo.add_file(file2)
-
- file3 = os.path.join(str(tmpdir), "file3")
- with open(file3, "w", encoding="utf-8") as f:
- f.write("test\n")
- branch_boundary = repo.add_file(file3)
-
- repo.checkout("branch")
- file4 = os.path.join(str(tmpdir), "file4")
- with open(file4, "w", encoding="utf-8") as f:
- f.write("test\n")
- tagged_ref = repo.add_file(file4)
- repo.add_annotated_tag("tag1", "tag1")
-
- head = repo.merge("master")
-
- config = repo.source_config()
- config["track"] = head
- config["track-tags"] = True
-
- # Write out our test target
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
-
- result = cli.run(project=project, args=["source", "track", "target.bst", "--deps", "all"])
- result.assert_success()
-
- checkout = os.path.join(str(tmpdir), "checkout")
-
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkout])
- result.assert_success()
-
- describe = subprocess.check_output(["git", "describe"], cwd=checkout, universal_newlines=True)
- assert describe.startswith("tag1-2-")
-
- rev_list = subprocess.check_output(["git", "rev-list", "--all"], cwd=checkout, universal_newlines=True)
-
- assert set(rev_list.splitlines()) == set([head, tagged_ref, branch_boundary])
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_default_do_not_track_tags(cli, tmpdir, datafiles):
- project = str(datafiles)
-
- project_config = load_yaml(os.path.join(project, "project.conf"))
- project_config["ref-storage"] = "inline"
- generate_project(project, config=project_config)
-
- repofiles = os.path.join(str(tmpdir), "repofiles")
- os.makedirs(repofiles, exist_ok=True)
- file0 = os.path.join(repofiles, "file0")
- with open(file0, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- repo = create_repo("git", str(tmpdir))
-
- repo.create(repofiles)
- repo.add_tag("tag")
-
- config = repo.source_config()
- config["track"] = repo.latest_commit()
-
- # Write out our test target
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
- element_path = os.path.join(project, "target.bst")
-
- result = cli.run(project=project, args=["source", "track", "target.bst"])
- result.assert_success()
-
- element = load_yaml(element_path)
- source = element.get_sequence("sources").mapping_at(0)
- assert "tags" not in source
-
-
-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
-@pytest.mark.datafiles(os.path.join(DATA_DIR, "template"))
-def test_overwrite_rogue_tag_multiple_remotes(cli, tmpdir, datafiles):
- """When using multiple remotes in cache (i.e. when using aliases), we
- need to make sure we override tags. This is not allowed to fetch
- tags that were present from different origins
- """
-
- project = str(datafiles)
-
- repofiles = os.path.join(str(tmpdir), "repofiles")
- os.makedirs(repofiles, exist_ok=True)
- file0 = os.path.join(repofiles, "file0")
- with open(file0, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- repo = create_repo("git", str(tmpdir))
-
- top_commit = repo.create(repofiles)
-
- repodir, reponame = os.path.split(repo.repo)
- project_config = load_yaml(os.path.join(project, "project.conf"))
- project_config["aliases"] = Node.from_dict({"repo": "http://example.com/"})
- project_config["mirrors"] = [{"name": "middle-earth", "aliases": {"repo": ["file://{}/".format(repodir)]}}]
- generate_project(project, config=project_config)
-
- repo.add_annotated_tag("tag", "tag")
-
- file1 = os.path.join(repofiles, "file1")
- with open(file1, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- ref = repo.add_file(file1)
-
- config = repo.source_config(ref=ref)
- del config["track"]
- config["url"] = "repo:{}".format(reponame)
-
- # Write out our test target
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
-
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
-
- repo.checkout(top_commit)
-
- file2 = os.path.join(repofiles, "file2")
- with open(file2, "w", encoding="utf-8") as f:
- f.write("test\n")
-
- new_ref = repo.add_file(file2)
-
- repo.delete_tag("tag")
- repo.add_annotated_tag("tag", "tag")
- repo.checkout("master")
-
- otherpath = os.path.join(str(tmpdir), "other_path")
- shutil.copytree(repo.repo, os.path.join(otherpath, "repo"))
- create_repo("git", otherpath)
-
- repodir, reponame = os.path.split(repo.repo)
-
- generate_project(project, config=project_config)
-
- config = repo.source_config(ref=new_ref)
- del config["track"]
- config["url"] = "repo:{}".format(reponame)
-
- element = {
- "kind": "import",
- "sources": [config],
- }
- generate_element(project, "target.bst", element)
-
- result = cli.run(project=project, args=["build", "target.bst"])
- result.assert_success()
diff --git a/tests/sources/git/project-override/project.conf b/tests/sources/git/project-override/project.conf
deleted file mode 100644
index 01c9016..0000000
--- a/tests/sources/git/project-override/project.conf
+++ /dev/null
@@ -1,12 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
-sources:
- git:
- config:
- checkout-submodules: False
-elements:
- manual:
- config:
- build-commands:
- - "foo"
diff --git a/tests/sources/git/project-override/repofiles/file.txt b/tests/sources/git/project-override/repofiles/file.txt
deleted file mode 100644
index f621448..0000000
--- a/tests/sources/git/project-override/repofiles/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-pony
diff --git a/tests/sources/git/project-override/subrepofiles/ponyfile.txt b/tests/sources/git/project-override/subrepofiles/ponyfile.txt
deleted file mode 100644
index f73f309..0000000
--- a/tests/sources/git/project-override/subrepofiles/ponyfile.txt
+++ /dev/null
@@ -1 +0,0 @@
-file
diff --git a/tests/sources/git/template/inconsistent-submodule/.gitmodules b/tests/sources/git/template/inconsistent-submodule/.gitmodules
deleted file mode 100644
index 67271b8..0000000
--- a/tests/sources/git/template/inconsistent-submodule/.gitmodules
+++ /dev/null
@@ -1,3 +0,0 @@
-[submodule "farm/pony"]
- path = farm/pony
- url = git://pony.com
diff --git a/tests/sources/git/template/othersubrepofiles/unicornfile.txt b/tests/sources/git/template/othersubrepofiles/unicornfile.txt
deleted file mode 100644
index f73f309..0000000
--- a/tests/sources/git/template/othersubrepofiles/unicornfile.txt
+++ /dev/null
@@ -1 +0,0 @@
-file
diff --git a/tests/sources/git/template/project.conf b/tests/sources/git/template/project.conf
deleted file mode 100644
index dc34380..0000000
--- a/tests/sources/git/template/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic project
-name: foo
-min-version: 2.0
diff --git a/tests/sources/git/template/repofiles/file.txt b/tests/sources/git/template/repofiles/file.txt
deleted file mode 100644
index f621448..0000000
--- a/tests/sources/git/template/repofiles/file.txt
+++ /dev/null
@@ -1 +0,0 @@
-pony
diff --git a/tests/sources/git/template/subrepofiles/ponyfile.txt b/tests/sources/git/template/subrepofiles/ponyfile.txt
deleted file mode 100644
index f73f309..0000000
--- a/tests/sources/git/template/subrepofiles/ponyfile.txt
+++ /dev/null
@@ -1 +0,0 @@
-file
[buildstream] 07/37: tests/integration/pip_source.py: Removing pip source integration test
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 349a8fe000d3c0db00b003d6b6af85a3507c200d
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:33:26 2022 +0900
tests/integration/pip_source.py: Removing pip source integration test
---
tests/integration/pip_source.py | 177 ---------------------
tests/integration/project/files/pip-source/app1.py | 11 --
.../project/files/pip-source/myreqs.txt | 1 -
3 files changed, 189 deletions(-)
diff --git a/tests/integration/pip_source.py b/tests/integration/pip_source.py
deleted file mode 100644
index 083f844..0000000
--- a/tests/integration/pip_source.py
+++ /dev/null
@@ -1,177 +0,0 @@
-# Pylint doesn't play well with fixtures and dependency injection from pytest
-# pylint: disable=redefined-outer-name
-
-import os
-import pytest
-
-from buildstream import _yaml
-
-from buildstream._testing import cli_integration as cli # pylint: disable=unused-import
-from buildstream._testing.integration import assert_contains
-from buildstream._testing._utils.site import HAVE_SANDBOX
-
-from tests.testutils.python_repo import setup_pypi_repo # pylint: disable=unused-import
-
-
-pytestmark = pytest.mark.integration
-
-
-DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project")
-
-
-@pytest.mark.datafiles(DATA_DIR)
-def test_pip_source_import_packages(cli, datafiles, setup_pypi_repo):
- project = str(datafiles)
- checkout = os.path.join(cli.directory, "checkout")
- element_path = os.path.join(project, "elements")
- element_name = "pip/hello.bst"
-
- # check that exotically named packages are imported correctly
- myreqs_packages = "hellolib"
- dependencies = ["app2", "app.3", "app-4", "app_5", "app.no.6", "app-no-7", "app_no_8"]
- mock_packages = {myreqs_packages: {package: {} for package in dependencies}}
-
- # create mock pypi repository
- pypi_repo = os.path.join(project, "files", "pypi-repo")
- os.makedirs(pypi_repo, exist_ok=True)
- setup_pypi_repo(mock_packages, pypi_repo)
-
- element = {
- "kind": "import",
- "sources": [
- {"kind": "local", "path": "files/pip-source"},
- {"kind": "pip", "url": "file://{}".format(os.path.realpath(pypi_repo)), "packages": [myreqs_packages]},
- ],
- }
- os.makedirs(os.path.dirname(os.path.join(element_path, element_name)), exist_ok=True)
- _yaml.roundtrip_dump(element, os.path.join(element_path, element_name))
-
- result = cli.run(project=project, args=["source", "track", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["build", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["artifact", "checkout", element_name, "--directory", checkout])
- assert result.exit_code == 0
-
- assert_contains(
- checkout,
- [
- "/.bst_pip_downloads",
- "/.bst_pip_downloads/hellolib-0.1.tar.gz",
- "/.bst_pip_downloads/app2-0.1.tar.gz",
- "/.bst_pip_downloads/app.3-0.1.tar.gz",
- "/.bst_pip_downloads/app-4-0.1.tar.gz",
- "/.bst_pip_downloads/app_5-0.1.tar.gz",
- "/.bst_pip_downloads/app.no.6-0.1.tar.gz",
- "/.bst_pip_downloads/app-no-7-0.1.tar.gz",
- "/.bst_pip_downloads/app_no_8-0.1.tar.gz",
- ],
- )
-
-
-@pytest.mark.datafiles(DATA_DIR)
-def test_pip_source_import_requirements_files(cli, datafiles, setup_pypi_repo):
- project = str(datafiles)
- checkout = os.path.join(cli.directory, "checkout")
- element_path = os.path.join(project, "elements")
- element_name = "pip/hello.bst"
-
- # check that exotically named packages are imported correctly
- myreqs_packages = "hellolib"
- dependencies = ["app2", "app.3", "app-4", "app_5", "app.no.6", "app-no-7", "app_no_8"]
- mock_packages = {myreqs_packages: {package: {} for package in dependencies}}
-
- # create mock pypi repository
- pypi_repo = os.path.join(project, "files", "pypi-repo")
- os.makedirs(pypi_repo, exist_ok=True)
- setup_pypi_repo(mock_packages, pypi_repo)
-
- element = {
- "kind": "import",
- "sources": [
- {"kind": "local", "path": "files/pip-source"},
- {
- "kind": "pip",
- "url": "file://{}".format(os.path.realpath(pypi_repo)),
- "requirements-files": ["myreqs.txt"],
- },
- ],
- }
- os.makedirs(os.path.dirname(os.path.join(element_path, element_name)), exist_ok=True)
- _yaml.roundtrip_dump(element, os.path.join(element_path, element_name))
-
- result = cli.run(project=project, args=["source", "track", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["build", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["artifact", "checkout", element_name, "--directory", checkout])
- assert result.exit_code == 0
-
- assert_contains(
- checkout,
- [
- "/.bst_pip_downloads",
- "/.bst_pip_downloads/hellolib-0.1.tar.gz",
- "/.bst_pip_downloads/app2-0.1.tar.gz",
- "/.bst_pip_downloads/app.3-0.1.tar.gz",
- "/.bst_pip_downloads/app-4-0.1.tar.gz",
- "/.bst_pip_downloads/app_5-0.1.tar.gz",
- "/.bst_pip_downloads/app.no.6-0.1.tar.gz",
- "/.bst_pip_downloads/app-no-7-0.1.tar.gz",
- "/.bst_pip_downloads/app_no_8-0.1.tar.gz",
- ],
- )
-
-
-@pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
-def test_pip_source_build(cli, datafiles, setup_pypi_repo):
- project = str(datafiles)
- element_path = os.path.join(project, "elements")
- element_name = "pip/hello.bst"
-
- # check that exotically named packages are imported correctly
- myreqs_packages = "hellolib"
- dependencies = ["app2", "app.3", "app-4", "app_5", "app.no.6", "app-no-7", "app_no_8"]
- mock_packages = {myreqs_packages: {package: {} for package in dependencies}}
-
- # create mock pypi repository
- pypi_repo = os.path.join(project, "files", "pypi-repo")
- os.makedirs(pypi_repo, exist_ok=True)
- setup_pypi_repo(mock_packages, pypi_repo)
-
- element = {
- "kind": "manual",
- "depends": ["base.bst"],
- "sources": [
- {"kind": "local", "path": "files/pip-source"},
- {
- "kind": "pip",
- "url": "file://{}".format(os.path.realpath(pypi_repo)),
- "requirements-files": ["myreqs.txt"],
- "packages": dependencies,
- },
- ],
- "config": {
- "install-commands": [
- "pip3 install --no-index --prefix %{install-root}/usr .bst_pip_downloads/*.tar.gz",
- "install app1.py %{install-root}/usr/bin/",
- ]
- },
- }
- os.makedirs(os.path.dirname(os.path.join(element_path, element_name)), exist_ok=True)
- _yaml.roundtrip_dump(element, os.path.join(element_path, element_name))
-
- result = cli.run(project=project, args=["source", "track", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["build", element_name])
- assert result.exit_code == 0
-
- result = cli.run(project=project, args=["shell", element_name, "/usr/bin/app1.py"])
- assert result.exit_code == 0
- assert result.output == "Hello App1! This is hellolib\n"
diff --git a/tests/integration/project/files/pip-source/app1.py b/tests/integration/project/files/pip-source/app1.py
deleted file mode 100644
index b96d14b..0000000
--- a/tests/integration/project/files/pip-source/app1.py
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/usr/bin/env python3
-
-from hellolib import hello
-
-
-def main():
- hello("App1")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tests/integration/project/files/pip-source/myreqs.txt b/tests/integration/project/files/pip-source/myreqs.txt
deleted file mode 100644
index c805aae..0000000
--- a/tests/integration/project/files/pip-source/myreqs.txt
+++ /dev/null
@@ -1 +0,0 @@
-hellolib
[buildstream] 19/37: tests/frontend/cross_junction_workspace.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 976d63d793c3db719333fe64a286375f0de944fa
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:23:51 2022 +0900
tests/frontend/cross_junction_workspace.py: Use tar repo instead of git repo
---
tests/frontend/cross_junction_workspace.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/frontend/cross_junction_workspace.py b/tests/frontend/cross_junction_workspace.py
index c0a728b..92bee86 100644
--- a/tests/frontend/cross_junction_workspace.py
+++ b/tests/frontend/cross_junction_workspace.py
@@ -23,7 +23,7 @@ def prepare_junction_project(cli, tmpdir):
import_repo_dir = tmpdir.join("import_repo")
os.makedirs(str(import_repo_dir))
- import_repo = create_repo("git", str(import_repo_dir))
+ import_repo = create_repo("tar", str(import_repo_dir))
import_ref = import_repo.create(str(import_dir))
_yaml.roundtrip_dump(
@@ -32,7 +32,7 @@ def prepare_junction_project(cli, tmpdir):
sub_repo_dir = tmpdir.join("sub_repo")
os.makedirs(str(sub_repo_dir))
- sub_repo = create_repo("git", str(sub_repo_dir))
+ sub_repo = create_repo("tar", str(sub_repo_dir))
sub_ref = sub_repo.create(str(sub_project))
_yaml.roundtrip_dump(
[buildstream] 24/37: tests/frontend/default_target.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 349ba8707a5e49918b3b0a57486196eea5ed7f07
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:00:08 2022 +0900
tests/frontend/default_target.py: Use tar repo instead of git repo
---
tests/frontend/default_target.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/frontend/default_target.py b/tests/frontend/default_target.py
index c98bc85..3a9e94e 100644
--- a/tests/frontend/default_target.py
+++ b/tests/frontend/default_target.py
@@ -110,7 +110,7 @@ def test_default_target_track(cli, tmpdir, datafiles):
target = "track-fetch-test.bst"
# First, create an element with trackable sources
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
repo.create(project)
element_conf = {"kind": "import", "sources": [repo.source_config()]}
_yaml.roundtrip_dump(element_conf, os.path.join(project, "elements", target))
@@ -140,7 +140,7 @@ def test_default_target_fetch(cli, tmpdir, datafiles):
target = "track-fetch-test.bst"
# First, create an element with trackable sources
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(project)
element_conf = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
_yaml.roundtrip_dump(element_conf, os.path.join(project, "elements", target))
[buildstream] 14/37: tests/format/project: Use tar source instead of git source for optional refs test
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 33bf323b4891cde28dfb993b19308da0d6c06bae
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 19:10:51 2022 +0900
tests/format/project: Use tar source instead of git source for optional refs test
---
tests/format/project/refs-options/target.bst | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/tests/format/project/refs-options/target.bst b/tests/format/project/refs-options/target.bst
index bf91922..8e96802 100644
--- a/tests/format/project/refs-options/target.bst
+++ b/tests/format/project/refs-options/target.bst
@@ -1,9 +1,8 @@
kind: import
description: |
- Import some git repo with optional refs
+ Import some tar repo with optional refs
sources:
-- kind: git
- url: http://pony.com/git
- track: master
+- kind: tar
+ url: http://pony.com/tarball.tgz
[buildstream] 22/37: tests/frontend/workspace.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0ec44992d24af9f45711751c8d218faa7c199ad7
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:44:08 2022 +0900
tests/frontend/workspace.py: Use tar repo instead of git repo
---
tests/frontend/workspace.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/frontend/workspace.py b/tests/frontend/workspace.py
index 7b39e31..e6976ac 100644
--- a/tests/frontend/workspace.py
+++ b/tests/frontend/workspace.py
@@ -707,7 +707,7 @@ def test_buildable_no_ref(cli, tmpdir, datafiles):
element_path = os.path.join(project, "elements")
# Write out our test target without any source ref
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
element = {"kind": "import", "sources": [repo.source_config()]}
_yaml.roundtrip_dump(element, os.path.join(element_path, element_name))
@@ -893,7 +893,7 @@ def test_list_supported_workspace(cli, tmpdir, datafiles, workspace_cfg, expecte
# Create our repo object of the given source type with
# the bin files, and then collect the initial ref.
#
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(bin_files_path)
# Write out our test target
[buildstream] 03/37: tests/sources/bzr.py: Removing bzr specific tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d1a074887535cb302af945bf6ab525abdb47c972
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:09:21 2022 +0900
tests/sources/bzr.py: Removing bzr specific tests
---
tests/sources/bzr.py | 68 ------------------------------------------
tests/sources/bzr/basic/test | 1 -
tests/sources/bzr/project.conf | 3 --
3 files changed, 72 deletions(-)
diff --git a/tests/sources/bzr.py b/tests/sources/bzr.py
deleted file mode 100644
index a4f8f22..0000000
--- a/tests/sources/bzr.py
+++ /dev/null
@@ -1,68 +0,0 @@
-# Pylint doesn't play well with fixtures and dependency injection from pytest
-# pylint: disable=redefined-outer-name
-
-import os
-import subprocess
-
-import pytest
-
-from buildstream import _yaml
-from buildstream._testing import cli # pylint: disable=unused-import
-from buildstream._testing import create_repo
-from buildstream._testing import generate_element
-from buildstream._testing._utils.site import HAVE_BZR
-
-pytestmark = pytest.mark.skipif(HAVE_BZR is False, reason="bzr is not available")
-DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "bzr")
-
-
-@pytest.mark.datafiles(os.path.join(DATA_DIR))
-def test_fetch_checkout(cli, tmpdir, datafiles):
- project = str(datafiles)
- checkoutdir = os.path.join(str(tmpdir), "checkout")
-
- repo = create_repo("bzr", str(tmpdir))
- ref = repo.create(os.path.join(project, "basic"))
-
- # Write out our test target
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- # Fetch, build, checkout
- result = cli.run(project=project, args=["source", "fetch", "target.bst"])
- assert result.exit_code == 0
- result = cli.run(project=project, args=["build", "target.bst"])
- assert result.exit_code == 0
- result = cli.run(project=project, args=["artifact", "checkout", "target.bst", "--directory", checkoutdir])
- assert result.exit_code == 0
-
- # Assert we checked out the file as it was commited
- with open(os.path.join(checkoutdir, "test"), encoding="utf-8") as f:
- text = f.read()
-
- assert text == "test\n"
-
-
-@pytest.mark.datafiles(DATA_DIR)
-def test_open_bzr_customize(cli, tmpdir, datafiles):
- project = str(datafiles)
- repo = create_repo("bzr", str(tmpdir))
- ref = repo.create(os.path.join(project, "basic"))
-
- element = {"kind": "import", "sources": [repo.source_config(ref=ref)]}
- generate_element(project, "target.bst", element)
-
- workspace = os.path.join(datafiles, "bzr-workspace")
- result = cli.run(cwd=project, project=project, args=["workspace", "open", "--directory", workspace, "target.bst"])
- result.assert_success()
-
- # Check that the .bzr dir exists
- assert os.path.isdir(os.path.join(workspace, ".bzr"))
-
- # Check that the correct origin branch is set
- element_config = _yaml.load(os.path.join(project, "target.bst"), shortname=None)
- source_config = element_config.get_sequence("sources").mapping_at(0)
- output = subprocess.check_output(["bzr", "info"], cwd=workspace)
- stripped_url = source_config.get_str("url").lstrip("file:///")
- expected_output_str = "checkout of branch: /{}/{}".format(stripped_url, source_config.get_str("track"))
- assert expected_output_str in str(output)
diff --git a/tests/sources/bzr/basic/test b/tests/sources/bzr/basic/test
deleted file mode 100644
index 9daeafb..0000000
--- a/tests/sources/bzr/basic/test
+++ /dev/null
@@ -1 +0,0 @@
-test
diff --git a/tests/sources/bzr/project.conf b/tests/sources/bzr/project.conf
deleted file mode 100644
index 08a9d60..0000000
--- a/tests/sources/bzr/project.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Basic Project
-name: foo
-min-version: 2.0
[buildstream] 28/37: tests/sourcecache/fetch.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 13b386559360395ecaf75e7d2a9ab4bf5b18a80a
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:27:08 2022 +0900
tests/sourcecache/fetch.py: Use tar repo instead of git repo
---
tests/sourcecache/fetch.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/tests/sourcecache/fetch.py b/tests/sourcecache/fetch.py
index 2274235..4e42bde 100644
--- a/tests/sourcecache/fetch.py
+++ b/tests/sourcecache/fetch.py
@@ -35,7 +35,7 @@ DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project")
def create_test_element(tmpdir, project_dir):
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
ref = repo.create(os.path.join(project_dir, "files"))
element_path = os.path.join(project_dir, "elements")
element_name = "fetch.bst"
@@ -89,7 +89,7 @@ def test_source_fetch(cli, tmpdir, datafiles):
res = cli.run(project=project_dir, args=["build", element_name])
res.assert_success()
- assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "git")) != []
+ assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "tar")) != []
# get root digest of source
sourcecache = context.sourcecache
@@ -123,7 +123,7 @@ def test_source_fetch(cli, tmpdir, datafiles):
# check that we have the source in the cas now and it's not fetched
element._query_source_cache()
assert element._cached_sources()
- assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "git")) == []
+ assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "tar")) == []
@pytest.mark.datafiles(DATA_DIR)
@@ -153,7 +153,7 @@ def test_fetch_fallback(cli, tmpdir, datafiles):
assert (
"Remote source service ({}) does not have source {} cached".format(share.repo, brief_key)
) in res.stderr
- assert ("SUCCESS Fetching from {}".format(repo.source_config(ref=ref)["url"])) in res.stderr
+ assert ("SUCCESS Fetching {}".format(repo.source_config(ref=ref)["url"])) in res.stderr
# Check that the source in both in the source dir and the local CAS
project = Project(project_dir, context)
@@ -186,7 +186,7 @@ def test_pull_fail(cli, tmpdir, datafiles):
# Should fail in stream, with a plugin task causing the error
res = cli.run(project=project_dir, args=["build", element_name])
res.assert_main_error(ErrorDomain.STREAM, None)
- res.assert_task_error(ErrorDomain.PLUGIN, None)
+ res.assert_task_error(ErrorDomain.SOURCE, None)
assert (
"Remote source service ({}) does not have source {} cached".format(
share.repo, source._get_brief_display_key()
@@ -218,7 +218,7 @@ def test_source_pull_partial_fallback_fetch(cli, tmpdir, datafiles):
res = cli.run(project=project_dir, args=["build", element_name])
res.assert_success()
- assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "git")) != []
+ assert os.listdir(os.path.join(str(tmpdir), "cache", "sources", "tar")) != []
# get root digest of source
sourcecache = context.sourcecache
@@ -243,4 +243,4 @@ def test_source_pull_partial_fallback_fetch(cli, tmpdir, datafiles):
res = cli.run(project=project_dir, args=["source", "fetch", element_name])
res.assert_success()
- assert ("SUCCESS Fetching from {}".format(repo.source_config(ref=ref)["url"])) in res.stderr
+ assert ("SUCCESS Fetching {}".format(repo.source_config(ref=ref)["url"])) in res.stderr
[buildstream] 25/37: tests/frontend/buildcheckout.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 10efdb42d5639f4b0694035337c8805aae3f4754
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:03:18 2022 +0900
tests/frontend/buildcheckout.py: Use tar repo instead of git repo
---
tests/frontend/buildcheckout.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/frontend/buildcheckout.py b/tests/frontend/buildcheckout.py
index 8bc8f1c..ab891b6 100644
--- a/tests/frontend/buildcheckout.py
+++ b/tests/frontend/buildcheckout.py
@@ -1038,7 +1038,7 @@ def test_partial_artifact_checkout_fetch(cli, datafiles, tmpdir):
project = str(datafiles)
checkout_dir = os.path.join(str(tmpdir), "checkout")
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
repo.create(os.path.join(str(datafiles), "files"))
element_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
[buildstream] 23/37: tests/frontend/mirror.py: Use internal git source for these tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 16a242e137a4dd033fa8dd14aeffea3bc8289bf4
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:55:49 2022 +0900
tests/frontend/mirror.py: Use internal git source for these tests
---
tests/frontend/mirror.py | 39 ++++++++++++++++++++++++++++++---------
1 file changed, 30 insertions(+), 9 deletions(-)
diff --git a/tests/frontend/mirror.py b/tests/frontend/mirror.py
index 4de64bb..ff20f82 100644
--- a/tests/frontend/mirror.py
+++ b/tests/frontend/mirror.py
@@ -9,6 +9,10 @@ from buildstream.exceptions import ErrorDomain
from buildstream._testing import create_repo
from buildstream._testing import cli # pylint: disable=unused-import
+from tests.testutils.repo.git import Git
+from tests.testutils.site import pip_sample_packages # pylint: disable=unused-import
+from tests.testutils.site import SAMPLE_PACKAGES_SKIP_REASON
+
# Project directory
TOP_DIR = os.path.dirname(os.path.realpath(__file__))
@@ -435,6 +439,7 @@ def test_mirror_fetch_default_cmdline_overrides_config(cli, tmpdir):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_mirror_git_submodule_fetch(cli, tmpdir, datafiles):
# Test that it behaves as expected with submodules, both defined in config
# and discovered when fetching.
@@ -444,15 +449,15 @@ def test_mirror_git_submodule_fetch(cli, tmpdir, datafiles):
dev_files_path = os.path.join(str(datafiles), "files", "dev-files", "usr")
mirror_dir = os.path.join(str(datafiles), "mirror")
- defined_subrepo = create_repo("git", str(tmpdir), "defined_subrepo")
+ defined_subrepo = Git(str(tmpdir), "defined_subrepo")
defined_subrepo.create(bin_files_path)
defined_subrepo.copy(mirror_dir)
defined_subrepo.add_file(foo_file)
- found_subrepo = create_repo("git", str(tmpdir), "found_subrepo")
+ found_subrepo = Git(str(tmpdir), "found_subrepo")
found_subrepo.create(dev_files_path)
- main_repo = create_repo("git", str(tmpdir))
+ main_repo = Git(str(tmpdir))
main_mirror_ref = main_repo.create(bin_files_path)
main_repo.add_submodule("defined", "file://" + defined_subrepo.repo)
main_repo.add_submodule("found", "file://" + found_subrepo.repo)
@@ -509,6 +514,7 @@ def test_mirror_git_submodule_fetch(cli, tmpdir, datafiles):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
# Main repo has no mirror or alias.
# One submodule is overridden to use a mirror.
@@ -524,12 +530,12 @@ def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
upstream_bin_repodir = os.path.join(str(tmpdir), "bin-upstream")
mirror_bin_repodir = os.path.join(str(tmpdir), "bin-mirror")
- upstream_bin_repo = create_repo("git", upstream_bin_repodir)
+ upstream_bin_repo = Git(upstream_bin_repodir)
upstream_bin_repo.create(bin_files_path)
mirror_bin_repo = upstream_bin_repo.copy(mirror_bin_repodir)
dev_repodir = os.path.join(str(tmpdir), "dev-upstream")
- dev_repo = create_repo("git", dev_repodir)
+ dev_repo = Git(dev_repodir)
dev_repo.create(dev_files_path)
main_files = os.path.join(str(tmpdir), "main-files")
@@ -537,7 +543,7 @@ def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
with open(os.path.join(main_files, "README"), "w", encoding="utf-8") as f:
f.write("TEST\n")
main_repodir = os.path.join(str(tmpdir), "main-upstream")
- main_repo = create_repo("git", main_repodir)
+ main_repo = Git(main_repodir)
main_repo.create(main_files)
upstream_url = "file://{}".format(upstream_bin_repo.repo)
@@ -571,6 +577,13 @@ def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
"min-version": "2.0",
"element-path": "elements",
"aliases": {alias: upstream_map + "/"},
+ "plugins": [
+ {
+ "origin": "pip",
+ "package-name": "sample-plugins",
+ "sources": ["git"],
+ }
+ ],
"mirrors": [
{
"name": "middle-earth",
@@ -600,6 +613,7 @@ def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif("not pip_sample_packages()", reason=SAMPLE_PACKAGES_SKIP_REASON)
def test_mirror_fallback_git_with_submodules(cli, tmpdir, datafiles):
# Main repo has mirror. But does not list submodules.
#
@@ -610,11 +624,11 @@ def test_mirror_fallback_git_with_submodules(cli, tmpdir, datafiles):
dev_files_path = os.path.join(str(datafiles), "files", "dev-files", "usr")
bin_repodir = os.path.join(str(tmpdir), "bin-repo")
- bin_repo = create_repo("git", bin_repodir)
+ bin_repo = Git(bin_repodir)
bin_repo.create(bin_files_path)
dev_repodir = os.path.join(str(tmpdir), "dev-repo")
- dev_repo = create_repo("git", dev_repodir)
+ dev_repo = Git(dev_repodir)
dev_repo.create(dev_files_path)
main_files = os.path.join(str(tmpdir), "main-files")
@@ -622,7 +636,7 @@ def test_mirror_fallback_git_with_submodules(cli, tmpdir, datafiles):
with open(os.path.join(main_files, "README"), "w", encoding="utf-8") as f:
f.write("TEST\n")
upstream_main_repodir = os.path.join(str(tmpdir), "main-upstream")
- upstream_main_repo = create_repo("git", upstream_main_repodir)
+ upstream_main_repo = Git(upstream_main_repodir)
upstream_main_repo.create(main_files)
upstream_main_repo.add_submodule("bin", url="file://{}".format(bin_repo.repo))
@@ -664,6 +678,13 @@ def test_mirror_fallback_git_with_submodules(cli, tmpdir, datafiles):
"min-version": "2.0",
"element-path": "elements",
"aliases": {alias: upstream_map + "/"},
+ "plugins": [
+ {
+ "origin": "pip",
+ "package-name": "sample-plugins",
+ "sources": ["git"],
+ }
+ ],
"mirrors": [
{
"name": "middle-earth",
[buildstream] 26/37: tests/frontend/pull.py: Use tar repo instead of git repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit a47d0f1b33285cd311de3e9f72fcf9275846971e
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 13:07:16 2022 +0900
tests/frontend/pull.py: Use tar repo instead of git repo
---
tests/frontend/pull.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/frontend/pull.py b/tests/frontend/pull.py
index 91a9423..52a30cd 100644
--- a/tests/frontend/pull.py
+++ b/tests/frontend/pull.py
@@ -344,7 +344,7 @@ def test_pull_missing_blob_split_share(cli, tmpdir, datafiles):
@pytest.mark.datafiles(DATA_DIR)
def test_pull_missing_local_blob(cli, tmpdir, datafiles):
project = os.path.join(datafiles.dirname, datafiles.basename)
- repo = create_repo("git", str(tmpdir))
+ repo = create_repo("tar", str(tmpdir))
repo.create(os.path.join(str(datafiles), "files"))
element_dir = os.path.join(str(tmpdir), "elements")
project = str(tmpdir)
[buildstream] 01/37: _testing/_cachekeys.py: Fix references to buildstream._testing
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit b386975f5890d900f853f8614567d624d576386e
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 12:13:20 2022 +0900
_testing/_cachekeys.py: Fix references to buildstream._testing
---
src/buildstream/_testing/_cachekeys.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/buildstream/_testing/_cachekeys.py b/src/buildstream/_testing/_cachekeys.py
index 3ddcf18..086ceac 100644
--- a/src/buildstream/_testing/_cachekeys.py
+++ b/src/buildstream/_testing/_cachekeys.py
@@ -47,7 +47,7 @@ def check_cache_key_stability(project_path: os.PathLike, cli: Cli) -> None:
project's directory.
:param project_path: Path to a project
- :param cli: a `cli` object as provided by the fixture :func:`buildstream.testing.runcli.cli`
+ :param cli: a `cli` object as provided by the fixture :func:`buildstream._testing.runcli.cli`
"""
result = cli.run(
project=project_path, silent=True, args=["show", "--format", "%{name}::%{full-key}", "target.bst"]
@@ -110,7 +110,7 @@ def _load_expected_keys(project_dir, actual_keys, raise_error=True):
raise Exception(
"Cache key test needs update, "
+ "expected file {} not found.\n\n".format(expected)
- + "Use python3 -m buildstream.testing._update_cachekeys in the"
+ + "Use python3 -m buildstream._testing._update_cachekeys in the"
+ " project's directory to automatically update this test case"
)
@@ -142,6 +142,6 @@ def _assert_cache_keys(project_dir, output):
raise AssertionError(
"Cache key mismatches occurred:\n{}\n".format(info)
- + "Use python3 -m buildstream.testing._update_cachekeys in the project's "
+ + "Use python3 -m buildstream._testing._update_cachekeys in the project's "
+ "directory to automatically update this test case"
)
[buildstream] 02/37: tests/conftest.py: Remove bzr from automated source tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7dc17437107c3cba3fc0bdad6c0014c980c4278e
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Mon Mar 28 16:08:25 2022 +0900
tests/conftest.py: Remove bzr from automated source tests
---
tests/conftest.py | 2 --
tests/testutils/repo/bzr.py | 47 ---------------------------------------------
2 files changed, 49 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index 168a8ca..0f3a2dd 100755
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -32,7 +32,6 @@ from buildstream._testing.integration import integration_cache # pylint: disabl
from tests.testutils.repo.git import Git
-from tests.testutils.repo.bzr import Bzr
from tests.testutils.repo.tar import Tar
@@ -130,7 +129,6 @@ def remote_services(request):
# Setup for templated source tests #
#################################################
register_repo_kind("git", Git, None)
-register_repo_kind("bzr", Bzr, None)
register_repo_kind("tar", Tar, None)
diff --git a/tests/testutils/repo/bzr.py b/tests/testutils/repo/bzr.py
deleted file mode 100644
index 038d759..0000000
--- a/tests/testutils/repo/bzr.py
+++ /dev/null
@@ -1,47 +0,0 @@
-import os
-import subprocess
-import pytest
-
-from buildstream._testing import Repo
-from buildstream._testing._utils.site import BZR, BZR_ENV, HAVE_BZR
-
-
-class Bzr(Repo):
- def __init__(self, directory, subdir="repo"):
- if not HAVE_BZR:
- pytest.skip("bzr is not available")
- super().__init__(directory, subdir)
- self.bzr = BZR
-
- self.env = os.environ.copy()
- self.env.update(BZR_ENV)
-
- def create(self, directory):
- # Work around race condition in bzr's creation of ~/.bazaar in
- # ensure_config_dir_exists() when running tests in parallel.
- bazaar_config_dir = os.path.expanduser("~/.bazaar")
- os.makedirs(bazaar_config_dir, exist_ok=True)
-
- branch_dir = os.path.join(self.repo, "trunk")
-
- subprocess.call([self.bzr, "init-repo", self.repo], env=self.env)
- subprocess.call([self.bzr, "init", branch_dir], env=self.env)
- self.copy_directory(directory, branch_dir)
- subprocess.call([self.bzr, "add", "."], env=self.env, cwd=branch_dir)
- subprocess.call([self.bzr, "commit", '--message="Initial commit"'], env=self.env, cwd=branch_dir)
-
- return self.latest_commit()
-
- def source_config(self, ref=None):
- config = {"kind": "bzr", "url": "file://" + self.repo, "track": "trunk"}
- if ref is not None:
- config["ref"] = ref
-
- return config
-
- def latest_commit(self):
- return subprocess.check_output(
- [self.bzr, "version-info", "--custom", "--template={revno}", os.path.join(self.repo, "trunk")],
- env=self.env,
- universal_newlines=True,
- ).strip()
[buildstream] 32/37: tests/testutils/element_generators.py: Port create_element_size() to use tar repo
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9d0dce42318da4d687ff60e38851d4cd47614527
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:02:22 2022 +0900
tests/testutils/element_generators.py: Port create_element_size() to use tar repo
Also remove unused update_element_size() utility function (which cannot be
implemented with a tar repo anyway, luckily this is unused).
---
tests/sourcecache/source-checkout.py | 6 ++---
tests/sourcecache/staging.py | 4 ++--
tests/sourcecache/workspace.py | 6 ++---
tests/testutils/__init__.py | 2 +-
tests/testutils/element_generators.py | 41 +++--------------------------------
5 files changed, 12 insertions(+), 47 deletions(-)
diff --git a/tests/sourcecache/source-checkout.py b/tests/sourcecache/source-checkout.py
index c87eaff..70a074d 100644
--- a/tests/sourcecache/source-checkout.py
+++ b/tests/sourcecache/source-checkout.py
@@ -52,7 +52,7 @@ def test_source_checkout(tmpdir, datafiles, cli):
# check implicit fetching
res = cli.run(project=project_dir, args=["source", "checkout", "--directory", target_dir, "target.bst"])
res.assert_success()
- assert "Fetching from" in res.stderr
+ assert "Fetching" in res.stderr
# remove the directory and check source checkout works with sources only in
# the CAS
@@ -62,11 +62,11 @@ def test_source_checkout(tmpdir, datafiles, cli):
res = cli.run(project=project_dir, args=["source", "checkout", "--directory", target_dir, "target.bst"])
res.assert_success()
- assert "Fetching from" not in res.stderr
+ assert "Fetching" not in res.stderr
# remove the CAS and check it doesn't work again
shutil.rmtree(target_dir)
shutil.rmtree(os.path.join(cache_dir, "cas"))
res = cli.run(project=project_dir, args=["source", "checkout", "--directory", target_dir, "target.bst"])
- res.assert_task_error(ErrorDomain.PLUGIN, None)
+ res.assert_task_error(ErrorDomain.SOURCE, None)
diff --git a/tests/sourcecache/staging.py b/tests/sourcecache/staging.py
index 09ff608..61c0feb 100644
--- a/tests/sourcecache/staging.py
+++ b/tests/sourcecache/staging.py
@@ -153,7 +153,7 @@ def test_staged_source_build(tmpdir, datafiles, cli):
# build and check that no fetching was done.
res = cli.run(project=project_dir, args=["build", "target.bst"])
res.assert_success()
- assert "Fetching from" not in res.stderr
+ assert "Fetching" not in res.stderr
# assert the source directory is still empty (though there may be
# directories from staging etc.)
@@ -172,4 +172,4 @@ def test_staged_source_build(tmpdir, datafiles, cli):
# Check that it now fetches from when building the target
res = cli.run(project=project_dir, args=["build", "target.bst"])
res.assert_success()
- assert "Fetching from" in res.stderr
+ assert "Fetching" in res.stderr
diff --git a/tests/sourcecache/workspace.py b/tests/sourcecache/workspace.py
index 00419c1..13eb18e 100644
--- a/tests/sourcecache/workspace.py
+++ b/tests/sourcecache/workspace.py
@@ -48,7 +48,7 @@ def test_workspace_source_fetch(tmpdir, datafiles, cli):
create_element_size("target.bst", project_dir, element_path, [], 10000)
res = cli.run(project=project_dir, args=["build", "target.bst"])
res.assert_success()
- assert "Fetching from" in res.stderr
+ assert "Fetching" in res.stderr
# remove the original sources
shutil.rmtree(source_dir)
@@ -56,7 +56,7 @@ def test_workspace_source_fetch(tmpdir, datafiles, cli):
# Open a workspace and check that fetches the original sources
res = cli.run(project=project_dir, args=["workspace", "open", "target.bst", "--directory", workspace])
res.assert_success()
- assert "Fetching from" in res.stderr
+ assert "Fetching" in res.stderr
assert os.listdir(workspace) != []
@@ -89,7 +89,7 @@ def test_workspace_open_no_source_push(tmpdir, datafiles, cli):
create_element_size("target.bst", project_dir, element_path, [], 10000)
res = cli.run(project=project_dir, args=["build", "target.bst"])
res.assert_success()
- assert "Fetching from" in res.stderr
+ assert "Fetching" in res.stderr
assert "Pushed source" in res.stderr
# clear the cas and open a workspace
diff --git a/tests/testutils/__init__.py b/tests/testutils/__init__.py
index 0fe23dc..87f61d9 100644
--- a/tests/testutils/__init__.py
+++ b/tests/testutils/__init__.py
@@ -24,7 +24,7 @@
from .artifactshare import create_artifact_share, create_split_share, assert_shared, assert_not_shared, ArtifactShare
from .context import dummy_context
-from .element_generators import create_element_size, update_element_size
+from .element_generators import create_element_size
from .junction import generate_junction
from .runner_integration import wait_for_cache_granularity
from .python_repo import setup_pypi_repo
diff --git a/tests/testutils/element_generators.py b/tests/testutils/element_generators.py
index 5c6fa5d..fa1f9c0 100644
--- a/tests/testutils/element_generators.py
+++ b/tests/testutils/element_generators.py
@@ -20,16 +20,15 @@ from buildstream._testing import create_repo
# size: (int) size of the element in bytes
#
# Returns:
-# (Repo): A git repo which can be used to introduce trackable changes
-# by using the update_element_size() function below.
+# (Repo): A repo representing the sized element
#
def create_element_size(name, project_dir, elements_path, dependencies, size):
full_elements_path = os.path.join(project_dir, elements_path)
os.makedirs(full_elements_path, exist_ok=True)
- # Create a git repo
+ # Create a repo
repodir = os.path.join(project_dir, "repos")
- repo = create_repo("git", repodir, subdir=name)
+ repo = create_repo("tar", repodir, subdir=name)
with utils._tempdir(dir=project_dir) as tmp:
@@ -61,37 +60,3 @@ def create_element_size(name, project_dir, elements_path, dependencies, size):
# Return the repo, so that it can later be used to add commits
return repo
-
-
-# update_element_size()
-#
-# Updates a repo returned by create_element_size() such that
-# the newly added commit is completely changed, and has the newly
-# specified size.
-#
-# The name and project_dir arguments must match the arguments
-# previously given to create_element_size()
-#
-# Args:
-# name: (str) of the element name (e.g. target.bst)
-# project_dir (str): The path to the project
-# repo: (Repo) The Repo returned by create_element_size()
-# size: (int) The new size which the element generates, in bytes
-#
-# Returns:
-# (Repo): A git repo which can be used to introduce trackable changes
-# by using the update_element_size() function below.
-#
-def update_element_size(name, project_dir, repo, size):
-
- with utils._tempdir(dir=project_dir) as tmp:
-
- new_file = os.path.join(tmp, name)
-
- # Use /dev/urandom to create the sized file in the datadir
- with open(new_file, "wb+") as f:
- f.write(os.urandom(size))
-
- # Modify the git repo with a new commit to the same path,
- # replacing the original file with a new one.
- repo.modify_file(new_file, os.path.join("data", name))
[buildstream] 15/37: tests/testutils/site.py: Move the pip_sample_plugins() check to a common site.py
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit c7020c34c241634684794154e7aa5115dded8a6d
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Tue Mar 29 15:45:27 2022 +0900
tests/testutils/site.py: Move the pip_sample_plugins() check to a common site.py
As we will be using the sample plugins package to carry some plugins we want
to use for general purpose testing of the API surface throughout buildstream,
we'll need to have this in a shared location.
---
tests/plugins/loading.py | 34 +++-------------------------------
tests/testutils/site.py | 29 +++++++++++++++++++++++++++++
2 files changed, 32 insertions(+), 31 deletions(-)
diff --git a/tests/plugins/loading.py b/tests/plugins/loading.py
index e0663c5..2863133 100644
--- a/tests/plugins/loading.py
+++ b/tests/plugins/loading.py
@@ -14,6 +14,9 @@ from buildstream.exceptions import ErrorDomain, LoadErrorReason
from buildstream._testing import cli # pylint: disable=unused-import
from buildstream import _yaml
+from tests.testutils.site import pip_sample_packages # pylint: disable=unused-import
+from tests.testutils.site import SAMPLE_PACKAGES_SKIP_REASON
+
DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "loading")
@@ -43,37 +46,6 @@ def setup_element(project_path, plugin_type, plugin_name):
_yaml.roundtrip_dump(element, element_path)
-# This function is used for pytest skipif() expressions.
-#
-# Tests which require our plugins in tests/plugins/pip-samples need
-# to check if these plugins are installed, they are only guaranteed
-# to be installed when running tox, but not when using pytest directly
-# to test that BuildStream works when integrated in your system.
-#
-def pip_sample_packages():
- import pkg_resources
-
- required = {"sample-plugins"}
- installed = {pkg.key for pkg in pkg_resources.working_set} # pylint: disable=not-an-iterable
- missing = required - installed
-
- if missing:
- return False
-
- return True
-
-
-SAMPLE_PACKAGES_SKIP_REASON = """
-The sample plugins package used to test pip plugin origins is not installed.
-
-This is usually tested automatically with `tox`, if you are running
-`pytest` directly then you can install these plugins directly using pip.
-
-The plugins are located in the tests/plugins/sample-plugins directory
-of your BuildStream checkout.
-"""
-
-
####################################################
# Tests #
####################################################
diff --git a/tests/testutils/site.py b/tests/testutils/site.py
new file mode 100644
index 0000000..d626cfe
--- /dev/null
+++ b/tests/testutils/site.py
@@ -0,0 +1,29 @@
+# This function is used for pytest skipif() expressions.
+#
+# Tests which require our plugins in tests/plugins/pip-samples need
+# to check if these plugins are installed, they are only guaranteed
+# to be installed when running tox, but not when using pytest directly
+# to test that BuildStream works when integrated in your system.
+#
+def pip_sample_packages():
+ import pkg_resources
+
+ required = {"sample-plugins"}
+ installed = {pkg.key for pkg in pkg_resources.working_set} # pylint: disable=not-an-iterable
+ missing = required - installed
+
+ if missing:
+ return False
+
+ return True
+
+
+SAMPLE_PACKAGES_SKIP_REASON = """
+The sample plugins package used to test pip plugin origins is not installed.
+
+This is usually tested automatically with `tox`, if you are running
+`pytest` directly then you can install these plugins directly using pip.
+
+The plugins are located in the tests/plugins/sample-plugins directory
+of your BuildStream checkout.
+"""
[buildstream] 36/37: tests/cachekey: Removing git source from cache key tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 6999d0c39f86d818c0ddb49fc00cdfdd713493f6
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:38:32 2022 +0900
tests/cachekey: Removing git source from cache key tests
---
tests/cachekey/project/sources/git1.bst | 5 -----
tests/cachekey/project/sources/git1.expected | 1 -
tests/cachekey/project/sources/git2.bst | 9 ---------
tests/cachekey/project/sources/git2.expected | 1 -
tests/cachekey/project/sources/git3.bst | 12 ------------
tests/cachekey/project/sources/git3.expected | 1 -
tests/cachekey/project/target.bst | 3 ---
tests/cachekey/project/target.expected | 2 +-
8 files changed, 1 insertion(+), 33 deletions(-)
diff --git a/tests/cachekey/project/sources/git1.bst b/tests/cachekey/project/sources/git1.bst
deleted file mode 100644
index 8763f64..0000000
--- a/tests/cachekey/project/sources/git1.bst
+++ /dev/null
@@ -1,5 +0,0 @@
-kind: import
-sources:
-- kind: git
- url: https://example.com/git/repo.git
- ref: 6ac68af3e80b7b17c23a3c65233043550a7fa685
diff --git a/tests/cachekey/project/sources/git1.expected b/tests/cachekey/project/sources/git1.expected
deleted file mode 100644
index 7cb0871..0000000
--- a/tests/cachekey/project/sources/git1.expected
+++ /dev/null
@@ -1 +0,0 @@
-d6dbf52d5f2f4cd72ae392ddef0bd8e3b98675bcb71f4a35901e10c0e4763cbe
\ No newline at end of file
diff --git a/tests/cachekey/project/sources/git2.bst b/tests/cachekey/project/sources/git2.bst
deleted file mode 100644
index 130ccf3..0000000
--- a/tests/cachekey/project/sources/git2.bst
+++ /dev/null
@@ -1,9 +0,0 @@
-kind: import
-sources:
-- kind: git
- url: https://example.com/git/repo.git
- ref: 6ac68af3e80b7b17c23a3c65233043550a7fa685
-
- submodules:
- plugins/foo:
- url: upstream:foo.git
diff --git a/tests/cachekey/project/sources/git2.expected b/tests/cachekey/project/sources/git2.expected
deleted file mode 100644
index c155b4c..0000000
--- a/tests/cachekey/project/sources/git2.expected
+++ /dev/null
@@ -1 +0,0 @@
-903f877bccb673d258b52a95e282bada0c0be1ffae62d30c21791c2d326c8c0a
\ No newline at end of file
diff --git a/tests/cachekey/project/sources/git3.bst b/tests/cachekey/project/sources/git3.bst
deleted file mode 100644
index b331a3a..0000000
--- a/tests/cachekey/project/sources/git3.bst
+++ /dev/null
@@ -1,12 +0,0 @@
-kind: import
-sources:
-- kind: git
- url: https://example.com/git/repo.git
- ref: 6ac68af3e80b7b17c23a3c65233043550a7fa685
- tags:
- - tag: lightweight
- commit: 0a3917d57477ee9afe7be49a0e8a76f56d176df1
- annotated: false
- - tag: annotated
- commit: 68c7f0bd386684742c41ec2a54ce2325e3922f6c
- annotated: true
diff --git a/tests/cachekey/project/sources/git3.expected b/tests/cachekey/project/sources/git3.expected
deleted file mode 100644
index c44807a..0000000
--- a/tests/cachekey/project/sources/git3.expected
+++ /dev/null
@@ -1 +0,0 @@
-b2e1b540e6b3fe0ace20e8d1afaca2c44c93be317e32890bbfc8118f41c10c0a
\ No newline at end of file
diff --git a/tests/cachekey/project/target.bst b/tests/cachekey/project/target.bst
index cd5cc31..c65b8eb 100644
--- a/tests/cachekey/project/target.bst
+++ b/tests/cachekey/project/target.bst
@@ -4,9 +4,6 @@ description: |
This is the main entry point including cases in the cache key test.
depends:
-- sources/git1.bst
-- sources/git2.bst
-- sources/git3.bst
- sources/local1.bst
- sources/local2.bst
- sources/remote1.bst
diff --git a/tests/cachekey/project/target.expected b/tests/cachekey/project/target.expected
index bccc890..8698767 100644
--- a/tests/cachekey/project/target.expected
+++ b/tests/cachekey/project/target.expected
@@ -1 +1 @@
-66c2423caaae90d058d6156ee82832ac817c9f44fac766acaceb6b2d54fabb9c
\ No newline at end of file
+f8c3972e07321cd129fe153d52b976191c334ee1f4ce47fa93c069af23903d6d
\ No newline at end of file
[buildstream] 35/37: tests/conftest.py: Remove git from generic source tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/remove-plugins
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 165ece8d68bd6870645f85a90df65a8b5412080c
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Wed Mar 30 14:36:30 2022 +0900
tests/conftest.py: Remove git from generic source tests
---
tests/conftest.py | 3 ---
1 file changed, 3 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index 0f3a2dd..69f9a07 100755
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -30,8 +30,6 @@ from buildstream._testing._fixtures import ( # pylint: disable=unused-import
)
from buildstream._testing.integration import integration_cache # pylint: disable=unused-import
-
-from tests.testutils.repo.git import Git
from tests.testutils.repo.tar import Tar
@@ -128,7 +126,6 @@ def remote_services(request):
#################################################
# Setup for templated source tests #
#################################################
-register_repo_kind("git", Git, None)
register_repo_kind("tar", Tar, None)