You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by no...@apache.org on 2020/12/29 12:37:31 UTC
[buildstream] branch tlater/buildbox-run-userchroot created (now
c4facc4)
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a change to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at c4facc4 utils.py: Use `onerror` in `_force_rmtree`
This branch includes the following new commits:
new e345658 testing/runcli.py: Add BST_CAS_STAGING_ROOT environment variable
new e1204a1 testing/_utils/site.py: Add BUILDBOX_RUN variable
new 1590e07 tests: source_determinism.py: Skip flaky test with buildbox-run
new f151ffa tests/integration/compose.py: xfail for buildbox-run-userchroot
new 6e417fa tests/integration/filter.py: xfail for buildbox-run-userchroot
new faad57f tests/integration/script.py: xfail for buildbox-run-userchroot
new 9aef33b tests/integration/shell.py: xfail for buildbox-run-userchroot
new 0218c87 .gitlab-ci.yml: Update docker image
new 15cb07d .gitlab-ci.yml: Add job to test buildbox-run-userchroot
new c4facc4 utils.py: Use `onerror` in `_force_rmtree`
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 09/10: .gitlab-ci.yml: Add job to test
buildbox-run-userchroot
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 15cb07dd6f97a31751057a80f6e8610d26e789fb
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Nov 7 11:23:36 2019 +0100
.gitlab-ci.yml: Add job to test buildbox-run-userchroot
---
.gitlab-ci.yml | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 10cec8f..82fa7d3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -124,6 +124,40 @@ tests-buildbox-run:
variables:
BST_FORCE_SANDBOX: "buildbox-run"
+tests-userchroot:
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
+ <<: *tests
+ variables:
+ BST_FORCE_SANDBOX: "buildbox-run"
+ BST_CAS_STAGING_ROOT: "/builds/userchroot"
+
+ script:
+ - mkdir -p "${INTEGRATION_CACHE}"
+ - useradd -Um buildstream
+
+ # Use buildbox-run-userchroot and hardlinking
+ - ln -svf buildbox-run-userchroot /usr/local/bin/buildbox-run
+ - rm -vf /usr/local/bin/buildbox-fuse
+
+ # When using userchroot, buildbox-casd must run as a separate user
+ - useradd -g buildstream buildbox-casd
+ - chown buildbox-casd:buildstream /usr/local/bin/buildbox-casd
+ - chmod u+s /usr/local/bin/buildbox-casd
+
+ # Set up staging root with permissions required by userchroot,
+ # must be on same filesystem as current directory to support hardlinks
+ - mkdir -p "${BST_CAS_STAGING_ROOT}"
+ - chown -R buildbox-casd:buildstream "${BST_CAS_STAGING_ROOT}"
+ # userchroot doesn't allow group/world-writable base directory
+ - chmod go-w /builds
+ - echo buildbox-casd:${BST_CAS_STAGING_ROOT} > /etc/userchroot.conf
+
+ - chown -R buildstream:buildstream .
+
+ # Run the tests as a simple user to test for permission issues
+ - su buildstream -c "umask 002 && ${TEST_COMMAND}"
+ - su buildstream -c "umask 002 && ${EXTERNAL_TESTS_COMMAND}"
+
tests-fedora-missing-deps:
# Ensure that tests behave nicely while missing bwrap and ostree
image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
@@ -415,6 +449,7 @@ coverage:
- tests-remote-execution
- tests-ubuntu-18.04
- tests-unix
+ - tests-userchroot
except:
- schedules
artifacts:
[buildstream] 06/10: tests/integration/script.py: xfail for
buildbox-run-userchroot
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit faad57f606162b7b228b4129c248c2debdfb7559
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Nov 26 11:57:48 2019 +0100
tests/integration/script.py: xfail for buildbox-run-userchroot
The root directory is not allowed to be writable by userchroot.
+ sh -c -e echo 'I can write to root' > /test
sh: can't create /test: Permission denied
---
tests/integration/script.py | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/tests/integration/script.py b/tests/integration/script.py
index 67dd310..a03824e 100644
--- a/tests/integration/script.py
+++ b/tests/integration/script.py
@@ -6,7 +6,7 @@ import pytest
from buildstream import _yaml
from buildstream.testing import cli_integration as cli # pylint: disable=unused-import
-from buildstream.testing._utils.site import HAVE_SANDBOX
+from buildstream.testing._utils.site import HAVE_SANDBOX, BUILDBOX_RUN
pytestmark = pytest.mark.integration
@@ -60,6 +60,10 @@ def test_script(cli, datafiles):
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="Root directory not writable with userchroot",
+)
def test_script_root(cli, datafiles):
project = str(datafiles)
checkout = os.path.join(cli.directory, "checkout")
@@ -92,7 +96,8 @@ def test_script_root(cli, datafiles):
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
@pytest.mark.xfail(
- HAVE_SANDBOX == "buildbox-run", reason="Read-only root directory not supported by buildbox-run",
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN != "buildbox-run-userchroot",
+ reason="Read-only root directory not supported by buildbox-run",
)
def test_script_no_root(cli, datafiles):
project = str(datafiles)
@@ -166,6 +171,10 @@ def test_script_layout(cli, datafiles):
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="Root directory not writable with userchroot",
+)
def test_regression_cache_corruption(cli, datafiles):
project = str(datafiles)
checkout_original = os.path.join(cli.directory, "checkout-original")
@@ -206,6 +215,10 @@ def test_regression_tmpdir(cli, datafiles):
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="Root directory not writable with userchroot",
+)
def test_regression_cache_corruption_2(cli, datafiles):
project = str(datafiles)
checkout_original = os.path.join(cli.directory, "checkout-original")
[buildstream] 03/10: tests: source_determinism.py: Skip flaky test
with buildbox-run
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1590e07c012236ca39738b64ac63454e9b69a6a1
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Nov 26 10:40:34 2019 +0100
tests: source_determinism.py: Skip flaky test with buildbox-run
The tests are flaky due to non-deterministic timestamps in the output of
`ls -l`. See https://gitlab.com/BuildStream/buildstream/issues/1218
---
src/buildstream/testing/_sourcetests/source_determinism.py | 4 ++++
tests/integration/source-determinism.py | 4 ++++
2 files changed, 8 insertions(+)
diff --git a/src/buildstream/testing/_sourcetests/source_determinism.py b/src/buildstream/testing/_sourcetests/source_determinism.py
index d51d0e5..ed00c71 100644
--- a/src/buildstream/testing/_sourcetests/source_determinism.py
+++ b/src/buildstream/testing/_sourcetests/source_determinism.py
@@ -50,6 +50,10 @@ def create_test_directory(*path, mode=0o644):
@pytest.mark.integration
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.skipif(
+ HAVE_SANDBOX == "buildbox-run" and CASD_SEPARATE_USER,
+ reason="Flaky due to timestamps: https://gitlab.com/BuildStream/buildstream/issues/1218",
+)
def test_deterministic_source_umask(cli, tmpdir, datafiles, kind):
project = str(datafiles)
element_name = "list.bst"
diff --git a/tests/integration/source-determinism.py b/tests/integration/source-determinism.py
index 1455975..3555881 100644
--- a/tests/integration/source-determinism.py
+++ b/tests/integration/source-determinism.py
@@ -29,6 +29,10 @@ def create_test_directory(*path, mode=0o644):
@pytest.mark.integration
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.skipif(
+ HAVE_SANDBOX == "buildbox-run" and CASD_SEPARATE_USER,
+ reason="Flaky due to timestamps: https://gitlab.com/BuildStream/buildstream/issues/1218",
+)
def test_deterministic_source_local(cli, tmpdir, datafiles):
"""Only user rights should be considered for local source.
"""
[buildstream] 04/10: tests/integration/compose.py: xfail for
buildbox-run-userchroot
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit f151ffa69a1e1d0f34f551e763bd07b7264cc2bb
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Wed Nov 27 11:19:56 2019 +0100
tests/integration/compose.py: xfail for buildbox-run-userchroot
The root directory is not allowed to be writable by userchroot.
+ sh -e -c mkdir -p /tests
mkdir: can't create directory '/tests': Permission denied
Command 'mkdir -p /tests' failed with exitcode 1
---
tests/integration/compose.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tests/integration/compose.py b/tests/integration/compose.py
index 2f38aa6..f08f2e8 100644
--- a/tests/integration/compose.py
+++ b/tests/integration/compose.py
@@ -8,7 +8,7 @@ from buildstream import _yaml
from buildstream.testing import cli_integration as cli # pylint: disable=unused-import
from buildstream.testing.integration import walk_dir
-from buildstream.testing._utils.site import HAVE_SANDBOX
+from buildstream.testing._utils.site import HAVE_SANDBOX, BUILDBOX_RUN
pytestmark = pytest.mark.integration
@@ -133,6 +133,10 @@ def test_compose_include(cli, datafiles, include_domains, exclude_domains, expec
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="Root directory not writable with userchroot",
+)
def test_compose_run_integration(cli, datafiles):
project = str(datafiles)
checkout = os.path.join(cli.directory, "checkout")
[buildstream] 07/10: tests/integration/shell.py: xfail for
buildbox-run-userchroot
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 9aef33baa94fe0c546443862f9261fb0659ca435
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Nov 26 12:08:08 2019 +0100
tests/integration/shell.py: xfail for buildbox-run-userchroot
buildbox-run-userchroot requires a shell.
---
tests/integration/shell.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tests/integration/shell.py b/tests/integration/shell.py
index cea6080..040ae53 100644
--- a/tests/integration/shell.py
+++ b/tests/integration/shell.py
@@ -6,7 +6,7 @@ import pytest
from buildstream import _yaml
from buildstream.testing import cli_integration as cli # pylint: disable=unused-import
-from buildstream.testing._utils.site import HAVE_SANDBOX
+from buildstream.testing._utils.site import HAVE_SANDBOX, BUILDBOX_RUN
from buildstream._exceptions import ErrorDomain
from buildstream import utils
@@ -134,6 +134,10 @@ def test_env_assign_isolated(cli, datafiles, animal):
# /bin/sh)
@pytest.mark.datafiles(DATA_DIR)
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="buildbox-run-userchroot requires a shell",
+)
def test_no_shell(cli, datafiles):
project = str(datafiles)
element_path = os.path.join(project, "elements")
[buildstream] 10/10: utils.py: Use `onerror` in `_force_rmtree`
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit c4facc4b8bf46803df6866ebedfe296dc07373d1
Author: Tristan Maat <tr...@codethink.co.uk>
AuthorDate: Tue Dec 10 17:52:36 2019 +0000
utils.py: Use `onerror` in `_force_rmtree`
If we don't, and encounter a file we don't own, but have permission to
delete, we'll fail with EPERM, since we won't be able to change
permissions but will be able to delete it.
Instead, we now try to change permissions and remove a file *after* we
realize we couldn't at first.
---
src/buildstream/utils.py | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/buildstream/utils.py b/src/buildstream/utils.py
index b6716a2..545816e 100644
--- a/src/buildstream/utils.py
+++ b/src/buildstream/utils.py
@@ -819,20 +819,25 @@ def _remove_path_with_parents(basedir: Union[Path, str], path: Union[Path, str])
# Recursively remove directories, ignoring file permissions as much as
# possible.
-def _force_rmtree(rootpath, **kwargs):
+def _force_rmtree(rootpath):
+ def fix_permissions(function, path, info):
+ parent = os.path.dirname(path)
- os.chmod(rootpath, 0o755)
- for root, dirs, _ in os.walk(rootpath):
- for d in dirs:
- path = os.path.join(root, d.lstrip("/"))
- if os.path.exists(path) and not os.path.islink(path):
- try:
- os.chmod(path, 0o755)
- except OSError as e:
- raise UtilError("Failed to ensure write permission on file '{}': {}".format(path, e))
+ try:
+ os.chmod(parent, 0o755)
+ except OSError as e:
+ raise UtilError("Failed to ensure write permission on directory '{}': {}".format(parent, e))
+
+ # Directories need to be removed with `rmdir`, though
+ # `os.path.isdir` will follow symlinks, so make sure it's
+ # not a symlink first
+ if not os.path.islink(path) and os.path.isdir(path):
+ os.rmdir(path)
+ else:
+ os.remove(path)
try:
- shutil.rmtree(rootpath, **kwargs)
+ shutil.rmtree(rootpath, onerror=fix_permissions)
except OSError as e:
raise UtilError("Failed to remove cache directory '{}': {}".format(rootpath, e))
[buildstream] 02/10: testing/_utils/site.py: Add BUILDBOX_RUN
variable
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e1204a1a4e8b25c763ec7e2e6bb301a11d3c77f6
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Dec 5 05:52:33 2019 +0100
testing/_utils/site.py: Add BUILDBOX_RUN variable
---
src/buildstream/testing/_utils/site.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/buildstream/testing/_utils/site.py b/src/buildstream/testing/_utils/site.py
index f3b45db..9fbddf1 100644
--- a/src/buildstream/testing/_utils/site.py
+++ b/src/buildstream/testing/_utils/site.py
@@ -82,3 +82,12 @@ if HAVE_SANDBOX is not None:
pass
elif IS_LINUX and HAVE_BWRAP and (not IS_WSL):
HAVE_SANDBOX = "bwrap"
+
+
+BUILDBOX_RUN = None
+if HAVE_SANDBOX == "buildbox-run":
+ try:
+ path = utils.get_host_tool("buildbox-run")
+ BUILDBOX_RUN = os.path.basename(os.readlink(path))
+ except (ProgramNotFoundError, OSError):
+ pass
[buildstream] 08/10: .gitlab-ci.yml: Update docker image
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0218c87f441d3f9688f4efbb6f4c83ef9e42dc5a
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Dec 10 16:27:34 2019 +0100
.gitlab-ci.yml: Update docker image
---
.gitlab-ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 556be26..10cec8f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -24,7 +24,7 @@ variables:
# Our own variables
# Version of the docker images we should use for all the images.
# This is taken from buildstream/buildstream-docker-images
- DOCKER_IMAGE_VERSION: master-101787517
+ DOCKER_IMAGE_VERSION: juerg-userchroot-101788082
PYTEST_ADDOPTS: "--color=yes"
INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
PYTEST_ARGS: "--color=yes --integration -n 2"
[buildstream] 05/10: tests/integration/filter.py: xfail for
buildbox-run-userchroot
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 6e417fa512b0766ed28b1c8c20957a5a0d5f32aa
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Nov 26 11:51:43 2019 +0100
tests/integration/filter.py: xfail for buildbox-run-userchroot
The root directory is not allowed to be writable by userchroot.
+ sh -e -c touch /foo
touch: /foo: Permission denied
---
tests/integration/filter.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tests/integration/filter.py b/tests/integration/filter.py
index 2fca895..12061fe 100644
--- a/tests/integration/filter.py
+++ b/tests/integration/filter.py
@@ -7,7 +7,7 @@ import pytest
from buildstream.testing import cli # pylint: disable=unused-import
from buildstream.testing.integration import assert_contains
-from buildstream.testing._utils.site import HAVE_SANDBOX
+from buildstream.testing._utils.site import HAVE_SANDBOX, BUILDBOX_RUN
pytestmark = pytest.mark.integration
@@ -18,6 +18,10 @@ DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project")
@pytest.mark.datafiles(os.path.join(DATA_DIR))
@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+@pytest.mark.xfail(
+ HAVE_SANDBOX == "buildbox-run" and BUILDBOX_RUN == "buildbox-run-userchroot",
+ reason="Root directory not writable with userchroot",
+)
def test_filter_pass_integration(datafiles, cli):
project = str(datafiles)
[buildstream] 01/10: testing/runcli.py: Add BST_CAS_STAGING_ROOT
environment variable
Posted by no...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
not-in-ldap pushed a commit to branch tlater/buildbox-run-userchroot
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e3456583dd2764eb7954665b0f12197f4201a7b7
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Tue Nov 12 15:35:02 2019 +0100
testing/runcli.py: Add BST_CAS_STAGING_ROOT environment variable
This is required for testing with userchroot to create staging
directories in a system-specific prefix.
---
src/buildstream/testing/runcli.py | 7 +++++++
tox.ini | 1 +
2 files changed, 8 insertions(+)
diff --git a/src/buildstream/testing/runcli.py b/src/buildstream/testing/runcli.py
index c0e278b..1e86860 100644
--- a/src/buildstream/testing/runcli.py
+++ b/src/buildstream/testing/runcli.py
@@ -842,6 +842,13 @@ def configured(directory, config=None):
if not config.get("logdir", False):
config["logdir"] = os.path.join(directory, "logs")
+ cas_stage_root = os.environ.get("BST_CAS_STAGING_ROOT")
+ if cas_stage_root:
+ symlink_path = os.path.join(config["cachedir"], "cas", "staging")
+ if not os.path.lexists(symlink_path):
+ os.makedirs(os.path.join(config["cachedir"], "cas"), exist_ok=True)
+ os.symlink(cas_stage_root, symlink_path)
+
# Dump it and yield the filename for test scripts to feed it
# to buildstream as an artument
filename = os.path.join(directory, "buildstream.conf")
diff --git a/tox.ini b/tox.ini
index b037e39..62737b0 100644
--- a/tox.ini
+++ b/tox.ini
@@ -39,6 +39,7 @@ deps =
randomized: pytest-random-order
passenv =
ARTIFACT_CACHE_SERVICE
+ BST_CAS_STAGING_ROOT
BST_FORCE_BACKEND
BST_FORCE_SANDBOX
BST_FORCE_START_METHOD