You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by GitBox <gi...@apache.org> on 2022/08/25 11:06:12 UTC
[GitHub] [buildstream] ssssam opened a new pull request, #1741: Draft: Update install instructions for final BuildStream 2.0 release
ssssam opened a new pull request, #1741:
URL: https://github.com/apache/buildstream/pull/1741
See commits for rationale for the various changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#issuecomment-1231713842
Updated following the above guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955092100
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
Looks to me it's `podman --log-level error exec --detach-keys --interactive --tty --user username --workdir /var/home/username/git/freedesktop-sdk --env=COLORTERM=truecolor --env=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus --env=DESKTOP_SESSION=gnome --env=DISPLAY=:0 --env=LANG=en_US.UTF-8 --env=SHELL=/bin/bash --env=SSH_AUTH_SOCK=/run/user/1000/keyring/ssh --env=TERM=xterm-256color --env=TOOLBOX_PATH=/usr/bin/toolbox --env=USER=username --env=VTE_VERSION=6800 --env=WAYLAND_DISPLAY=wayland-0 --env=XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.NU99Q1 --env=XDG_CURRENT_DESKTOP=GNOME --env=XDG_DATA_DIRS=/var/home/username/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/ --env=XDG_MENU_PREFIX=gnome- --env=XDG_RUNTIME_DIR=/run/user/1000 --env=XDG_SESSION_DESKTOP=gnome --env=XDG_SESSION_TYPE=wayland bst2 capsh --caps= -- -c exec "$@" /bin/sh /bin/bash -l` more or less.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955103684
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
You can check with `registry.gitlab.com/freedesktop-sdk/infrastructure/freedesktop-sdk-docker-images/bst2:latest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r975306329
##########
doc/source/main_install.rst:
##########
@@ -1,221 +1,182 @@
-Installing
-==========
+Installing from Source
+======================
-Until BuildStream is available in your distro, you may need to install
-it yourself from source. The necessary steps are:
+This page explains how to build and install this version of BuildStream from
+source. For general purpose installation instructions consult the
+`website <https://buildstream.build/install.html>`_.
+
+For full install instructions, read on:
* :ref:`Install dependencies<install-dependencies>`
* :ref:`Install BuildBox<install-buildbox>`
-* :ref:`Install BuildStream<install-buildstream>` (from a git checkout, or from PyPi)
-* :ref:`Update PATH<post-install>`
-
-Alternatively, BuildStream can be run in :ref:`a container<install-container>`.
-
+* :ref:`Install BuildStream<install-buildstream>`
+* :ref:`Install completions<install-completions>`
.. _install-dependencies:
Installing Dependencies
-----------------------
-Before installing BuildStream from source, it is necessary to first install
-the system dependencies. Below are some linux distribution specific instructions
-for installing these dependencies.
+Runtime requirements
+~~~~~~~~~~~~~~~~~~~~
-BuildStream requires the following base system requirements:
+BuildStream requires the following Python environment to run:
- python3 >= 3.7
-- pip
-- lzip (optional, for ``.tar.lz`` support)
-- :ref:`buildbox-casd<install-buildbox>`
-
-BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
-Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
-requirements of specific plugins.
-
-
-Arch Linux
-~~~~~~~~~~
-Install the dependencies with::
+- PyPI packages as specified in
+ `requirements.txt <https://github.com/apache/buildstream/blob/master/requirements/requirements.txt>`_.
+Some :mod:`Source <buildstream.source>` plugins require specific tools installed
+on the host. Here is a commonly used subset based on the
+:ref:`core source plugins <plugins_sources>`
+and `buildstream-plugins <https://apache.github.io/buildstream-plugins/>`_.
- sudo pacman -S python python-pip
+- git (for ``git`` sources)
+- lzip (for ``.tar.lz`` support in ``tar`` sources)
+- patch (for ``patch`` sources)
+Some BuildBox tools used by BuildStream require additional host tools:
-Debian
-~~~~~~
-Install the dependencies with::
+- bubblewrap (for ``buildbox-run-bubblewrap``)
+- fusermount3 (for ``buildbox-fuse``)
+Install-time requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~
- sudo apt-get install \
- python3 python3-pip python3-dev
+BuildStream contains Cython code which implies the following extra
+dependencies at install-time only:
+- C and C++ toolchain
+- Python development headers
-Fedora
-~~~~~~
-For recent fedora systems, the following line should get you the system
-requirements you need::
+These instructions use ``pip3`` to install necessary PyPI packages.
+Packagers and integrators may use a different tool and can ignore
+the `pip` dependency below.
+Distribution-specific guides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- dnf install -y \
- python3 python3-pip python3-devel
+This table gives you a list of packages for specific distros:
+.. list-table::
-Ubuntu
-~~~~~~
-Install the dependencies with::
-
-
- sudo apt install \
- python3 python3-pip python3-dev
-
+ * - **Distribution**
+ - **Runtime requires**
+ - **Install requires**
+ * - Arch Linux
+ - bubblewrap fuse3 git lzip patch python
+ - gcc python-pip
+ * - Debian
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
+ * - Fedora
+ - bubblewrap fuse3 git lzip patch python3
+ - gcc-c++ python3-devel python3-pip
+ * - Ubuntu
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
.. _install-buildbox:
Installing BuildBox
-------------------
-BuildStream master now depends on buildbox-casd to manage the local CAS cache
-and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox
-components are still in development and there are no stable releases yet.
-Thus, they're not available yet in Linux distros and they have to be manually
-installed.
+BuildStream depends on the following tools from
+`BuildBox <https://gitlab.com/BuildGrid/buildbox/>`_:
+
+ * ``buildbox-casd`` (to manage local and remote content-addressed storage)
+ * ``buildbox-fuse`` (to check out content from the local CAS)
+ * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox)
These components can be installed from binaries, or built from source.
Install binaries
~~~~~~~~~~~~~~~~
-Linux x86-64 users can download the `latest statically linked binaries here
-<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/binaries.tgz>`_,
-or browse the `release history of static binaries here
+Browse the `release history of static binaries here
<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases>`_.
-The contents of the ``binaries.tgz`` tarball should be extracted into a directory
-in ``PATH``, e.g., ``~/.local/bin``.
+Linux x86-64 users can download the `latest statically linked binaries here
+<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/buildbox-x86_64-linux-gnu.tgz>`_,
+The contents of the tarball should be extracted into a directory in ``PATH``,
+e.g., ``~/.local/bin``.
Build from source
~~~~~~~~~~~~~~~~~
Each of the 4 buildbox components can be installed separately from their
respective git repositiories, and each respository has individual install
-instructions. Make sure that you're installing the correct version of
-each component.
+instructions. We recommend installing the latest release tag of each
+component.
| **Buildbox-common:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-casd:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-fuse:** See
-| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst
-| (Be sure to install from the 0.0.14 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-run-bublewrap:** See the installation section in:
| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst
-| (Be sure to install from the 0.0.8 tag.)
+| (Be sure to install from the latest stable release tag.)
Finally, configure buildbox-run-bubblewrap as the default buildbox-run
implementation::
ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run
-.. _install-buildstream:
-
-Installing BuildStream
-----------------------
-
-Installing from PyPI
-~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
-
- pip3 install --user --pre BuildStream
-
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
-
-.. note::
- At time of writing, BuildStream 2 is only available as dev snapshots; this
- is why the ``--pre`` option is required. Running
- ``pip3 install --user BuildStream`` (without the ``--pre`` option)
- will install Buildsteam 1.
-
-You can also install a specific dev snapshot of Buildstream by specifying the
-version. eg ``pip3 install --user BuildStream==1.93.2.dev0``.
-Available versions can be found on the BuildStream history page `on PyPi
-<https://pypi.org/project/BuildStream/#history>`_.
-Note that some of the oldest versions are not available on PyPI.
-
-Keep following the :ref:`instructions below<post-install>` to ensure that the ``bst``
-command is in your ``PATH``.
-
-Upgrading from PyPI
-+++++++++++++++++++
-Once you have already installed BuildStream from PyPI, you can later update
-to the latest dev snapshot like so::
-
-
- pip3 install --user --upgrade --pre BuildStream
-
-
-
-Installing from a git checkout
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To install directly from the `git repository <https://github.com/apache/buildstream>`_
-using python's ``pip`` package manager, you will additionally require:
-
-- pip for python3 (only required for setup)
-- Python 3 development libraries and headers
-- git (to checkout BuildStream)
+.. _install-buildstream:
-Before installing, please check the existing tags in the git repository
-and determine which version you want to install.
+Installing BuildStream from a git checkout
+------------------------------------------
-Run the following commands::
+First, clone the repository. Please check the existing tags in the git
+repository and determine which version you want to install::
git clone https://github.com/apache/buildstream.git
cd buildstream
git checkout <desired release tag>
+
+We recommend ``pip`` as a frontend to the underlying ``setuptools`` build
+system. The following command will build and install BuildStream into your
+user's homedir in ``~/.local``, and will attempt to fetch and install any
+required PyPI dependencies from the internet at the same time::
+
+
pip3 install --user .
-This will install BuildStream's pure python dependencies into
-your user's homedir in ``~/.local`` and will run BuildStream directly
-from the git checkout directory.
+You can optionally use Pip's
Review Comment:
Opened https://github.com/apache/buildstream/issues/1760 so we have somewhere to answer the question "Why can't I use `pip -e`" and will link that from the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955114743
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
I also see differences in `bwrap` behaviour with and without `--privileged`:
Without --privileged:
```
# bwrap --unshare-all --ro-bind / / --dev-bind /dev /dev --uid 0 /bin/bash
bwrap: Can't bind mount /oldroot/ on /newroot/: Unable to mount source on destination: Operation not permitted
```
With --privileged: I get a running `bash` instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955046159
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
With podman at least this is not required. We (freedesktop-sdk) have an image that works *just fine* in unprivileged toolbox under user namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan merged pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
gtristan merged PR #1741:
URL: https://github.com/apache/buildstream/pull/1741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
gtristan commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r972075940
##########
doc/source/main_install.rst:
##########
@@ -1,221 +1,182 @@
-Installing
-==========
+Installing from Source
+======================
-Until BuildStream is available in your distro, you may need to install
-it yourself from source. The necessary steps are:
+This page explains how to build and install this version of BuildStream from
+source. For general purpose installation instructions consult the
+`website <https://buildstream.build/install.html>`_.
+
+For full install instructions, read on:
* :ref:`Install dependencies<install-dependencies>`
* :ref:`Install BuildBox<install-buildbox>`
-* :ref:`Install BuildStream<install-buildstream>` (from a git checkout, or from PyPi)
-* :ref:`Update PATH<post-install>`
-
-Alternatively, BuildStream can be run in :ref:`a container<install-container>`.
-
+* :ref:`Install BuildStream<install-buildstream>`
+* :ref:`Install completions<install-completions>`
.. _install-dependencies:
Installing Dependencies
-----------------------
-Before installing BuildStream from source, it is necessary to first install
-the system dependencies. Below are some linux distribution specific instructions
-for installing these dependencies.
+Runtime requirements
+~~~~~~~~~~~~~~~~~~~~
-BuildStream requires the following base system requirements:
+BuildStream requires the following Python environment to run:
- python3 >= 3.7
-- pip
-- lzip (optional, for ``.tar.lz`` support)
-- :ref:`buildbox-casd<install-buildbox>`
-
-BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
-Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
-requirements of specific plugins.
-
-
-Arch Linux
-~~~~~~~~~~
-Install the dependencies with::
+- PyPI packages as specified in
+ `requirements.txt <https://github.com/apache/buildstream/blob/master/requirements/requirements.txt>`_.
+Some :mod:`Source <buildstream.source>` plugins require specific tools installed
+on the host. Here is a commonly used subset based on the
+:ref:`core source plugins <plugins_sources>`
+and `buildstream-plugins <https://apache.github.io/buildstream-plugins/>`_.
- sudo pacman -S python python-pip
+- git (for ``git`` sources)
+- lzip (for ``.tar.lz`` support in ``tar`` sources)
+- patch (for ``patch`` sources)
+Some BuildBox tools used by BuildStream require additional host tools:
-Debian
-~~~~~~
-Install the dependencies with::
+- bubblewrap (for ``buildbox-run-bubblewrap``)
+- fusermount3 (for ``buildbox-fuse``)
+Install-time requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~
- sudo apt-get install \
- python3 python3-pip python3-dev
+BuildStream contains Cython code which implies the following extra
+dependencies at install-time only:
+- C and C++ toolchain
+- Python development headers
-Fedora
-~~~~~~
-For recent fedora systems, the following line should get you the system
-requirements you need::
+These instructions use ``pip3`` to install necessary PyPI packages.
+Packagers and integrators may use a different tool and can ignore
+the `pip` dependency below.
+Distribution-specific guides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- dnf install -y \
- python3 python3-pip python3-devel
+This table gives you a list of packages for specific distros:
+.. list-table::
-Ubuntu
-~~~~~~
-Install the dependencies with::
-
-
- sudo apt install \
- python3 python3-pip python3-dev
-
+ * - **Distribution**
+ - **Runtime requires**
+ - **Install requires**
+ * - Arch Linux
+ - bubblewrap fuse3 git lzip patch python
+ - gcc python-pip
+ * - Debian
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
+ * - Fedora
+ - bubblewrap fuse3 git lzip patch python3
+ - gcc-c++ python3-devel python3-pip
+ * - Ubuntu
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
.. _install-buildbox:
Installing BuildBox
-------------------
-BuildStream master now depends on buildbox-casd to manage the local CAS cache
-and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox
-components are still in development and there are no stable releases yet.
-Thus, they're not available yet in Linux distros and they have to be manually
-installed.
+BuildStream depends on the following tools from
+`BuildBox <https://gitlab.com/BuildGrid/buildbox/>`_:
+
+ * ``buildbox-casd`` (to manage local and remote content-addressed storage)
+ * ``buildbox-fuse`` (to check out content from the local CAS)
+ * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox)
These components can be installed from binaries, or built from source.
Install binaries
~~~~~~~~~~~~~~~~
-Linux x86-64 users can download the `latest statically linked binaries here
-<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/binaries.tgz>`_,
-or browse the `release history of static binaries here
+Browse the `release history of static binaries here
<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases>`_.
-The contents of the ``binaries.tgz`` tarball should be extracted into a directory
-in ``PATH``, e.g., ``~/.local/bin``.
+Linux x86-64 users can download the `latest statically linked binaries here
+<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/buildbox-x86_64-linux-gnu.tgz>`_,
+The contents of the tarball should be extracted into a directory in ``PATH``,
+e.g., ``~/.local/bin``.
Build from source
~~~~~~~~~~~~~~~~~
Each of the 4 buildbox components can be installed separately from their
respective git repositiories, and each respository has individual install
-instructions. Make sure that you're installing the correct version of
-each component.
+instructions. We recommend installing the latest release tag of each
+component.
| **Buildbox-common:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-casd:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-fuse:** See
-| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst
-| (Be sure to install from the 0.0.14 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-run-bublewrap:** See the installation section in:
| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst
-| (Be sure to install from the 0.0.8 tag.)
+| (Be sure to install from the latest stable release tag.)
Finally, configure buildbox-run-bubblewrap as the default buildbox-run
implementation::
ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run
-.. _install-buildstream:
-
-Installing BuildStream
-----------------------
-
-Installing from PyPI
-~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
-
- pip3 install --user --pre BuildStream
-
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
-
-.. note::
- At time of writing, BuildStream 2 is only available as dev snapshots; this
- is why the ``--pre`` option is required. Running
- ``pip3 install --user BuildStream`` (without the ``--pre`` option)
- will install Buildsteam 1.
-
-You can also install a specific dev snapshot of Buildstream by specifying the
-version. eg ``pip3 install --user BuildStream==1.93.2.dev0``.
-Available versions can be found on the BuildStream history page `on PyPi
-<https://pypi.org/project/BuildStream/#history>`_.
-Note that some of the oldest versions are not available on PyPI.
-
-Keep following the :ref:`instructions below<post-install>` to ensure that the ``bst``
-command is in your ``PATH``.
-
-Upgrading from PyPI
-+++++++++++++++++++
-Once you have already installed BuildStream from PyPI, you can later update
-to the latest dev snapshot like so::
-
-
- pip3 install --user --upgrade --pre BuildStream
-
-
-
-Installing from a git checkout
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To install directly from the `git repository <https://github.com/apache/buildstream>`_
-using python's ``pip`` package manager, you will additionally require:
-
-- pip for python3 (only required for setup)
-- Python 3 development libraries and headers
-- git (to checkout BuildStream)
+.. _install-buildstream:
-Before installing, please check the existing tags in the git repository
-and determine which version you want to install.
+Installing BuildStream from a git checkout
+------------------------------------------
-Run the following commands::
+First, clone the repository. Please check the existing tags in the git
+repository and determine which version you want to install::
git clone https://github.com/apache/buildstream.git
cd buildstream
git checkout <desired release tag>
+
+We recommend ``pip`` as a frontend to the underlying ``setuptools`` build
+system. The following command will build and install BuildStream into your
+user's homedir in ``~/.local``, and will attempt to fetch and install any
+required PyPI dependencies from the internet at the same time::
+
+
pip3 install --user .
-This will install BuildStream's pure python dependencies into
-your user's homedir in ``~/.local`` and will run BuildStream directly
-from the git checkout directory.
+You can optionally use Pip's
Review Comment:
I don't believe the `-e` option has worked for a very long time (with buildstream specifically, this might have been since the transition to use cython).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r972118277
##########
doc/source/main_install.rst:
##########
@@ -1,221 +1,182 @@
-Installing
-==========
+Installing from Source
+======================
-Until BuildStream is available in your distro, you may need to install
-it yourself from source. The necessary steps are:
+This page explains how to build and install this version of BuildStream from
+source. For general purpose installation instructions consult the
+`website <https://buildstream.build/install.html>`_.
+
+For full install instructions, read on:
* :ref:`Install dependencies<install-dependencies>`
* :ref:`Install BuildBox<install-buildbox>`
-* :ref:`Install BuildStream<install-buildstream>` (from a git checkout, or from PyPi)
-* :ref:`Update PATH<post-install>`
-
-Alternatively, BuildStream can be run in :ref:`a container<install-container>`.
-
+* :ref:`Install BuildStream<install-buildstream>`
+* :ref:`Install completions<install-completions>`
.. _install-dependencies:
Installing Dependencies
-----------------------
-Before installing BuildStream from source, it is necessary to first install
-the system dependencies. Below are some linux distribution specific instructions
-for installing these dependencies.
+Runtime requirements
+~~~~~~~~~~~~~~~~~~~~
-BuildStream requires the following base system requirements:
+BuildStream requires the following Python environment to run:
- python3 >= 3.7
-- pip
-- lzip (optional, for ``.tar.lz`` support)
-- :ref:`buildbox-casd<install-buildbox>`
-
-BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
-Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
-requirements of specific plugins.
-
-
-Arch Linux
-~~~~~~~~~~
-Install the dependencies with::
+- PyPI packages as specified in
+ `requirements.txt <https://github.com/apache/buildstream/blob/master/requirements/requirements.txt>`_.
+Some :mod:`Source <buildstream.source>` plugins require specific tools installed
+on the host. Here is a commonly used subset based on the
+:ref:`core source plugins <plugins_sources>`
+and `buildstream-plugins <https://apache.github.io/buildstream-plugins/>`_.
- sudo pacman -S python python-pip
+- git (for ``git`` sources)
+- lzip (for ``.tar.lz`` support in ``tar`` sources)
+- patch (for ``patch`` sources)
+Some BuildBox tools used by BuildStream require additional host tools:
-Debian
-~~~~~~
-Install the dependencies with::
+- bubblewrap (for ``buildbox-run-bubblewrap``)
+- fusermount3 (for ``buildbox-fuse``)
+Install-time requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~
- sudo apt-get install \
- python3 python3-pip python3-dev
+BuildStream contains Cython code which implies the following extra
+dependencies at install-time only:
+- C and C++ toolchain
+- Python development headers
-Fedora
-~~~~~~
-For recent fedora systems, the following line should get you the system
-requirements you need::
+These instructions use ``pip3`` to install necessary PyPI packages.
+Packagers and integrators may use a different tool and can ignore
+the `pip` dependency below.
+Distribution-specific guides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- dnf install -y \
- python3 python3-pip python3-devel
+This table gives you a list of packages for specific distros:
+.. list-table::
-Ubuntu
-~~~~~~
-Install the dependencies with::
-
-
- sudo apt install \
- python3 python3-pip python3-dev
-
+ * - **Distribution**
+ - **Runtime requires**
+ - **Install requires**
+ * - Arch Linux
+ - bubblewrap fuse3 git lzip patch python
+ - gcc python-pip
+ * - Debian
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
+ * - Fedora
+ - bubblewrap fuse3 git lzip patch python3
+ - gcc-c++ python3-devel python3-pip
+ * - Ubuntu
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
.. _install-buildbox:
Installing BuildBox
-------------------
-BuildStream master now depends on buildbox-casd to manage the local CAS cache
-and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox
-components are still in development and there are no stable releases yet.
-Thus, they're not available yet in Linux distros and they have to be manually
-installed.
+BuildStream depends on the following tools from
+`BuildBox <https://gitlab.com/BuildGrid/buildbox/>`_:
+
+ * ``buildbox-casd`` (to manage local and remote content-addressed storage)
+ * ``buildbox-fuse`` (to check out content from the local CAS)
+ * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox)
These components can be installed from binaries, or built from source.
Install binaries
~~~~~~~~~~~~~~~~
-Linux x86-64 users can download the `latest statically linked binaries here
-<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/binaries.tgz>`_,
-or browse the `release history of static binaries here
+Browse the `release history of static binaries here
<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases>`_.
-The contents of the ``binaries.tgz`` tarball should be extracted into a directory
-in ``PATH``, e.g., ``~/.local/bin``.
+Linux x86-64 users can download the `latest statically linked binaries here
+<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/buildbox-x86_64-linux-gnu.tgz>`_,
+The contents of the tarball should be extracted into a directory in ``PATH``,
+e.g., ``~/.local/bin``.
Build from source
~~~~~~~~~~~~~~~~~
Each of the 4 buildbox components can be installed separately from their
respective git repositiories, and each respository has individual install
-instructions. Make sure that you're installing the correct version of
-each component.
+instructions. We recommend installing the latest release tag of each
+component.
| **Buildbox-common:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-casd:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-fuse:** See
-| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst
-| (Be sure to install from the 0.0.14 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-run-bublewrap:** See the installation section in:
| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst
-| (Be sure to install from the 0.0.8 tag.)
+| (Be sure to install from the latest stable release tag.)
Finally, configure buildbox-run-bubblewrap as the default buildbox-run
implementation::
ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run
-.. _install-buildstream:
-
-Installing BuildStream
-----------------------
-
-Installing from PyPI
-~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
-
- pip3 install --user --pre BuildStream
-
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
-
-.. note::
- At time of writing, BuildStream 2 is only available as dev snapshots; this
- is why the ``--pre`` option is required. Running
- ``pip3 install --user BuildStream`` (without the ``--pre`` option)
- will install Buildsteam 1.
-
-You can also install a specific dev snapshot of Buildstream by specifying the
-version. eg ``pip3 install --user BuildStream==1.93.2.dev0``.
-Available versions can be found on the BuildStream history page `on PyPi
-<https://pypi.org/project/BuildStream/#history>`_.
-Note that some of the oldest versions are not available on PyPI.
-
-Keep following the :ref:`instructions below<post-install>` to ensure that the ``bst``
-command is in your ``PATH``.
-
-Upgrading from PyPI
-+++++++++++++++++++
-Once you have already installed BuildStream from PyPI, you can later update
-to the latest dev snapshot like so::
-
-
- pip3 install --user --upgrade --pre BuildStream
-
-
-
-Installing from a git checkout
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To install directly from the `git repository <https://github.com/apache/buildstream>`_
-using python's ``pip`` package manager, you will additionally require:
-
-- pip for python3 (only required for setup)
-- Python 3 development libraries and headers
-- git (to checkout BuildStream)
+.. _install-buildstream:
-Before installing, please check the existing tags in the git repository
-and determine which version you want to install.
+Installing BuildStream from a git checkout
+------------------------------------------
-Run the following commands::
+First, clone the repository. Please check the existing tags in the git
+repository and determine which version you want to install::
git clone https://github.com/apache/buildstream.git
cd buildstream
git checkout <desired release tag>
+
+We recommend ``pip`` as a frontend to the underlying ``setuptools`` build
+system. The following command will build and install BuildStream into your
+user's homedir in ``~/.local``, and will attempt to fetch and install any
+required PyPI dependencies from the internet at the same time::
+
+
pip3 install --user .
-This will install BuildStream's pure python dependencies into
-your user's homedir in ``~/.local`` and will run BuildStream directly
-from the git checkout directory.
+You can optionally use Pip's
Review Comment:
OK, will remove. In fact, probably better to specifically note that it can't be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
gtristan commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955445509
##########
doc/source/main_install.rst:
##########
@@ -130,28 +116,23 @@ Installing BuildStream
Installing from PyPI
~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
+For the latest pre-release of BuildStream 2, including the necessary Python
+dependencies and BuildBox tools, run the following command::
pip3 install --user --pre BuildStream
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
+This will install BuildStream and its dependencies into your user's homedir in
+``~/.local``. Pip will use binary "wheel" packages from PyPI where these are
Review Comment:
Nit:
> Pip will use binary "wheel" packages from PyPI where these are available for your platform.
Would probably read better with *"if these are available for your platform"* (whereas *"where"* would be more suitable if *"for your platform"* was removed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#issuecomment-1252305803
Last 2 review comments now addressed in 7925769d9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955094659
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
Right, so --privileged I think means different things here. With Docker it means "you can privilege escalate to root", with podman it means "you can privilege escalate to user".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r958487960
##########
doc/source/main_install.rst:
##########
@@ -130,28 +116,23 @@ Installing BuildStream
Installing from PyPI
~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
+For the latest pre-release of BuildStream 2, including the necessary Python
+dependencies and BuildBox tools, run the following command::
pip3 install --user --pre BuildStream
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
+This will install BuildStream and its dependencies into your user's homedir in
+``~/.local``. Pip will use binary "wheel" packages from PyPI where these are
Review Comment:
Removed in new version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955108207
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
I didn't get as far as creating namespaces as it turns out `fusermount3` can't mount:
```
> podman run --device /dev/fuse -i -t --rm registry.gitlab.com/freedesktop-sdk/infrastructure/freedesktop-sdk-docker-images/bst2
[root@de014e636534 /]# bst --version
1.95.2.dev0
[root@de014e636534 /]# cd
[root@de014e636534 ~]# git clone https://github.com/apache/buildstream
[root@de014e636534 buildstream]# cd doc/examples/running-commands/
[root@de014e636534 running-commands]# bst build hello.bst
...
Failure Summary
hello.bst:
[00:00:00][8cebf405][ build:hello.bst ] FAILURE buildbox-run failed with returncode 1
Printing the last 20 lines from log file:
/root/.cache/buildstream/logs/running-commands/hello/8cebf405-build.110.log
======================================================================
LC_ALL: C
HOME: /tmp
TZ: UTC
SOURCE_DATE_EPOCH: 1321009871
[--:--:--] START [8cebf405] hello.bst: Staging dependencies at: /
[--:--:--] STATUS [b59f4520] base/alpine.bst: Staging base/alpine.bst/b59f4520
[--:--:--] STATUS [bb393e5a] base.bst: Staging base.bst/bb393e5a
[00:00:00] SUCCESS [8cebf405] hello.bst: Staging dependencies at: /
[--:--:--] START [8cebf405] hello.bst: Integrating sandbox
[00:00:00] SUCCESS [8cebf405] hello.bst: Integrating sandbox
[--:--:--] START [8cebf405] hello.bst: Staging sources
[00:00:00] SUCCESS [8cebf405] hello.bst: Staging sources
[--:--:--] START hello.bst: Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
2022-08-25T15:20:08.005+0000 [191:139995975073728] [buildboxcommon_casclient.cpp:95] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
2022-08-25T15:20:08.155+0000 [191:139995975073728] [buildboxcommon_runner.cpp:437] [ERROR] [actionDigest=ace830804d554a11f87ee8018bde83888d28775d4208e1798a010fdec4cd680f/178] Error executing command: std::runtime_error exception thrown at [buildboxcommon_casclient.cpp:900], errMsg = "Error staging "63db766975a429f55445f8e0a24f4107be339b311f002cb631841a71d91234ed/1438" into "": "std::runtime_error exception thrown at [buildboxcasd_fusestager.cpp:144], errMsg = "The FUSE stager child process unexpectedly died with exit code 1"""
[00:00:00] FAILURE hello.bst: Running commands
[00:00:00] FAILURE [8cebf405] hello.bst: buildbox-run failed with returncode 1
======================================================================
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 3, skipped 0, failed 0
Build Queue: processed 2, skipped 0, failed 1
[root@de014e636534 running-commands]# mkdir cas local
[root@de014e636534 running-commands]# buildbox-fuse --local=cas local
fusermount3: mount failed: Operation not permitted
```
So it seems I can't usefully build stuff if I don't supply `--privileged` to podman. Adding that to the commandline got me a successful build.
anyway, I will check if `bwrap` does work unprivileged and reword the doc. thanks for the pointers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955108207
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
I didn't get as far as creating namespaces as it turns out `fusermount3` can't mount:
```
> podman run --device /dev/fuse -i -t --rm registry.gitlab.com/freedesktop-sdk/infrastructure/freedesktop-sdk-docker-images/bst2
[root@de014e636534 /]# bst --version
1.95.2.dev0
[root@de014e636534 /]# cd
[root@de014e636534 ~]# git clone https://github.com/apache/buildstream
[root@de014e636534 buildstream]# cd doc/examples/running-commands/
[root@de014e636534 running-commands]# bst build hello.bst
...
Failure Summary
hello.bst:
[00:00:00][8cebf405][ build:hello.bst ] FAILURE buildbox-run failed with returncode 1
Printing the last 20 lines from log file:
/root/.cache/buildstream/logs/running-commands/hello/8cebf405-build.110.log
======================================================================
LC_ALL: C
HOME: /tmp
TZ: UTC
SOURCE_DATE_EPOCH: 1321009871
[--:--:--] START [8cebf405] hello.bst: Staging dependencies at: /
[--:--:--] STATUS [b59f4520] base/alpine.bst: Staging base/alpine.bst/b59f4520
[--:--:--] STATUS [bb393e5a] base.bst: Staging base.bst/bb393e5a
[00:00:00] SUCCESS [8cebf405] hello.bst: Staging dependencies at: /
[--:--:--] START [8cebf405] hello.bst: Integrating sandbox
[00:00:00] SUCCESS [8cebf405] hello.bst: Integrating sandbox
[--:--:--] START [8cebf405] hello.bst: Staging sources
[00:00:00] SUCCESS [8cebf405] hello.bst: Staging sources
[--:--:--] START hello.bst: Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
2022-08-25T15:20:08.005+0000 [191:139995975073728] [buildboxcommon_casclient.cpp:95] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
2022-08-25T15:20:08.155+0000 [191:139995975073728] [buildboxcommon_runner.cpp:437] [ERROR] [actionDigest=ace830804d554a11f87ee8018bde83888d28775d4208e1798a010fdec4cd680f/178] Error executing command: std::runtime_error exception thrown at [buildboxcommon_casclient.cpp:900], errMsg = "Error staging "63db766975a429f55445f8e0a24f4107be339b311f002cb631841a71d91234ed/1438" into "": "std::runtime_error exception thrown at [buildboxcasd_fusestager.cpp:144], errMsg = "The FUSE stager child process unexpectedly died with exit code 1"""
[00:00:00] FAILURE hello.bst: Running commands
[00:00:00] FAILURE [8cebf405] hello.bst: buildbox-run failed with returncode 1
======================================================================
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 3, skipped 0, failed 0
Build Queue: processed 2, skipped 0, failed 1
[root@de014e636534 running-commands]# mkdir cas local
[root@de014e636534 running-commands]# buildbox-fuse --local=cas local
fusermount3: mount failed: Operation not permitted
```
So it seems I can't usefully build stuff if I don't supply `--privileged` to podman. Adding that to the commandline got me a successful build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955085456
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
I suspect Toolbx containers are `--privileged` by default. I had a quick look at the code, and it certainly passes `--privileged` to Podman in a couple of places. User namespaces require at least CAP_SYS_ADMIN, and maybe more capabilities that your random "run container for free" CI service often won't allow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#issuecomment-1231527256
I'm taking another look at this today, thanks for the detailed guidance.
On dependencies and package lists, I will move them into a table, but we should still separate the install-time requirements (gcc, pip, python3-devel) from the runtime requirements. Let's try a table with 2 columns.
About a suitable 'install' command for distro packagers, it's useful to give some guidance on this. The `python3 setup.py install` command will work (as long as Bst doesn't change buildsystem), but it may not be what distro package guidelines recommend now. did some quick research to satisfy my own curiousity:
* Fedora recommends use of [a %pyproject_install macro](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_build_macros), which runs `pip install --no-deps --no-index --use-pep517 .` ([link](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_60))
* Debian doesn't give any strong recommendation, the existing [buildstream package](https://tracker.debian.org/pkg/buildstream) uses [pybuild](https://manpages.debian.org/bullseye/dh-python/pybuild.1.en.html) via debhelper, which autodetects one of multiple backends. Running `pybuild --detect` in buildstream.git reports `distutils`, so they do indeed [run `setup.py` directly](https://salsa.debian.org/python-team/tools/dh-python/-/blob/master/dhpython/build/plugin_distutils.py), although there is a `pyproject` backend that may become default in future which [runs `python3 -m build`](https://salsa.debian.org/python-team/tools/dh-python/-/blob/master/dhpython/build/plugin_pyproject.py).
All this to highlight there's no "one size fits all" distro packager command, they should use whatever the Python guidelines for that distro recommends, and that can be either running setuptools directly or any PEP517 compatible frontend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
gtristan commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r972089510
##########
doc/source/main_install.rst:
##########
@@ -258,40 +219,23 @@ and set up other necessary environment variables::
If you do not want to manage your virtual environments manually, you can
consider using `pipx <https://docs.python.org/3/tutorial/venv.html>`_.
+.. _install-completions:
-.. _post-install:
-
-Post-install setup
-------------------
-
-After having installed from source using any of the above methods, some
-setup will be required to use BuildStream.
-
-
-
-Adjust ``PATH``
-~~~~~~~~~~~~~~~
-Since BuildStream is now installed under your local user's install directories,
-you need to ensure that ``PATH`` is adjusted.
-
-A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
-
- export PATH="${PATH}:${HOME}/.local/bin"
-
-.. note::
+Installing completions
+----------------------
- You will have to restart your terminal in order for these changes to take effect.
+BuildStream integrates with Bash to provide helpful tab-completion. These
+are provided by the script ``src/buildstream/data/bst`` which requires
Review Comment:
Wouldn't it be good to expand the content of `src/buildstream/data/bst` here (with `::literalinclude`) or at least provide a link to the master branch version of the `bst` completions script on github ?
Also I thought there was an easier way to get this working for a single user, by simply copying the content of the script (from this page) and pasting the content directly into `~/.bash_completion`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955095599
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
The whole point of user namespaces is they work without privileges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] abderrahim commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
abderrahim commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r956551512
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
@nanonyme AFAIK toolbx uses a `--privileged` podman container. You're probably confusing rootless with non-privileged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
gtristan commented on PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#issuecomment-1227820050
Overall, I like the install from source instructions here, however it looks like this is unfortunately again growing into a lot of overlap with the user facing instructions at: https://buildstream.build/install.html
---
I wonder if the statement about *"Installing from distro packages if it is available for your distro"* and the whole convenience packaging method of *"Installation from pip"* is better suited to https://buildstream.build/install.html ... where it appears the pip install method is already mentioned.
Lets keep the user facing install paths published at https://buildstream.build/install.html and have the version specific developer/packager facing instructions at https://docs.buildstream.build/master/main_install.html. This makes the two pages serve obviously different purposes, and makes the reading of either of these less crowded and intense.
---
It could be good to have a forward in `main_install.rst` saying something like:
> This page explains how to build and install this version of BuildStream from sources. If you are trying to install BuildStream on your computer it is recommended to try following the [general purpose installation instructions here](https://buildstream.build/install.html).
---
This also goes for the section on *Using docker images*, if there are user facing images ready to be used by users, we should trust https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md to document how to use the latest BuildStream 1 and BuildStream 2 images respectively, and link to that page from the user facing documentation at https://buildstream.build/install.html (without going into a whole detailed shpiel about it)... and of course we should touch up https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md a bit to make sure it at least says `docker run -it -v `pwd`:/build <bst1imagename:latest>` and `docker run -it -v `pwd`:/build <bst2imagename:latest>`.
---
After trimming out all that fat above, this would leave us with a more manageable structure of only:
* Dependencies for building/running this version of buildstream
* This would look so much more concise as a condensed table with two columns: *"Distro name"*, and *"Distro specific command"* instead of scrolling down lots of text talking about multiple distros
* We don't need to separate the dependencies with extra information about "If you need a C++ compiler" I think, this is all about building from source anyway, just specify one command for each distro would be perfect.
* Installing buildbox
* Instructions about binaries
* Link(s) to how to build buildbox from sources in upstream buildbox
* Cloning the right git tag/branch
* Installing buildstream from git checkout
* Here I would remove the whole extra shpiel about *"Upgrading from git"*, if we have instructions to install from git, then the user will figure out that they can always go back and use an updated git version to install
* Here it could be worth noting that `setup.py` installation is **supported** for both building, installing and testing, because this is the typical distro install path
* But use `pip install .` as the normal install method
* Here it is also worth noting that you can make the installation in a python virtual environment
* Here it is also worth noting that depending on the system prefix or virtual environment where you installed, you may need to set `$PATH` accordingly or activate your virtual environment
* This part allowing us to remove the conspicuous **Post-install setup** section which is all only about telling you to remember to setup `$PATH`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r972118277
##########
doc/source/main_install.rst:
##########
@@ -1,221 +1,182 @@
-Installing
-==========
+Installing from Source
+======================
-Until BuildStream is available in your distro, you may need to install
-it yourself from source. The necessary steps are:
+This page explains how to build and install this version of BuildStream from
+source. For general purpose installation instructions consult the
+`website <https://buildstream.build/install.html>`_.
+
+For full install instructions, read on:
* :ref:`Install dependencies<install-dependencies>`
* :ref:`Install BuildBox<install-buildbox>`
-* :ref:`Install BuildStream<install-buildstream>` (from a git checkout, or from PyPi)
-* :ref:`Update PATH<post-install>`
-
-Alternatively, BuildStream can be run in :ref:`a container<install-container>`.
-
+* :ref:`Install BuildStream<install-buildstream>`
+* :ref:`Install completions<install-completions>`
.. _install-dependencies:
Installing Dependencies
-----------------------
-Before installing BuildStream from source, it is necessary to first install
-the system dependencies. Below are some linux distribution specific instructions
-for installing these dependencies.
+Runtime requirements
+~~~~~~~~~~~~~~~~~~~~
-BuildStream requires the following base system requirements:
+BuildStream requires the following Python environment to run:
- python3 >= 3.7
-- pip
-- lzip (optional, for ``.tar.lz`` support)
-- :ref:`buildbox-casd<install-buildbox>`
-
-BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
-Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
-requirements of specific plugins.
-
-
-Arch Linux
-~~~~~~~~~~
-Install the dependencies with::
+- PyPI packages as specified in
+ `requirements.txt <https://github.com/apache/buildstream/blob/master/requirements/requirements.txt>`_.
+Some :mod:`Source <buildstream.source>` plugins require specific tools installed
+on the host. Here is a commonly used subset based on the
+:ref:`core source plugins <plugins_sources>`
+and `buildstream-plugins <https://apache.github.io/buildstream-plugins/>`_.
- sudo pacman -S python python-pip
+- git (for ``git`` sources)
+- lzip (for ``.tar.lz`` support in ``tar`` sources)
+- patch (for ``patch`` sources)
+Some BuildBox tools used by BuildStream require additional host tools:
-Debian
-~~~~~~
-Install the dependencies with::
+- bubblewrap (for ``buildbox-run-bubblewrap``)
+- fusermount3 (for ``buildbox-fuse``)
+Install-time requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~
- sudo apt-get install \
- python3 python3-pip python3-dev
+BuildStream contains Cython code which implies the following extra
+dependencies at install-time only:
+- C and C++ toolchain
+- Python development headers
-Fedora
-~~~~~~
-For recent fedora systems, the following line should get you the system
-requirements you need::
+These instructions use ``pip3`` to install necessary PyPI packages.
+Packagers and integrators may use a different tool and can ignore
+the `pip` dependency below.
+Distribution-specific guides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- dnf install -y \
- python3 python3-pip python3-devel
+This table gives you a list of packages for specific distros:
+.. list-table::
-Ubuntu
-~~~~~~
-Install the dependencies with::
-
-
- sudo apt install \
- python3 python3-pip python3-dev
-
+ * - **Distribution**
+ - **Runtime requires**
+ - **Install requires**
+ * - Arch Linux
+ - bubblewrap fuse3 git lzip patch python
+ - gcc python-pip
+ * - Debian
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
+ * - Fedora
+ - bubblewrap fuse3 git lzip patch python3
+ - gcc-c++ python3-devel python3-pip
+ * - Ubuntu
+ - bubblewrap fuse3 git lzip patch python3
+ - g++ python3-dev python3-pip
.. _install-buildbox:
Installing BuildBox
-------------------
-BuildStream master now depends on buildbox-casd to manage the local CAS cache
-and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox
-components are still in development and there are no stable releases yet.
-Thus, they're not available yet in Linux distros and they have to be manually
-installed.
+BuildStream depends on the following tools from
+`BuildBox <https://gitlab.com/BuildGrid/buildbox/>`_:
+
+ * ``buildbox-casd`` (to manage local and remote content-addressed storage)
+ * ``buildbox-fuse`` (to check out content from the local CAS)
+ * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox)
These components can be installed from binaries, or built from source.
Install binaries
~~~~~~~~~~~~~~~~
-Linux x86-64 users can download the `latest statically linked binaries here
-<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/binaries.tgz>`_,
-or browse the `release history of static binaries here
+Browse the `release history of static binaries here
<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases>`_.
-The contents of the ``binaries.tgz`` tarball should be extracted into a directory
-in ``PATH``, e.g., ``~/.local/bin``.
+Linux x86-64 users can download the `latest statically linked binaries here
+<https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/buildbox-x86_64-linux-gnu.tgz>`_,
+The contents of the tarball should be extracted into a directory in ``PATH``,
+e.g., ``~/.local/bin``.
Build from source
~~~~~~~~~~~~~~~~~
Each of the 4 buildbox components can be installed separately from their
respective git repositiories, and each respository has individual install
-instructions. Make sure that you're installing the correct version of
-each component.
+instructions. We recommend installing the latest release tag of each
+component.
| **Buildbox-common:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-casd:** See the installation section in:
-| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \
-| (Be sure to install from the 0.0.38 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-fuse:** See
-| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst
-| (Be sure to install from the 0.0.14 tag.)
+| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst
+| (Be sure to install from the latest stable release tag.)
| **Buildbox-run-bublewrap:** See the installation section in:
| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst
-| (Be sure to install from the 0.0.8 tag.)
+| (Be sure to install from the latest stable release tag.)
Finally, configure buildbox-run-bubblewrap as the default buildbox-run
implementation::
ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run
-.. _install-buildstream:
-
-Installing BuildStream
-----------------------
-
-Installing from PyPI
-~~~~~~~~~~~~~~~~~~~~
-Once you have the base system dependencies, you can install the BuildStream
-python package as a regular user.
-
-To install from PyPI, you will additionally require:
-
- - pip for python3 (only required for setup)
- - Python 3 development libraries and headers
-
-
-For the latest dev snapshot of BuildStream 2, simply run the following command::
-
- pip3 install --user --pre BuildStream
-
-This will install latest dev snapshot of BuildStream and its pure python
-dependencies into your user's homedir in ``~/.local``.
-
-.. note::
- At time of writing, BuildStream 2 is only available as dev snapshots; this
- is why the ``--pre`` option is required. Running
- ``pip3 install --user BuildStream`` (without the ``--pre`` option)
- will install Buildsteam 1.
-
-You can also install a specific dev snapshot of Buildstream by specifying the
-version. eg ``pip3 install --user BuildStream==1.93.2.dev0``.
-Available versions can be found on the BuildStream history page `on PyPi
-<https://pypi.org/project/BuildStream/#history>`_.
-Note that some of the oldest versions are not available on PyPI.
-
-Keep following the :ref:`instructions below<post-install>` to ensure that the ``bst``
-command is in your ``PATH``.
-
-Upgrading from PyPI
-+++++++++++++++++++
-Once you have already installed BuildStream from PyPI, you can later update
-to the latest dev snapshot like so::
-
-
- pip3 install --user --upgrade --pre BuildStream
-
-
-
-Installing from a git checkout
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To install directly from the `git repository <https://github.com/apache/buildstream>`_
-using python's ``pip`` package manager, you will additionally require:
-
-- pip for python3 (only required for setup)
-- Python 3 development libraries and headers
-- git (to checkout BuildStream)
+.. _install-buildstream:
-Before installing, please check the existing tags in the git repository
-and determine which version you want to install.
+Installing BuildStream from a git checkout
+------------------------------------------
-Run the following commands::
+First, clone the repository. Please check the existing tags in the git
+repository and determine which version you want to install::
git clone https://github.com/apache/buildstream.git
cd buildstream
git checkout <desired release tag>
+
+We recommend ``pip`` as a frontend to the underlying ``setuptools`` build
+system. The following command will build and install BuildStream into your
+user's homedir in ``~/.local``, and will attempt to fetch and install any
+required PyPI dependencies from the internet at the same time::
+
+
pip3 install --user .
-This will install BuildStream's pure python dependencies into
-your user's homedir in ``~/.local`` and will run BuildStream directly
-from the git checkout directory.
+You can optionally use Pip's
Review Comment:
OK, will remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955120374
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
Fair. So the documentation is correct, it just might just be in podman case be less privileged than docker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r975317313
##########
doc/source/main_install.rst:
##########
@@ -258,40 +219,23 @@ and set up other necessary environment variables::
If you do not want to manage your virtual environments manually, you can
consider using `pipx <https://docs.python.org/3/tutorial/venv.html>`_.
+.. _install-completions:
-.. _post-install:
-
-Post-install setup
-------------------
-
-After having installed from source using any of the above methods, some
-setup will be required to use BuildStream.
-
-
-
-Adjust ``PATH``
-~~~~~~~~~~~~~~~
-Since BuildStream is now installed under your local user's install directories,
-you need to ensure that ``PATH`` is adjusted.
-
-A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
-
- export PATH="${PATH}:${HOME}/.local/bin"
-
-.. note::
+Installing completions
+----------------------
- You will have to restart your terminal in order for these changes to take effect.
+BuildStream integrates with Bash to provide helpful tab-completion. These
+are provided by the script ``src/buildstream/data/bst`` which requires
Review Comment:
We expect that the user has a Git clone if they're reading these instructions.
I'm not convinced that 'add this text to this file' is much simpler to do than 'copy this file to this location'. It seems simpler to express it though :)
I will rework as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] nanonyme commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
nanonyme commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955046159
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
With podman at least this is not required. We have an image that works *just fine* in unprivileged toolbox under user namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on a diff in pull request #1741: Draft: Update install instructions for final BuildStream 2.0 release
Posted by GitBox <gi...@apache.org>.
ssssam commented on code in PR #1741:
URL: https://github.com/apache/buildstream/pull/1741#discussion_r955100135
##########
doc/source/main_install.rst:
##########
@@ -285,13 +327,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
.. _install-container:
-
Buildstream Inside a Container
-------------------------------
-If your system cannot provide the base requirements, it is possible to run
-BuildStream within a container. This gives you an easy way to get started
-using BuildStream on any Unix-like platform where containers are available,
-including macOS.
-For details, see the `Buildstream Docker documentation
-<https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+It is possible to run BuildStream in an OCI container tool such as Docker.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where containers are available, including macOS.
+
+Prebuilt images are available, see the documentation
+`here <https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md>`_
+
+You can also produce your own container images, either by adapting the
+`buildstream-docker-images project <https://gitlab.com/BuildStream/buildstream-docker-images/>`_,
+or by following the full installation instructions above.
+
+Note that some special configuration is often needed to run BuildStream in a container:
+
+ * User namespaces are used to isolate and control builds. This requires the
+ Docker ``--privileged`` mode.
Review Comment:
I will be happy to be wrong here, but I want to test a bit more to be sure.
I will try a Bst build in one of the FDSDK docker images and see if that works without anything more than `--device /dev/fuse`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org