You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2022/01/01 10:34:36 UTC
[buildstream] branch tristan/porting-guide created (now 3cfe04b)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 3cfe04b doc/source/porting_project.rst: Documenting breaking changes
This branch includes the following new commits:
new e83b08e _frontend/cli.py: Tidy up some whitespace
new 0e16be2 doc/source/format_project.rst: Stop referring to broken link.
new 58682e6 doc/source/format_declaring.rst: Slight rewording
new 912b5d6 doc: Adding initial structure for a porting guide
new f406682 doc/source/porting_user_configuration.rst: Documenting breaking changes
new 47d83f0 doc/source/porting_command_line.rst: Documenting breaking changes
new 3cfe04b doc/source/porting_project.rst: Documenting breaking changes
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 04/07: doc: Adding initial structure for a porting guide
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 912b5d693e194ba6ed9edf0aaee601251be0696b
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Fri Dec 31 14:47:04 2021 +0900
doc: Adding initial structure for a porting guide
---
doc/source/index.rst | 1 +
doc/source/main_porting.rst | 16 ++++++++++++++++
doc/source/porting_command_line.rst | 6 ++++++
doc/source/porting_project.rst | 6 ++++++
doc/source/porting_user_configuration.rst | 6 ++++++
5 files changed, 35 insertions(+)
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 68b13da..57f9b21 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -20,6 +20,7 @@ Later sections provide detailed information on BuildStream internals.
main_install
main_using
main_core
+ main_porting.rst
CONTRIBUTING
main_architecture
main_glossary
diff --git a/doc/source/main_porting.rst b/doc/source/main_porting.rst
new file mode 100644
index 0000000..07ac9ef
--- /dev/null
+++ b/doc/source/main_porting.rst
@@ -0,0 +1,16 @@
+
+
+.. _main_porting:
+
+Porting guide
+=============
+This document focuses on porting your existing BuildStream 1 projects to
+using BuildStream 2.
+
+
+.. toctree::
+ :maxdepth: 1
+
+ porting_user_configuration
+ porting_command_line
+ porting_project
diff --git a/doc/source/porting_command_line.rst b/doc/source/porting_command_line.rst
new file mode 100644
index 0000000..b0cd905
--- /dev/null
+++ b/doc/source/porting_command_line.rst
@@ -0,0 +1,6 @@
+
+
+.. _porting_command_line:
+
+Porting command line usage
+==========================
diff --git a/doc/source/porting_project.rst b/doc/source/porting_project.rst
new file mode 100644
index 0000000..4c240d7
--- /dev/null
+++ b/doc/source/porting_project.rst
@@ -0,0 +1,6 @@
+
+
+.. _porting_project:
+
+Porting the project.conf
+========================
diff --git a/doc/source/porting_user_configuration.rst b/doc/source/porting_user_configuration.rst
new file mode 100644
index 0000000..f862da7
--- /dev/null
+++ b/doc/source/porting_user_configuration.rst
@@ -0,0 +1,6 @@
+
+
+.. _porting_user_configuration:
+
+Porting the buildstream.conf
+============================
[buildstream] 07/07: doc/source/porting_project.rst: Documenting breaking changes
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 3cfe04b043339f06b821e4d3f93c0764448fb113
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jan 1 17:51:22 2022 +0900
doc/source/porting_project.rst: Documenting breaking changes
---
doc/source/porting_project.rst | 268 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 266 insertions(+), 2 deletions(-)
diff --git a/doc/source/porting_project.rst b/doc/source/porting_project.rst
index 4c240d7..f5c1496 100644
--- a/doc/source/porting_project.rst
+++ b/doc/source/porting_project.rst
@@ -2,5 +2,269 @@
.. _porting_project:
-Porting the project.conf
-========================
+Porting the project
+===================
+This document outlines breaking changes made to the project format in BuildStream 2.
+
+
+The project.conf
+----------------
+This section outlines breaking changes made to the :ref:`project.conf format <projectconf>`.
+
+
+Project name
+~~~~~~~~~~~~
+Various features related to :mod:`junction <elements.junction>` elements have been added
+which allow addressing projects by their :ref:`project names <project_format_name>`. For this
+reason, it is important to ensure that your project names are appropriately unique.
+
+
+Project versioning
+~~~~~~~~~~~~~~~~~~
+Instead of maintaining a separate version number for the format and for BuildStream releases,
+projects now declare the minimum version of BuildStream 2 they depend on.
+
+The ``format-version`` attribute should be removed from your project.conf (if present) and
+the :ref:`min-version <project_min_version>` attribute must be added.
+
+
+Some attributes can only be specified in project.conf
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Whenever specifying any of the following toplevel project attributes, they must be
+specified inside the project.conf itself and cannot :ref:`included <format_directives_include>`
+from a separate file:
+
+* :ref:`name <project_format_name>`
+* :ref:`element-path <project_element_path>`
+* :ref:`min-version <project_min_version>`
+* :ref:`plugins <project_plugins>`
+
+
+Artifact cache configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The format for declaring :ref:`artifact caches <project_artifact_cache>` which are associated to
+your project have been completely redesigned.
+
+**BuildStream 1:**
+
+.. code:: yaml
+
+ #
+ # We used to specify a single URL
+ #
+ artifacts:
+ url: https://foo.com/artifacts
+
+**BuildStream 2:**
+
+.. code:: yaml
+
+ #
+ # Now we declare a list, and credentials have been split
+ # out into a separate "auth" dictionary
+ #
+ artifacts:
+ - url: https://foo.com:11001
+ auth:
+ server-cert: server.crt
+
+
+Loading plugins
+~~~~~~~~~~~~~~~
+The format for :ref:`loading plugins <project_plugins>` has been completely redesigned.
+
+.. tip::
+
+ A new method for :ref:`loading plugins through junctions <project_plugins_junction>`
+ has been added. In the interest of ensuring strong determinism and reliability it is
+ strongly recommended to use this new method.
+
+
+Local plugins
+'''''''''''''
+Here is an example of how loading :ref:`local plugins <project_plugins_local>` has changed.
+
+**BuildStream 1:**
+
+.. code:: yaml
+
+ plugins:
+
+ - origin: local
+ path: plugins/sources
+
+ #
+ # We used to specify version numbers, these no longer exist.
+ #
+ sources:
+ mysource: 0
+
+**BuildStream 2:**
+
+.. code:: yaml
+
+ plugins:
+
+ - origin: local
+ path: plugins/sources
+
+ #
+ # Now we merely specify a list of plugins to load from
+ # a given project local directory
+ #
+ sources:
+ - mysource
+
+
+Pip plugins
+'''''''''''
+Here is an example of how loading :ref:`pip plugins <project_plugins_pip>` has changed.
+
+**BuildStream 1:**
+
+.. code:: yaml
+
+ plugins:
+
+ - origin: pip
+
+ package-name: vegetables
+
+ #
+ # We used to specify version numbers, these no longer exist.
+ #
+ elements:
+ potato: 0
+
+**BuildStream 2:**
+
+.. code:: yaml
+
+ plugins:
+
+ - origin: pip
+
+ #
+ # We can now specify version constraints
+ #
+ package-name: vegetables>=1.2
+
+ #
+ # Now we merely specify a list of plugins to load from
+ # a given pip package that is expected to be installed
+ #
+ elements:
+ - potato
+
+
+Core elements
+-------------
+This section outlines breaking changes made to :ref:`core element plugins <plugins>` which
+may require you to make changes to your project.
+
+
+The :mod:`stack <elements.stack>` element
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Stack elements dependencies are now hard required to be both build and runtime dependencies.
+
+
+The :mod:`script <elements.script>` element
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The ``layout`` attribute has now been removed in favor of dependency level configuration.
+
+Here is an example script which collects a manifest of all files in the hypothetical
+``system.bst`` element, using a hypothetical base runtime element ``base-utilities.bst``.
+
+**BuildStream 1:**
+
+.. code:: yaml
+
+ kind: script
+
+ build-depends:
+ - base-utilities.bst
+ - system.bst
+
+ config:
+ #
+ # The old format was redundant and required explicit layout
+ # of the dependencies already declared above.
+ #
+ layout:
+ - element: base-utilities.bst
+ destination: /
+ - element: system.bst
+ destination: "%{build-root}"
+
+ commands:
+ - find %{build-root} > %{install-root}/manifest.log
+
+**BuildStream 2:**
+
+.. code:: yaml
+
+ kind: script
+
+ #
+ # The default location is "/" so there is no need to configure
+ # the "base-utilities.bst" dependency
+ #
+ build-depends:
+ - base-utilities.bst
+ - system.bst
+ config:
+ location: "%{build-root}"
+
+ config:
+ commands:
+ - find %{build-root} > %{install-root}/manifest.log
+
+.. tip::
+
+ The ``location`` dependency level configuration is also supported by all
+ :mod:`BuildElement <buildstream.buildelement>` plugins.
+
+
+The :mod:`junction <elements.junction>` element
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The YAML format for declaring junctions has not changed, however the way that
+multiple junctions interact in a loaded pipeline has changed significantly.
+
+Specifically, the :ref:`element name <format_element_names>` used to declare
+a junction no longer has any special significance, whereas in BuildStream 1
+the junction's name is used to coalesce matching junctions in subprojects.
+
+BuildStream 2 offers more flexibility in this regard, and allows you to *inherit*
+a junction from a subproject, by using a :mod:`link <elements.link>` element directly
+in place of a junction, and/or explicitly override the configuration of a subproject's
+junction using the new ``overrides`` configuration attribute which the junction
+element now provides.
+
+Consult the :mod:`junction <elements.junction>` element documentation for a more
+detailed explanation.
+
+
+Miscellaneous
+-------------
+
+
+Element naming
+~~~~~~~~~~~~~~
+The names of elements have :ref:`become more restrictive <format_element_names>`, for example
+they must have the ``.bst`` extension.
+
+
+Overlap whitelist
+~~~~~~~~~~~~~~~~~
+The :ref:`overlap whitelist <public_overlap_whitelist>`, which is the public data
+found on elements which indicate which files an element can overwrite, must now
+be an absolute path.
+
+
+Strip commands
+~~~~~~~~~~~~~~
+The default ``strip-commands`` which :mod:`BuildElement <buildstream.buildelement>` implementations
+use to split out debug symbols from binaries have been removed.
+
+This can be solved by declaring a value for the ``%{strip-binaries}`` variable which
+will be used for this purpose.
[buildstream] 01/07: _frontend/cli.py: Tidy up some whitespace
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e83b08edb58f68048f7594c0da75ccdb92a819fb
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jan 1 15:54:19 2022 +0900
_frontend/cli.py: Tidy up some whitespace
---
src/buildstream/_frontend/cli.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/buildstream/_frontend/cli.py b/src/buildstream/_frontend/cli.py
index 65f95c5..79e19ab 100644
--- a/src/buildstream/_frontend/cli.py
+++ b/src/buildstream/_frontend/cli.py
@@ -958,7 +958,7 @@ def source_track(app, elements, deps, except_, cross_junctions):
default=None,
metavar="LOCATION",
type=click.Path(),
- help="Create a tarball containing the sources instead " "of a file tree.",
+ help="Create a tarball containing the sources instead of a file tree.",
)
@click.option(
"--compression",
[buildstream] 06/07: doc/source/porting_command_line.rst: Documenting breaking changes
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 47d83f066036dba0fa7830c9caf157629dd8e6e9
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jan 1 15:54:49 2022 +0900
doc/source/porting_command_line.rst: Documenting breaking changes
---
doc/source/porting_command_line.rst | 167 ++++++++++++++++++++++++++++++++++++
1 file changed, 167 insertions(+)
diff --git a/doc/source/porting_command_line.rst b/doc/source/porting_command_line.rst
index b0cd905..4520c07 100644
--- a/doc/source/porting_command_line.rst
+++ b/doc/source/porting_command_line.rst
@@ -4,3 +4,170 @@
Porting command line usage
==========================
+This document outlines breaking changes made to the :ref:`command line interface <commands>`
+in BuildStream 2.
+
+
+:ref:`bst init <invoking_init>`
+-------------------------------
+
+* The global ``--directory`` option is no longer observed by ``bst init``, instead
+ the command accepts an optional target directory argument.
+
+* The ``--format-version`` option has been removed in favor of the new ``--min-version`` option.
+
+
+:ref:`bst build <invoking_build>`
+---------------------------------
+
+* Tracking is no longer supported at build time and must be performed separately, this
+ removes the ``--track``, ``--track-all``, ``--track-except``, ``--track-cross-junctions``
+ and ``--track-save`` options from the command.
+
+ To track your elements in BuildStream 2, use the :ref:`bst source track <invoking_source_track>`
+ command instead.
+
+* The ``--all`` option which was used to indicate that all dependencies should be built
+ regardless of whether they are needed for producing the target elements has been removed
+ in favor of adding the ``--deps`` option.
+
+ To acheive the same functionality, use ``bst build --deps all ...``.
+
+
+:ref:`bst show <invoking_show>`
+-------------------------------
+
+* The ``plan`` value is no longer supported as a value for the ``--deps`` option.
+
+* Values for the ``%{state}`` format have changed
+
+ * :mod:`junction <elements.junction>` elements will display ``junction``, as these cannot be built
+ * In the case a cached failed build artifact is found, then ``failed`` will be displayed
+ * Due to changes in the scheduler, we may observe changes as to when ``waiting``, ``buildable``, ``fetch needed``
+ are displayed for a given element.
+
+
+:ref:`bst fetch <invoking_source_fetch>`
+----------------------------------------
+
+* This command has been removed as a top-level command and now exists as :ref:`bst source fetch <invoking_source_fetch>`
+
+* Tracking is no longer supported at fetch time and must be performed separately, this
+ removes the ``--track`` and ``--track-cross-junctions`` options from the command.
+
+ To track your elements in BuildStream 2, use the :ref:`bst source track <invoking_source_track>`
+ command instead.
+
+* The ``plan`` value is no longer supported as a value for the ``--deps`` option. The default
+ value for the ``--deps`` option is now ``none``.
+
+
+:ref:`bst track <invoking_source_track>`
+----------------------------------------
+
+* This command has been removed as a top-level command and now exists as :ref:`bst source track <invoking_source_track>`
+
+
+:ref:`bst pull <invoking_artifact_pull>`
+----------------------------------------
+
+* This command has been removed as a top-level command and now exists as :ref:`bst artifact pull <invoking_artifact_pull>`
+
+* The ``--remote`` option has been removed in favor the ``--artifact-remote`` option, which can be
+ specified multiple times.
+
+* The values which can be specified by ``--artifact-remote`` options have a new format which
+ is :ref:`documented here <invoking_specify_remotes>`.
+
+
+:ref:`bst push <invoking_artifact_push>`
+----------------------------------------
+
+* This command has been removed as a top-level command and now exists as :ref:`bst artifact push <invoking_artifact_push>`
+
+* The ``--remote`` option has been removed in favor the ``--artifact-remote`` option, which can be
+ specified multiple times.
+
+* The values which can be specified by ``--artifact-remote`` options have a new format which
+ is :ref:`documented here <invoking_specify_remotes>`.
+
+
+
+:ref:`bst checkout <invoking_artifact_checkout>`
+------------------------------------------------
+
+* This command has been removed as a top-level command and now exists as :ref:`bst artifact checkout <invoking_artifact_checkout>`
+
+* The trailing ``LOCATION`` argument has been removed in favor of a ``--directory`` option.
+
+ **BuildStream 1:**
+
+ .. code:: shell
+
+ bst checkout element.bst ~/checkout
+
+ **BuildStream 2:**
+
+ .. code:: shell
+
+ bst artifact checkout --directory ~/checkout element.bst
+
+
+:ref:`bst shell <invoking_shell>`
+---------------------------------
+
+* The ``--sysroot`` option has been completely removed.
+
+ This is no longer needed for failed builds as the build tree will be cached in a failed build artifact.
+
+* Sources and artifacts required to produce the shell environment will now be downloaded
+ automatically by default.
+
+
+:ref:`bst workspace open <invoking_workspace_open>`
+---------------------------------------------------
+
+* The ``--track`` option is now removed.
+
+* The trailing ``LOCATION`` argument has been removed in favor of a ``--directory`` option.
+
+ **BuildStream 1:**
+
+ .. code:: shell
+
+ bst workspace open element.bst ~/workspace
+
+ **BuildStream 2:**
+
+ .. code:: shell
+
+ bst workspace open --directory ~/workspace element.bst
+
+
+:ref:`bst workspace reset <invoking_workspace_reset>`
+-----------------------------------------------------
+
+* The ``--track`` option is now removed.
+
+
+:ref:`bst source-bundle <invoking_source_checkout>`
+---------------------------------------------------
+This command has been completely removed, but similar behavior can be achieved
+using the :ref:`bst source checkout <invoking_source_checkout>` command.
+
+
+**BuildStream 1:**
+
+.. code:: shell
+
+ bst source-bundle --directory ~/bundle element.bst
+
+**BuildStream 2:**
+
+.. code:: shell
+
+ bst source checkout \
+ --tar ~/sources.tgz \
+ --compression gz \
+ --include-build-scripts \
+ element.bst
[buildstream] 02/07: doc/source/format_project.rst: Stop referring to broken link.
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0e16be24f41b1f94181c158777200919ba1246c0
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jan 1 17:03:31 2022 +0900
doc/source/format_project.rst: Stop referring to broken link.
There used to be a good human readable document about versioning
constraints at: https://python-poetry.org/docs/versions/
But it's gone.
---
doc/source/format_project.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/source/format_project.rst b/doc/source/format_project.rst
index 69b1e4c..dc089bf 100644
--- a/doc/source/format_project.rst
+++ b/doc/source/format_project.rst
@@ -389,8 +389,8 @@ When loading plugins from the ``pip`` plugin origin, it is possible to
specify constraints on the versions of packages you want to load
your plugins from.
-The syntax for specifying constraints are `explained here <https://python-poetry.org/docs/versions/>`_,
-and they are the same format supported by the ``pip`` package manager.
+The syntax for specifying versioning constraints is the same format supported by
+the ``pip`` package manager.
.. note::
[buildstream] 03/07: doc/source/format_declaring.rst: Slight rewording
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 58682e6e9b85c66882585129be5a42eabfac2c20
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Sat Jan 1 17:50:27 2022 +0900
doc/source/format_declaring.rst: Slight rewording
---
doc/source/format_declaring.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/source/format_declaring.rst b/doc/source/format_declaring.rst
index 8d6417c..fede3e0 100644
--- a/doc/source/format_declaring.rst
+++ b/doc/source/format_declaring.rst
@@ -97,7 +97,7 @@ Element naming rules
''''''''''''''''''''
When naming the elements, use the following rules:
-* The name of the file must have ``.bst`` extension.
+* The name of the file must have the ``.bst`` extension.
* All characters in the name must be printable 7-bit ASCII characters.
[buildstream] 05/07: doc/source/porting_user_configuration.rst: Documenting breaking changes
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/porting-guide
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit f406682a3a0e7d0c0fcaa1682c9ca137d9a5da22
Author: Tristan van Berkom <tr...@codethink.co.uk>
AuthorDate: Fri Dec 31 15:04:24 2021 +0900
doc/source/porting_user_configuration.rst: Documenting breaking changes
---
doc/source/porting_user_configuration.rst | 39 +++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/doc/source/porting_user_configuration.rst b/doc/source/porting_user_configuration.rst
index f862da7..1971ddb 100644
--- a/doc/source/porting_user_configuration.rst
+++ b/doc/source/porting_user_configuration.rst
@@ -4,3 +4,42 @@
Porting the buildstream.conf
============================
+This document outlines breaking changes made to the :ref:`user configuration <user_config>`
+in BuildStream 2.
+
+
+Filename and parallel installation
+----------------------------------
+The default filename to load user configuration remains unchanged, however,
+if you plan to install and use both versions of BuildStream on the same
+host, it is recommended to keep your BuildStream 2 configuration in a
+file named ``buildstream2.conf``.
+
+
+Working directories
+-------------------
+The ``builddir`` and ``artifactdir`` have been removed in favor of the new ``cachedir``.
+
+
+BuildStream 1:
+~~~~~~~~~~~~~~
+
+.. code:: yaml
+
+ builddir: ${XDG_CACHE_HOME}/buildstream/build
+ artifactdir: ${XDG_CACHE_HOME}/buildstream/artifacts
+
+
+BuildStream 2:
+~~~~~~~~~~~~~~
+
+.. code:: yaml
+
+ cachedir: ${XDG_CACHE_HOME}/buildstream/cache
+
+
+Remote cache configuration
+--------------------------
+The configuration for remote artifact caches has been completely
+redesigned, please refer to the :ref:`artifact cache configuration documentation <config_artifact_caches>`
+for details on how to configure remotes in BuildStream 2.