You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2021/02/04 08:07:13 UTC
[buildstream] 37/41: doc/source/artifacts.rst: Update documentation
for CAS artifact server
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch jmac/googlecas_and_virtual_directories_1
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit dc9ec43bcdf8cfe6c1e1ae7b04ae1f29c2bf9fbe
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Wed May 23 10:51:43 2018 +0200
doc/source/artifacts.rst: Update documentation for CAS artifact server
---
doc/source/artifacts.rst | 140 ++++++++++++++++++++++-------------------------
1 file changed, 64 insertions(+), 76 deletions(-)
diff --git a/doc/source/artifacts.rst b/doc/source/artifacts.rst
index e0808e8..8e533c8 100644
--- a/doc/source/artifacts.rst
+++ b/doc/source/artifacts.rst
@@ -25,12 +25,11 @@ Users can declare additional remote caches in the :ref:`user configuration
define its own cache, it may be useful to have a local mirror of its cache, or
you may have a reason to share artifacts privately.
-Remote artifact caches are identified by their URL. There are currently three
+Remote artifact caches are identified by their URL. There are currently two
supported protocols:
-* ``http``: Pull-only access, without transport-layer security
-* ``https``: Pull-only access, with transport-layer security
-* ``ssh``: Push access, authenticated via SSH
+* ``http``: Pull and push access, without transport-layer security
+* ``https``: Pull and push access, with transport-layer security
BuildStream allows you to configure as many caches as you like, and will query
them in a specific order:
@@ -54,17 +53,23 @@ The rest of this page outlines how to set up a shared artifact cache.
Setting up the user
~~~~~~~~~~~~~~~~~~~
-A specific user is not needed for downloading artifacts, but since we
-are going to use ssh to upload the artifacts, you will want a dedicated
-user to own the artifact cache.
+A specific user is not needed, however, a dedicated user to own the
+artifact cache is recommended.
.. code:: bash
useradd artifacts
+The recommended approach is to run two instances on different ports.
+One instance has push disabled and doesn't require client authentication.
+The other instance has push enabled and requires client authentication.
-Installing the receiver
-~~~~~~~~~~~~~~~~~~~~~~~
+Alternatively, you can set up a reverse proxy and handle authentication
+and authorization there.
+
+
+Installing the server
+~~~~~~~~~~~~~~~~~~~~~
You will also need to install BuildStream on the artifact server in order
to receive uploaded artifacts over ssh. Follow the instructions for installing
BuildStream :ref:`here <install>`
@@ -74,10 +79,10 @@ in a system wide location, with ``pip3 install .`` in the BuildStream
checkout directory.
Otherwise, some tinkering is required to ensure BuildStream is available
-in ``PATH`` when it's companion ``bst-artifact-receive`` program is run
+in ``PATH`` when it's companion ``bst-artifact-server`` program is run
remotely.
-You can install only the artifact receiver companion program without
+You can install only the artifact server companion program without
requiring BuildStream's more exigent dependencies by setting the
``BST_ARTIFACTS_ONLY`` environment variable at install time, like so:
@@ -86,81 +91,57 @@ requiring BuildStream's more exigent dependencies by setting the
BST_ARTIFACTS_ONLY=1 pip3 install .
-Initializing the cache
-~~~~~~~~~~~~~~~~~~~~~~
-Now that you have a dedicated user to own the artifact cache, change
-to that user, and create the artifact cache ostree repository directly
-in it's home directory as such:
+Command reference
+~~~~~~~~~~~~~~~~~
-.. code:: bash
+.. click:: buildstream._artifactcache.casserver:server_main
+ :prog: bst-artifact-server
- ostree init --mode archive-z2 --repo artifacts
-This should result in an artifact cache residing at the path ``/home/artifacts/artifacts``
+Key pair for the server
+~~~~~~~~~~~~~~~~~~~~~~~
+For TLS you need a key pair for the server. The following example creates
+a self-signed key, which requires clients to have a copy of the server certificate
+(e.g., in the project directory).
+You can also use a key pair obtained from a trusted certificate authority instead.
-Serve the cache over https
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-This part should be pretty simple, you can do this with various technologies, all
-we really require is that you make the artifacts available over https (you can use
-http but until we figure out using gpg signed ostree commits for the artifacts, it's
-better to serve over https).
+.. code:: bash
-Here is an example, note that you must have a certificate **pem** file to use, as
-is the case for hosting anything over https.
+ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=artifacts.com" -out server.crt -keyout server.key
-.. code:: python
- import http.server, ssl, os
+Authenticating users
+~~~~~~~~~~~~~~~~~~~~
+In order to give permission to a given user to upload
+artifacts, create a TLS key pair on the client.
- # Maybe use a custom port, especially if you are serving
- # other web pages on the same computer
- server_address = ('localhost', 443)
- artifact_path = '/home/artifacts'
+.. code:: bash
- # The http server will serve from it's current
- # working directory
- os.chdir(artifact_path)
+ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -out client.crt -keyout client.key
- # Create Server
- httpd = http.server.HTTPServer(
- server_address,
- http.server.SimpleHTTPRequestHandler)
+Copy the public client certificate ``client.crt`` to the server and then add it
+to the authorized keys, like so:
- # Add ssl
- httpd.socket = ssl.wrap_socket(httpd.socket,
- server_side=True,
- certfile='localhost.pem',
- ssl_version=ssl.PROTOCOL_TLSv1)
+.. code:: bash
- # Run it
- httpd.serve_forever()
+ cat client.crt >> /home/artifacts/authorized.crt
-Configure and run sshd
-~~~~~~~~~~~~~~~~~~~~~~
-You will need to run the sshd service to allow uploading artifacts.
+Serve the cache over https
+~~~~~~~~~~~~~~~~~~~~~~~~~~
-For this you will want something like the following in your ``/etc/ssh/sshd_config``
+Public instance without push:
.. code:: bash
- # Allow ssh logins/commands with the artifacts user
- AllowUsers artifacts
+ bst-artifact-server --port 11001 --server-key server.key --server-cert server.crt /home/artifacts/artifacts
- # Some specifics for the artifacts user
- Match user artifacts
+Instance with push and requiring client authentication:
- # Dont allow password authentication for artifacts user
- #
- PasswordAuthentication no
+.. code:: bash
- # Also lets dedicate this login for only running the
- # bst-artifact-receive program, note that the full
- # command must be specified here; 'artifacts' is
- # the HOME relative path to the artifact cache.
- # The exact pull URL must also be specified.
- ForceCommand bst-artifact-receive --pull-url https://example.com/artifacts --verbose artifacts
+ bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts
User configuration
@@ -172,6 +153,8 @@ Assuming you have the same setup used in this document, and that your
host is reachable on the internet as ``artifacts.com`` (for example),
then a user can use the following user configuration:
+Pull-only:
+
.. code:: yaml
#
@@ -179,22 +162,27 @@ then a user can use the following user configuration:
#
artifacts:
- url: https://artifacts.com/artifacts
+ url: https://artifacts.com:11001
- # Alternative form if you have push access to the cache
- #url: ssh://artifacts@artifacts.com:22200/artifacts
- #push: true
+ # Optional server certificate if not trusted by system root certificates
+ server-cert: server.crt
+Pull and push:
-Authenticating users
-~~~~~~~~~~~~~~~~~~~~
-In order to give permission to a given user to upload
-artifacts, simply use the regular ``ssh`` method.
+.. code:: yaml
-First obtain the user's public ssh key, and add it
-to the authorized keys, like so:
+ #
+ # Artifacts
+ #
+ artifacts:
-.. code:: bash
+ url: https://artifacts.com:11002
+
+ # Optional server certificate if not trusted by system root certificates
+ server-cert: server.crt
- cat user_id_rsa.pub >> /home/artifacts/.ssh/authorized_keys
+ # Optional client key pair for authentication
+ client-key: client.key
+ client-cert: client.crt
+ push: true