[GitHub] [buildstream] juergbi commented on a change in pull request #1431: Support `storage-service` in cache configuration

[GitBox <gi...@apache.org>]

juergbi commented on a change in pull request #1431:
URL: https://github.com/apache/buildstream/pull/1431#discussion_r655395343



##########
File path: doc/source/using_config.rst
##########
@@ -178,6 +189,19 @@ Attributes
   * ``auto``: Only cache the build trees where necessary (e.g. for failed builds)
   * ``always``: Always cache the build tree.
 
+* ``storage-service``
+
+  An optional :ref:`service configuration <user_config_remote_execution_service>`
+  to use a *Content Addressable Storage* service as a remote cache. Write access
+  is required.
+
+  This allows operation with a small local cache even with large projects.
+  This is especially useful in combination with
+  :ref:`remote execution <user_config_remote_execution>` to minimize downloads
+  of build outputs, which may not be needed locally.
+
+  CAS objects will be downloaded on demand, e.g. on ``bst artifact checkout``.

Review comment:
       The contents of the `storage-service` are considered part of the cache, so blobs are not guaranteed to exist locally with the current branch after `bst artifact pull` (or `bst source fetch`).
   
   It could make sense in some cases. E.g. when the `storage-service` in the cache configuration is on a local network and you're pulling from an artifact server outside the local network. Or if you want to separately call `bst source fetch` before `bst build` with remote execution, you may not need the sources locally, you just want them in the cache `storage-service`, ready to be used for the remote execution build.
   
   However, I can definitely see that this behavior could be confusing, especially if the cache `storage-service` is not on the local network. Technically it is consistent with other commands, though.
   
   I'm not sure whether it would be ideal to simply ensure all blobs are in the local cache after `bst artifact pull` (and `bst source fetch`) as the behavior would be somewhat inconsistent. And with a low local cache quota it might even fail.
   
   A possible alternative would be to print a warning/info message when running `bst artifact pull` with a cache `storage-service` configured and add a global CLI option to disable the cache `storage-service` without having to edit `buildstream.conf`.
   
   I don't know whether it will be common for people to frequently switch between configurations with a cache `storage-service` and offline operations. My guess is that people who use remote execution will not frequently work offline (with the same project).
   
   Any thoughts which solution makes most sense?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org