You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Tristan Van Berkom <tr...@codethink.co.uk> on 2020/11/29 13:01:18 UTC
Proposal: Redesign of configuration surface for all remote services
Hi all,
As I raised in my previous, still unresolved proposal earlier this
year[0], the configuration surface surrounding artifact caches, where,
when and what to pull or push, is highly confusing. In the current
state, it is hard to guess what will happen when combining certain
configurations.
So I want to raise this again but I think it is appropriate to widen
the scope of this proposal in order to cover the configuration data
pertaining to all remote services which can potentially be configured.
If I am not missing anything, these currently include:
* Artifact content services (CAS)
* Artifact indexing services (Remote Assets)
* Source cache content services (CAS)
* Source cache indexing services (Remote Assets)
* Remote execution services
In essence, we've added so many switches in various places, all for
various purposes, but the resulting combinatorial complexity of all of
these options is largely unjustified.
I think we need to take a step back and look at the purposes for each
of these options and switches, reevaluate what is actually useful and
why, and come up with a more coherent picture, such that it becomes
obvious both to the user and the developer what will happen as a result
of any configuration combination.
Service configuration is user configuration
===========================================
As I did specify in my last email, I will reiterate that all of the
configuration pertaining to remote serives, their end points, whether
they are trusted, whether to carry assets forward across projects, all
of this is strictly user configuration.
We specifically extended the way user configuration can be augmented by
"Project Recommendations"[1] so that it was possible for a project to
"suggest" an artifact cache which is likely to contain artifacts for
the given project.
However, actual project data pertains to what and how something gets
built and will affect cache keys, and user configuration pertains to
things such as log file formatting, location of where to store
artifacts locally and remotely, and generally how the session runs, and
cannot ever affect cache keys.
User configuration constitutes the context in which BuildStream was
invoked, and BuildStream guarantees a deterministic build for the
project data regardless of context (ergo, the name of the Context
object where all invocation context is stored for the session).
Sorry for the extra emphasis here, I think it's important to stress
that these are separate configuration surfaces.
Use cases we want
=================
Here I will try to provide a birds eye view of what our use cases are,
what does a BuildStream client application require from these
services ?
* The ability to store and retrieve artifacts on a remote artifact
server.
* The ability to store and retrieve staged source packages, indexed
by source cache key, on remote source cache services.
* The ability to farm out builds to a remote execution service
* The ability to make requirements of worker instances on a remote
execution service.
- Possibly also the ability to bail out early if the remote
execution service knows that it cannot provide a worker which
with the properties which some of the project elements require.
* Ability to have redundancies in configuration of remote servers, in
case a service is down we usually allow configuration of services
in list format.
* Ability to carry artifacts forward from a third party artifact
cache which was recommended by project configuration across a
junction boundary.
I.e. for better repeatability, it is often desirable to re-cache
the artifacts from an upstream project on your own infra in order
to ensure you have your own copy.
NOTE: This is currently only available in project data and not
overridable by user configuration in the form of the
`cache-junction-elements`[2] configuration, which I already
pointed out was problematic in my original report[0].
* Ability to avoid downloading artifacts found on third party
infrastructure.
I.e. for better trustability, you may want to ensure that all of
the built artifacts you end up consuming were built on
infrastructure you control, rather than downloaded from an upstream
project's artifact server.
NOTE: This is currently only available in project data and not
overridable by user configuration in the form of the
`ignore-junction-remotes`[2] configuration, which I already
pointed out was problematic in my original report[0].
* Ability to farm out any local caching work a remote service, to
reduce uploads and downloads for builds when configured on an RE
service (by way of specifying the RE service's CAS here),
configuring multiple build machines which run without RE may also
be optimized by way of using the same remote CAS for this.
NOTE: This has not yet landed and is a part of Jürg's ongoing
work[3].
* Ability to clearly override the recommendation of any project.conf
in the loaded pipeline using user configuration, which should
always have the last word on any user configuration.
Use cases might not want
========================
* Multiple RE services in a single session.
Right now since we have the ability for a project itself to declare
what remote execution service it intends to build on[4].
This means that when loading a pipeline with multiple projects, the
user should expect BuildStream to cope with a scenario where
elements from a given project must be built on their own respective
RE service
I don't think anyone ever requested that BuildStream should cope
with situations where the user instructs us to build some elements
on one RE service and then it's reverse dependencies on other RE
services, and I don't think anyone wants this weird feature.
* Ability to "recommend" push remotes, or any remotes which would
require write access on a remote service, using a project.conf
recommendation.
While it makes sense for read-only access, write access to a
service will undoubtedly require a special private key/certificate
in order to access.
I think that if we are going to require the machine running
BuildStream to have credentials anyway, then it makes sense to also
require that the service be specified in a buildstream.conf file on
that machine.
Removing this configuration service would result in:
- No more RE service recommendations in project.conf (an RE service
AFAICS is only available with write access, using RE causes side
effects on the remote and is a privileged operation).
- No option to configure "push" in artifact cache recommendations
in project.conf
- No option to configure "push" in source cache recommendations in
project.conf
* Per project granularity in the configuration of what is currently
called `cache-junction-elements` and `ignore-junction-remotes`[2].
I think that the decisions made by these configurations are only
pertinent to the toplevel project being built, but not pertinent
to intermediate projects.
If for instance, I have elements in projects (A) (B) and (C),
configured as such:
(A): Pull artifacts from https://project-a.com/cache
(B): Pull and push artifacts at https://project-b.com/cache,
and also cache elements from subproject (A) using the
cache-junction-elements setting
(C): Pull and push artifacts at https://project-c.com/cache,
and also cache elements from subproject (B) using the
cache-junction-elements setting
What should be happening when I build (C), is that *all* elements
from all subprojects should be re-cached at
https://project-c.com/cache.
What the current configuration structure dictates, is that when I
build (C), then artifacts from project (A) will be re-cached in the
artifact server belonging to project (B), but not in the artifact
server belonging to project (C) unless (C) also explicitly defines
a junction.
Essentially, this re-caching structure should simply be a blanket
statement that is applied to all subproject elements recursively,
and the configuration should be entirely ignored in subprojects.
The same essentially goes for `ignore-junction-remotes`, we only
really care about this in the context of the toplevel project we
are building, and how upstream projects deal with these things are
their own decision.
Draft Format Proposal
=====================
Largely reposted from the original proposal[0], I propose that we
remove junction level configuration[2] of artifact server behaviors
completely, and replace this with a project.conf recommended behavior
which can be overridden by buildstream.conf.
Furthermore, as per the above, I would propose that we drop the "push"
option completely from project.conf.
For the project and user configuration, I would suggest that we break
the format such that `artifacts` becomes a dictionary with a `servers`
list, adding the two extra configurations on that dictionary, e.g.:
#
# Artifacts
#
artifacts:
#
# Whether to try pull artifacts from artifact
# caches recommended by subprojects.
#
# XXX Arguably, I would like to remove this completely, even
#
pull-subproject-artifacts: True
#
# Whether to push subproject artifacts into
# the following servers.
#
push-subprojects-artifacts: True
#
# Same old server list as before, this is a per project hint
# of where to pull artifacts from (or perhaps push to if you
# have credentials).
#
servers:
- url: https://artifacts.com/artifacts:11001
server-cert: server.crt
When expressed in project configuration, I would clarify that the two
added settings are only considered as a default behavior when building
this project as a toplevel project, that they are completely ignored
when building the project as a subproject, and that these can be
overridden
by user configuration.
Further, I would propose that we clarify in the user configuration
documentation that the per-project settings regarding artifact caches
are NOT taken into consideration separately when building a project
which junctions another project in your list.
For example:
#
# My buildstream.conf
#
projects:
foo:
artifacts:
pull-subproject-artifacts: False
bar:
artifacts:
pull-subproject-artifacts: True
If there is a junction between `foo` and `bar`, in either direction,
there is no conflict here to resolve or differing behavior on a per
project basis: Only the toplevel project is ever considered to
determine artifact caching behaviors here.
Further proposed changes
========================
In addition to actual format changes, I would propose the following:
* Remove the ability of the project.conf to recommend remote
execution services, as access to a remote execution service is
already privileged.
Furthermore, connection to a remote execution service is really
an attribute of a session and nothing to do with the project
itself, it should apply to the whole session equally and there
should be no doubt that only one RE service will be accessed in the
session.
* Service lists should override eachother, rather than extend
eachother.
In the user configuration we have the following text[5]:
"Although project’s often specify a remote artifact cache in
their project.conf, you may also want to specify extra caches."
This text suggests that a list of artifact services recommended by
the project.conf of a given project continues to be observed, even
after being overridden in user configuration.
I think this goes against the spirit of our configuration style,
and the recommendations of the loaded project.conf should be
overridable rather than merely extensible.
Even with the currently existing note:
"Caches listed here will be considered a higher priority than
those specified by the project..."
This leaves us with no way to completely override the
recommendation of the project.
Personally, I am more comfortable with the certainty that I have
control to override these, than I am with the convenience of
falling back on project recommendations, although if there are
strong opposing feelings here, we may need to support both the
override and extend semantics separately.
* Global service lists override projects instead of act as a fallback
Currently also specified in the user configuration docs, is:
"Caches declared here will be used by all BuildStream project’s
on the user’s machine and are considered a lower priority than
those specified in the project configuration."
I think instead, the global list should override anything specified
by any project.conf, except for services specified on a per project
basis in the user configuration.
NOTE: In the previous section I commented on
"pull-subproject-artifacts", this can completely be removed,
as the same behaviour can be achieved simply by providing an
empty service list for the global artifact services
configured in buildstream.conf.
I realize there is a lot to process here, it is a complete teardown and
rewrite of user configuration surface for services. I would like to
hear your input and possibly alternative recommendations.
While it may not end up exactly as I described above, I am convinced
that some urgent house cleaning is necessary.
Cheers,
-Tristan
[0]: https://mail.gnome.org/archives/buildstream-list/2020-May/msg00018.html
[1]: https://docs.buildstream.build/master/arch_data_model.html#context
[2]: https://docs.buildstream.build/master/elements/junction.html
[3]: https://gitlab.com/BuildStream/buildstream/-/merge_requests/2095
[4]: https://docs.buildstream.build/master/format_project.html#remote-execution
[5]: https://docs.buildstream.build/master/using_config.html#artifact-server
Re: Proposal: Redesign of configuration surface for all remote services
Posted by Sander Striker <s....@striker.nl>.
Hi,
This all makes sense to me. Once the migration is done again, I'll file an
issue for RE Scenario 2, such that we don't forget :-).
Cheers,
Sander
On Wed, Feb 3, 2021 at 7:50 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Hi again,
>
> One other thought I forgot to include...
>
> On Tue, 2021-02-02 at 23:28 +0900, Tristan Van Berkom wrote:
> > Hi Sander,
> >
> > Replies inline.
> >
> [...]
> > * I am reading between the lines that there is an idea that
> > the entire "push" configuration option is redundant, as one
> > could infer this by the presence of certain credentials.
> >
> > It was my original naive understanding that we could get away
> > with renaming the confusingly named `server-cert`, `client-cert`
> > and `client-key` configurations with `pull-cert` and `push-cert`
> > and `push-key`, alleviating some cognitive effort for the user.
> >
> > However I was mistaken about the relation here, and I don't think
> > that we can infer the ability or decision to push based on what
> > credentials have been provided.
> >
> > Instead, the "server-cert" is only needed when encrypted traffic
> > is only required on the client receiving end, and the client
> > cert/key are additionally required when bidirectional TLS is
> > required.
> >
> > I think it is better to consider this requirement as an attribute
> > of the client server relationship, and the implementation of the
> > service, rather than to assume that two way encryption will never
> > be required for activities which the client deems to be "pulling"
> > or "fetching".
> >
> > Even though the implementations we are currently working with
> > happen to require all three credentials only for push operations,
> > this does not mean it will always be so, for every service
> > implementation (at least, not as far as I understand).
>
> As I am not a big fan of the "push" boolean switch either, I was also
> thinking this might be more appropriate as an enumeration.
>
> Instead of "push" being either true/false, we could have an "action"
> attribute with three values: "push" / "pull" / "both".
>
> I did not do this in the branch as of yet, but what it would provide in
> addition to the current API is the ability to define push remotes where
> you only want to push but want to make sure you never try to pull from
> those.
>
> However, I haven't come up with a sensible reason for a user to desire
> this option, which is why I didn't change this part.
> Cheers,
> -Tristan
>
>
>
Re: Proposal: Redesign of configuration surface for all remote
services
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi again,
One other thought I forgot to include...
On Tue, 2021-02-02 at 23:28 +0900, Tristan Van Berkom wrote:
> Hi Sander,
>
> Replies inline.
>
[...]
> * I am reading between the lines that there is an idea that
> the entire "push" configuration option is redundant, as one
> could infer this by the presence of certain credentials.
>
> It was my original naive understanding that we could get away
> with renaming the confusingly named `server-cert`, `client-cert`
> and `client-key` configurations with `pull-cert` and `push-cert`
> and `push-key`, alleviating some cognitive effort for the user.
>
> However I was mistaken about the relation here, and I don't think
> that we can infer the ability or decision to push based on what
> credentials have been provided.
>
> Instead, the "server-cert" is only needed when encrypted traffic
> is only required on the client receiving end, and the client
> cert/key are additionally required when bidirectional TLS is
> required.
>
> I think it is better to consider this requirement as an attribute
> of the client server relationship, and the implementation of the
> service, rather than to assume that two way encryption will never
> be required for activities which the client deems to be "pulling"
> or "fetching".
>
> Even though the implementations we are currently working with
> happen to require all three credentials only for push operations,
> this does not mean it will always be so, for every service
> implementation (at least, not as far as I understand).
As I am not a big fan of the "push" boolean switch either, I was also
thinking this might be more appropriate as an enumeration.
Instead of "push" being either true/false, we could have an "action"
attribute with three values: "push" / "pull" / "both".
I did not do this in the branch as of yet, but what it would provide in
addition to the current API is the ability to define push remotes where
you only want to push but want to make sure you never try to pull from
those.
However, I haven't come up with a sensible reason for a user to desire
this option, which is why I didn't change this part.
Cheers,
-Tristan
Re: Proposal: Redesign of configuration surface for all remote
services
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Sander,
Replies inline.
On Tue, 2021-02-02 at 12:35 +0100, Sander Striker wrote:
> Hi Tristan,
>
> First of all, Thank You for the write up as well as the progress made so
> far.
>
> I'll be honest in that I am not sure I got the full picture of implications
> - maybe as a result of your attempt to outline it in detail ;-). But
> that's on me.
>
> I have a couple of questions:
> *Remote Execution*
> There seems to be a skew towards only using a single service in the
> proposal. There are some aspects to Remote Execution that we haven't
> clearly articulated or fully implemented. We have only implemented
> scenario 1 currently.
>
> Scenario 1: Full Remote Execution [implemented]
> - BuildStream checks ActionCache
> - on cache hit, get ActionResult and blobs from CAS associated with
> Remote Execution. Note: the blob content fetching could be on-demand
> - on cache miss, uploads any missing blobs for the Action to the CAS
> associated with Remote Execution, finally it uses the Execution service to
> execute the Action.
>
> Scenario 2: Remote Cache (read-only) [not currently implemented]
> - BuildStream checks ActionCache
> - on cache hit, get ActionResult and blobs from CAS associated with
> Remote Execution. Note: the blob content fetching could be on-demand
> - on cache miss fall back to local execution
>
> Scenario 3: Remote Cache (read-write) [not currently implemented]
> - BuildStream checks ActionCache
> - on cache hit, get ActionResult and blobs from CAS associated with
> Remote Execution. Note: the blob content fetching could be on-demand
> - on cache miss fall back to local execution, and upload result blobs to
> CAS associated with Remote Execution, finally write ActionResult to
> ActionCache
>
> Especially in case of Scenario 2, it is not unimaginable to have a project
> propose a read-only endpoint. Getting these implemented is currently
> mostly tricky due to the "fall back to local execution" bit - as we choose
> to either remote or to locally execute up front.
As far as I can see, yes it can make sense, once those features are
implemented in some way, to figure out a sensible way to have the
project make recommendations of where to pull certain things from.
I think it makes better sense to postpone exposing such options until a
time when they have a defined purpose.
> *Write access configuration*
> With the current credentials it may make sense to only have push be enabled
> when auth credentials have been set up in the user configuration. To also
> set `push: true` seems redundant if that is also set in project.conf. If
> you don't want to push as a user, set `push: false` in user conf if you
> have credentials configured.
> At some point in the future, I can imagine credential prompting as well,
> and I would imagine bst helps me out by caching those creds or e.g. storing
> a JWT for later use.
A few things:
* Setting 'push' to true or false is strictly related to a specific
endpoint declaration.
These endpoint lists are recommended in projects and ultimately
decided in user configuration.
A given endpoint is never inherited and augmented, it is either
completely replaced by user configuration, or the user
configuration decides to keep this end point.
In other words, you don't set `push: True` in the user config
for an endpoint that is recommended by the project.conf, that
never happens.
* I am reading between the lines that there is an idea that
the entire "push" configuration option is redundant, as one
could infer this by the presence of certain credentials.
It was my original naive understanding that we could get away
with renaming the confusingly named `server-cert`, `client-cert`
and `client-key` configurations with `pull-cert` and `push-cert`
and `push-key`, alleviating some cognitive effort for the user.
However I was mistaken about the relation here, and I don't think
that we can infer the ability or decision to push based on what
credentials have been provided.
Instead, the "server-cert" is only needed when encrypted traffic
is only required on the client receiving end, and the client
cert/key are additionally required when bidirectional TLS is
required.
I think it is better to consider this requirement as an attribute
of the client server relationship, and the implementation of the
service, rather than to assume that two way encryption will never
be required for activities which the client deems to be "pulling"
or "fetching".
Even though the implementations we are currently working with
happen to require all three credentials only for push operations,
this does not mean it will always be so, for every service
implementation (at least, not as far as I understand).
* Regarding user prompting for credentials and suchlike, I think
the refactor and cleanup does bring the code into much better shape
to handle this sort of thing, as we have a clear decision early
on of all the services which the client will attempt to access
over the course of a session, all boiled down into a clean
focal point/function.
I don't think the prompting and storing of credentials has
any effect on the API configuration changes put forward by
this branch.
Does this make sense ?
Cheers,
-Tristan
> Any thoughts on how the above fits with the current changes?
> Cheers,
>
> Sander
>
>
> On Thu, Jan 28, 2021 at 9:12 AM Tristan Van Berkom <
> tristan.vanberkom@codethink.co.uk> wrote:
>
> > Hi all,
> >
> > Since there was no activity on this thread for a very long time, I
> > decided to go ahead and take a crack at this.
> >
> > I have a good branch now that is ready for review. The MR is up here:
> > https://github.com/apache/buildstream/pull/1453
> >
> >
> > I'm sending a detailed email because it's a large proposal and I would
> > like this to be visible, so that people can chime in incase we've
> > missed an important use case.
> >
> > Cheers,
> > -Tristan
> >
> >
> > Here is the design/changes I've come up with.
> > =============================================
> >
> >
> > The offending junction configurations
> > -------------------------------------
> > The junction configurations:
> >
> > "cache-junction-elements"
> > "ignore-junction-remotes"
> >
> > Are completely removed, reducing the worrisome ambiguity of what
> > happens in what configuration.
> >
> > This is replaced by the enhanced user configuration.
> >
> >
> > Authentication
> > --------------
> > For all of the authentication related properties, `server-cert`,
> > `client-cert` and `client-key`, these have been split out into a
> > subdictionary named "auth" for any remote configuration.
> >
> > This may allow better extensibility for alternative authentication
> > methods in the future, however right now it serves us very well to be
> > able to document the "auth" dictionary in one central place in the
> > documentation.
> >
> >
> > Remote Execution Configuration
> > ------------------------------
> > As described in this thread, this is now only configurable with user
> > configuration, and only one "remote-execution" block is ever
> > considered.
> >
> > There is no longer any ambiguity here, "remote-execution" applies to an
> > entire session, one build session cannot be built across multiple
> > different remote execution build clusters.
> >
> >
> > Artifact and Source cache configuration
> > ---------------------------------------
> > Projects are still allowed to provide recommendations for artifact and
> > source cache servers.
> >
> > User configuration now has the ability to override them, i.e. disregard
> > artifact and source cache servers declared in projects.
> >
> > Also, it is no longer possible to declare an artifact/source cache
> > server as a dictionary, it MUST be a list.
> >
> > This choice is simply because it the dict-or-list tactic here does not
> > buy us any convenience whatsoever, and clarity that it is in fact a
> > list of dictionaries is more worthwhile.
> >
> > Consider:
> > ~~~~~~~~~
> >
> > artifacts:
> > url: https://pony.moose/zebra:4040
> > push: true
> >
> > Versus:
> > ~~~~~~~
> >
> > artifacts:
> > - url: https://pony.moose/zebra:4040
> > push: true
> >
> > It is exactly the same amount of typing, no point in supporting both
> > here.
> >
> >
> > Project Configuration
> > ~~~~~~~~~~~~~~~~~~~~~
> >
> > #
> > # This is mostly unchanged, except for the `auth`
> > #
> > artifacts:
> > - url: https://pony.com:9999
> > type: both
> > push: false
> > instance-name: this-shard
> > auth:
> > server-cert: server.crt
> >
> >
> > User Configuration
> > ~~~~~~~~~~~~~~~~~~
> > We can declare global artifact configuration, which either
> > overrides or augments project recommended cache servers.
> >
> > When "augmenting", the user configuration is still at a higher priority
> > than the project recommendations (as in: user configuration caches will
> > be consulted *first* when interacting with remotes).
> >
> >
> > #
> > # Global artifact configuration
> > #
> > artifacts:
> >
> > #
> > # Here we decide whether user configuration overrides
> > # project recommendations.
> > #
> > override-project-caches: true
> >
> > #
> > # And we declare the global artifact configurations
> > # under the new "servers" sub-dictionary instead
> > #
> > servers:
> > - url: https://pony.com:9999
> > type: both
> > push: true
> > instance-name: this-shard
> > auth:
> > server-cert: server.crt
> > client-key: client.key
> > client-cert: client.crt
> >
> >
> > We can still declare artifact configuration in the overrides, with
> > exactly the same new configuration:
> >
> > #
> > # Artifact configuration for project "foo"
> > #
> > projects:
> > foo:
> >
> > artifacts:
> > #
> > # Lets completely override the cache for only project "foo"
> > #
> > override-project-caches: true
> >
> > #
> > # And declare the servers here
> > #
> > servers:
> > - url: https://pony.com:9999
> > type: both
> > push: true
> > instance-name: this-shard
> > auth:
> > server-cert: server.crt
> > client-key: client.key
> > client-cert: client.crt
> >
> >
> > Use case overview
> > =================
> > Here is an overview of the previously discussed desirable use cases.
> >
> > Inline responses to my initial proposal:
> >
> > [...]
> > > Use cases we want
> > > =================
> > > Here I will try to provide a birds eye view of what our use cases are,
> > > what does a BuildStream client application require from these
> > > services ?
> > >
> > > * The ability to store and retrieve artifacts on a remote artifact
> > > server.
> >
> > Of course.
> >
> > > * The ability to store and retrieve staged source packages, indexed
> > > by source cache key, on remote source cache services.
> >
> > Of course.
> >
> > > * The ability to farm out builds to a remote execution service
> >
> > Depending only on your ability to setup a remote execution build
> > cluster, of course.
> >
> > > * The ability to make requirements of worker instances on a remote
> > > execution service.
> > >
> > > - Possibly also the ability to bail out early if the remote
> > > execution service knows that it cannot provide a worker which
> > > with the properties which some of the project elements require.
> >
> > Nothing changes here thus far, although that is not to say we are
> > perfect in this regard yet, but this patch does not effect this.
> >
> > > * Ability to have redundancies in configuration of remote servers, in
> > > case a service is down we usually allow configuration of services
> > > in list format.
> >
> > We still have this.
> >
> > > * Ability to carry artifacts forward from a third party artifact
> > > cache which was recommended by project configuration across a
> > > junction boundary.
> > >
> > > I.e. for better repeatability, it is often desirable to re-cache
> > > the artifacts from an upstream project on your own infra in order
> > > to ensure you have your own copy.
> > >
> > > NOTE: This is currently only available in project data and not
> > > overridable by user configuration in the form of the
> > > `cache-junction-elements`[2] configuration, which I already
> > > pointed out was problematic in my original report[0].
> >
> > With this patch, we can achieve this use case by configuring a global
> > cache server for pushing, and setting `override-project-caches` to
> > `false`.
> >
> > The result will be:
> >
> > * When pulling, we will:
> >
> > - First try to pull from the globally defined cache servers
> > - Fall back on project defined cache servers
> >
> > * When pushing, we will:
> >
> > - First push to our globally defined cache servers
> > - Not push to the project defined cache servers
> >
> > Why will we not push to the project defined cache servers ?
> >
> > Well, it is currently only a consequence of the simple fact that
> > normally people simply do not configure "push" servers in a
> > project.conf, as that would likely imply that they are publishing the
> > private key needed to push to their server along with their project, so
> > that everyone and their dog can push anything they like to the artifact
> > server.
> >
> > We could additionally police this in the code and completely disallow
> > such nonsense configurations, but for now I've just left a fat notice
> > in the project.conf documentation which points out that is is a very
> > bad idea to configure a "push" remote from your project.conf (following
> > the "let people shoot their own feet if they really want to" policy).
> >
> > > * Ability to avoid downloading artifacts found on third party
> > > infrastructure.
> > >
> > > I.e. for better trustability, you may want to ensure that all of
> > > the built artifacts you end up consuming were built on
> > > infrastructure you control, rather than downloaded from an upstream
> > > project's artifact server.
> > >
> > > NOTE: This is currently only available in project data and not
> > > overridable by user configuration in the form of the
> > > `ignore-junction-remotes`[2] configuration, which I already
> > > pointed out was problematic in my original report[0].
> >
> > This is now possible by simply declaring `override-project-caches` to
> > `true` in the global configuration, regardless of whether or not you
> > have provided any remotes in your global configuration.
> >
> > > * Ability to farm out any local caching work a remote service, to
> > > reduce uploads and downloads for builds when configured on an RE
> > > service (by way of specifying the RE service's CAS here),
> > > configuring multiple build machines which run without RE may also
> > > be optimized by way of using the same remote CAS for this.
> > >
> > > NOTE: This has not yet landed and is a part of Jürg's ongoing
> > > work[3].
> >
> > This orthogonal feature has yet to land in master.
> >
> > > * Ability to clearly override the recommendation of any project.conf
> > > in the loaded pipeline using user configuration, which should
> > > always have the last word on any user configuration.
> >
> > This can be achieved on a per-project bases by setting the new
> > `override-project-caches` attribute to `true` in the overrides section
> > of the user configuration.
> >
> >
> >
> >
> >
Re: Proposal: Redesign of configuration surface for all remote services
Posted by Sander Striker <s....@striker.nl>.
Hi Tristan,
First of all, Thank You for the write up as well as the progress made so
far.
I'll be honest in that I am not sure I got the full picture of implications
- maybe as a result of your attempt to outline it in detail ;-). But
that's on me.
I have a couple of questions:
*Remote Execution*
There seems to be a skew towards only using a single service in the
proposal. There are some aspects to Remote Execution that we haven't
clearly articulated or fully implemented. We have only implemented
scenario 1 currently.
Scenario 1: Full Remote Execution [implemented]
- BuildStream checks ActionCache
- on cache hit, get ActionResult and blobs from CAS associated with
Remote Execution. Note: the blob content fetching could be on-demand
- on cache miss, uploads any missing blobs for the Action to the CAS
associated with Remote Execution, finally it uses the Execution service to
execute the Action.
Scenario 2: Remote Cache (read-only) [not currently implemented]
- BuildStream checks ActionCache
- on cache hit, get ActionResult and blobs from CAS associated with
Remote Execution. Note: the blob content fetching could be on-demand
- on cache miss fall back to local execution
Scenario 3: Remote Cache (read-write) [not currently implemented]
- BuildStream checks ActionCache
- on cache hit, get ActionResult and blobs from CAS associated with
Remote Execution. Note: the blob content fetching could be on-demand
- on cache miss fall back to local execution, and upload result blobs to
CAS associated with Remote Execution, finally write ActionResult to
ActionCache
Especially in case of Scenario 2, it is not unimaginable to have a project
propose a read-only endpoint. Getting these implemented is currently
mostly tricky due to the "fall back to local execution" bit - as we choose
to either remote or to locally execute up front.
*Write access configuration*
With the current credentials it may make sense to only have push be enabled
when auth credentials have been set up in the user configuration. To also
set `push: true` seems redundant if that is also set in project.conf. If
you don't want to push as a user, set `push: false` in user conf if you
have credentials configured.
At some point in the future, I can imagine credential prompting as well,
and I would imagine bst helps me out by caching those creds or e.g. storing
a JWT for later use.
Any thoughts on how the above fits with the current changes?
Cheers,
Sander
On Thu, Jan 28, 2021 at 9:12 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Hi all,
>
> Since there was no activity on this thread for a very long time, I
> decided to go ahead and take a crack at this.
>
> I have a good branch now that is ready for review. The MR is up here:
> https://github.com/apache/buildstream/pull/1453
>
>
> I'm sending a detailed email because it's a large proposal and I would
> like this to be visible, so that people can chime in incase we've
> missed an important use case.
>
> Cheers,
> -Tristan
>
>
> Here is the design/changes I've come up with.
> =============================================
>
>
> The offending junction configurations
> -------------------------------------
> The junction configurations:
>
> "cache-junction-elements"
> "ignore-junction-remotes"
>
> Are completely removed, reducing the worrisome ambiguity of what
> happens in what configuration.
>
> This is replaced by the enhanced user configuration.
>
>
> Authentication
> --------------
> For all of the authentication related properties, `server-cert`,
> `client-cert` and `client-key`, these have been split out into a
> subdictionary named "auth" for any remote configuration.
>
> This may allow better extensibility for alternative authentication
> methods in the future, however right now it serves us very well to be
> able to document the "auth" dictionary in one central place in the
> documentation.
>
>
> Remote Execution Configuration
> ------------------------------
> As described in this thread, this is now only configurable with user
> configuration, and only one "remote-execution" block is ever
> considered.
>
> There is no longer any ambiguity here, "remote-execution" applies to an
> entire session, one build session cannot be built across multiple
> different remote execution build clusters.
>
>
> Artifact and Source cache configuration
> ---------------------------------------
> Projects are still allowed to provide recommendations for artifact and
> source cache servers.
>
> User configuration now has the ability to override them, i.e. disregard
> artifact and source cache servers declared in projects.
>
> Also, it is no longer possible to declare an artifact/source cache
> server as a dictionary, it MUST be a list.
>
> This choice is simply because it the dict-or-list tactic here does not
> buy us any convenience whatsoever, and clarity that it is in fact a
> list of dictionaries is more worthwhile.
>
> Consider:
> ~~~~~~~~~
>
> artifacts:
> url: https://pony.moose/zebra:4040
> push: true
>
> Versus:
> ~~~~~~~
>
> artifacts:
> - url: https://pony.moose/zebra:4040
> push: true
>
> It is exactly the same amount of typing, no point in supporting both
> here.
>
>
> Project Configuration
> ~~~~~~~~~~~~~~~~~~~~~
>
> #
> # This is mostly unchanged, except for the `auth`
> #
> artifacts:
> - url: https://pony.com:9999
> type: both
> push: false
> instance-name: this-shard
> auth:
> server-cert: server.crt
>
>
> User Configuration
> ~~~~~~~~~~~~~~~~~~
> We can declare global artifact configuration, which either
> overrides or augments project recommended cache servers.
>
> When "augmenting", the user configuration is still at a higher priority
> than the project recommendations (as in: user configuration caches will
> be consulted *first* when interacting with remotes).
>
>
> #
> # Global artifact configuration
> #
> artifacts:
>
> #
> # Here we decide whether user configuration overrides
> # project recommendations.
> #
> override-project-caches: true
>
> #
> # And we declare the global artifact configurations
> # under the new "servers" sub-dictionary instead
> #
> servers:
> - url: https://pony.com:9999
> type: both
> push: true
> instance-name: this-shard
> auth:
> server-cert: server.crt
> client-key: client.key
> client-cert: client.crt
>
>
> We can still declare artifact configuration in the overrides, with
> exactly the same new configuration:
>
> #
> # Artifact configuration for project "foo"
> #
> projects:
> foo:
>
> artifacts:
> #
> # Lets completely override the cache for only project "foo"
> #
> override-project-caches: true
>
> #
> # And declare the servers here
> #
> servers:
> - url: https://pony.com:9999
> type: both
> push: true
> instance-name: this-shard
> auth:
> server-cert: server.crt
> client-key: client.key
> client-cert: client.crt
>
>
> Use case overview
> =================
> Here is an overview of the previously discussed desirable use cases.
>
> Inline responses to my initial proposal:
>
> [...]
> > Use cases we want
> > =================
> > Here I will try to provide a birds eye view of what our use cases are,
> > what does a BuildStream client application require from these
> > services ?
> >
> > * The ability to store and retrieve artifacts on a remote artifact
> > server.
>
> Of course.
>
> > * The ability to store and retrieve staged source packages, indexed
> > by source cache key, on remote source cache services.
>
> Of course.
>
> > * The ability to farm out builds to a remote execution service
>
> Depending only on your ability to setup a remote execution build
> cluster, of course.
>
> > * The ability to make requirements of worker instances on a remote
> > execution service.
> >
> > - Possibly also the ability to bail out early if the remote
> > execution service knows that it cannot provide a worker which
> > with the properties which some of the project elements require.
>
> Nothing changes here thus far, although that is not to say we are
> perfect in this regard yet, but this patch does not effect this.
>
> > * Ability to have redundancies in configuration of remote servers, in
> > case a service is down we usually allow configuration of services
> > in list format.
>
> We still have this.
>
> > * Ability to carry artifacts forward from a third party artifact
> > cache which was recommended by project configuration across a
> > junction boundary.
> >
> > I.e. for better repeatability, it is often desirable to re-cache
> > the artifacts from an upstream project on your own infra in order
> > to ensure you have your own copy.
> >
> > NOTE: This is currently only available in project data and not
> > overridable by user configuration in the form of the
> > `cache-junction-elements`[2] configuration, which I already
> > pointed out was problematic in my original report[0].
>
> With this patch, we can achieve this use case by configuring a global
> cache server for pushing, and setting `override-project-caches` to
> `false`.
>
> The result will be:
>
> * When pulling, we will:
>
> - First try to pull from the globally defined cache servers
> - Fall back on project defined cache servers
>
> * When pushing, we will:
>
> - First push to our globally defined cache servers
> - Not push to the project defined cache servers
>
> Why will we not push to the project defined cache servers ?
>
> Well, it is currently only a consequence of the simple fact that
> normally people simply do not configure "push" servers in a
> project.conf, as that would likely imply that they are publishing the
> private key needed to push to their server along with their project, so
> that everyone and their dog can push anything they like to the artifact
> server.
>
> We could additionally police this in the code and completely disallow
> such nonsense configurations, but for now I've just left a fat notice
> in the project.conf documentation which points out that is is a very
> bad idea to configure a "push" remote from your project.conf (following
> the "let people shoot their own feet if they really want to" policy).
>
> > * Ability to avoid downloading artifacts found on third party
> > infrastructure.
> >
> > I.e. for better trustability, you may want to ensure that all of
> > the built artifacts you end up consuming were built on
> > infrastructure you control, rather than downloaded from an upstream
> > project's artifact server.
> >
> > NOTE: This is currently only available in project data and not
> > overridable by user configuration in the form of the
> > `ignore-junction-remotes`[2] configuration, which I already
> > pointed out was problematic in my original report[0].
>
> This is now possible by simply declaring `override-project-caches` to
> `true` in the global configuration, regardless of whether or not you
> have provided any remotes in your global configuration.
>
> > * Ability to farm out any local caching work a remote service, to
> > reduce uploads and downloads for builds when configured on an RE
> > service (by way of specifying the RE service's CAS here),
> > configuring multiple build machines which run without RE may also
> > be optimized by way of using the same remote CAS for this.
> >
> > NOTE: This has not yet landed and is a part of Jürg's ongoing
> > work[3].
>
> This orthogonal feature has yet to land in master.
>
> > * Ability to clearly override the recommendation of any project.conf
> > in the loaded pipeline using user configuration, which should
> > always have the last word on any user configuration.
>
> This can be achieved on a per-project bases by setting the new
> `override-project-caches` attribute to `true` in the overrides section
> of the user configuration.
>
>
>
>
>
Re: Proposal: Redesign of configuration surface for all remote
services
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi,
Thanks for the reply Abderrahim :)
FWIW, the new PR (after the migration correction) is up at:
https://github.com/apache/buildstream/pull/1434
On Fri, 2021-02-05 at 21:13 +0100, Abderrahim Kitouni wrote:
> Hi Tristan,
[...]
> > Authentication
> > --------------
> > For all of the authentication related properties, `server-cert`,
> > `client-cert` and `client-key`, these have been split out into a
> > subdictionary named "auth" for any remote configuration.
> >
> > This may allow better extensibility for alternative authentication
> > methods in the future, however right now it serves us very well to be
> > able to document the "auth" dictionary in one central place in the
> > documentation.
>
> I think it's better to keep `server-cert` out of the sub-dictionary.
> The server certificate is needed when the server is using an untrusted
> certificate regardless of whether we want to authenticate with the
> server and regardless of the method we use to authenticate (currently
> client certificate, but we may want to support some kind of access
> token in the future like buildbox)
I understand that the YAML might be one dict less wordy in a lot of
cases where you only want to download, and I also take your point that
the server cert is not exactly for the purpose of "authenticating the
client".
The main motivations behind the `auth` dict are:
* We can clearly document the authentication related attributes in a
single location.
The key/cert exchange is a bit confusing in my opinion, I wanted
to rename them in terms of "pull" and "push" credentials so that
the user could be unburdened of the knowledge of how key exchanges
work, but that is also incorrect.
I think that the client/server cert/key is all clearly the same
topic / subject material, and as it is used in multiple places
(different kinds of remote asset services, remote execution
services), it makes good sense to have a central place to
document this.
* Extensibility.
Right now we only support http/https URIs, but this could
hypothetically be extended in the future.
By having an `auth` dict we can naturally add a `kind` field
to the dict and support different types of authentication in
the future in a nice clean component.
> > Artifact and Source cache configuration
> > ---------------------------------------
> > Projects are still allowed to provide recommendations for artifact and
> > source cache servers.
> >
> > User configuration now has the ability to override them, i.e. disregard
> > artifact and source cache servers declared in projects.
> >
> > Also, it is no longer possible to declare an artifact/source cache
> > server as a dictionary, it MUST be a list.
> >
> > This choice is simply because it the dict-or-list tactic here does not
> > buy us any convenience whatsoever, and clarity that it is in fact a
> > list of dictionaries is more worthwhile.
>
> I think this makes sense.
>
> > Project Configuration
> > ~~~~~~~~~~~~~~~~~~~~~
> >
> > #
> > # This is mostly unchanged, except for the `auth`
> > #
> > artifacts:
> > - url: https://pony.com:9999
> > type: both
> > push: false
> > instance-name: this-shard
> > auth:
> > server-cert: server.crt
>
> I assume `both` here means index+storage? (as in CAS + RA)
Yes, that is the current name (I didn't change this), although we could
rename this if you have a better name in mind (now is a good time...).
> > User Configuration
> > ~~~~~~~~~~~~~~~~~~
> > We can declare global artifact configuration, which either
> > overrides or augments project recommended cache servers.
> >
> > When "augmenting", the user configuration is still at a higher priority
> > than the project recommendations (as in: user configuration caches will
> > be consulted *first* when interacting with remotes).
>
> I think this makes a lot of sense, but is the reverse of what bst1
> does. I hope there won't be objections to changing it.
Indeed.
The point is that all of user configuration can override project
suggestions, which has always been the design, e.g. see:
https://docs.buildstream.build/master/arch_data_model.html#context
The cache service configuration went a bit off the reservation in this
regard so I want to bring it back in line :)
> > #
> > # Global artifact configuration
> > #
> > artifacts:
> >
> > #
> > # Here we decide whether user configuration overrides
> > # project recommendations.
> > #
> > override-project-caches: true
>
> This would be useful as a command line argument too.
Yup, that is already included in my follow up branch :)
https://github.com/apache/buildstream/pull/1438
Cheers,
-Tristan
Re: Proposal: Redesign of configuration surface for all remote services
Posted by Abderrahim Kitouni <a....@gmail.com>.
Hi Tristan,
Le jeu. 28 janv. 2021 à 09:12, Tristan Van Berkom
<tr...@codethink.co.uk> a écrit :
>
> Hi all,
>
> Since there was no activity on this thread for a very long time, I
> decided to go ahead and take a crack at this.
>
> I have a good branch now that is ready for review. The MR is up here:
> https://github.com/apache/buildstream/pull/1453
>
>
> I'm sending a detailed email because it's a large proposal and I would
> like this to be visible, so that people can chime in incase we've
> missed an important use case.
This is something I've been meaning to reply to for a long time, but
never found the time. I've now gone through your proposal, and I must
say I like it. A couple remarks inline:
> Authentication
> --------------
> For all of the authentication related properties, `server-cert`,
> `client-cert` and `client-key`, these have been split out into a
> subdictionary named "auth" for any remote configuration.
>
> This may allow better extensibility for alternative authentication
> methods in the future, however right now it serves us very well to be
> able to document the "auth" dictionary in one central place in the
> documentation.
I think it's better to keep `server-cert` out of the sub-dictionary.
The server certificate is needed when the server is using an untrusted
certificate regardless of whether we want to authenticate with the
server and regardless of the method we use to authenticate (currently
client certificate, but we may want to support some kind of access
token in the future like buildbox)
> Artifact and Source cache configuration
> ---------------------------------------
> Projects are still allowed to provide recommendations for artifact and
> source cache servers.
>
> User configuration now has the ability to override them, i.e. disregard
> artifact and source cache servers declared in projects.
>
> Also, it is no longer possible to declare an artifact/source cache
> server as a dictionary, it MUST be a list.
>
> This choice is simply because it the dict-or-list tactic here does not
> buy us any convenience whatsoever, and clarity that it is in fact a
> list of dictionaries is more worthwhile.
I think this makes sense.
> Project Configuration
> ~~~~~~~~~~~~~~~~~~~~~
>
> #
> # This is mostly unchanged, except for the `auth`
> #
> artifacts:
> - url: https://pony.com:9999
> type: both
> push: false
> instance-name: this-shard
> auth:
> server-cert: server.crt
I assume `both` here means index+storage? (as in CAS + RA)
> User Configuration
> ~~~~~~~~~~~~~~~~~~
> We can declare global artifact configuration, which either
> overrides or augments project recommended cache servers.
>
> When "augmenting", the user configuration is still at a higher priority
> than the project recommendations (as in: user configuration caches will
> be consulted *first* when interacting with remotes).
I think this makes a lot of sense, but is the reverse of what bst1
does. I hope there won't be objections to changing it.
> #
> # Global artifact configuration
> #
> artifacts:
>
> #
> # Here we decide whether user configuration overrides
> # project recommendations.
> #
> override-project-caches: true
This would be useful as a command line argument too.
Regards,
Abderrahim
Re: Proposal: Redesign of configuration surface for all remote
services
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
Since there was no activity on this thread for a very long time, I
decided to go ahead and take a crack at this.
I have a good branch now that is ready for review. The MR is up here:
https://github.com/apache/buildstream/pull/1453
I'm sending a detailed email because it's a large proposal and I would
like this to be visible, so that people can chime in incase we've
missed an important use case.
Cheers,
-Tristan
Here is the design/changes I've come up with.
=============================================
The offending junction configurations
-------------------------------------
The junction configurations:
"cache-junction-elements"
"ignore-junction-remotes"
Are completely removed, reducing the worrisome ambiguity of what
happens in what configuration.
This is replaced by the enhanced user configuration.
Authentication
--------------
For all of the authentication related properties, `server-cert`,
`client-cert` and `client-key`, these have been split out into a
subdictionary named "auth" for any remote configuration.
This may allow better extensibility for alternative authentication
methods in the future, however right now it serves us very well to be
able to document the "auth" dictionary in one central place in the
documentation.
Remote Execution Configuration
------------------------------
As described in this thread, this is now only configurable with user
configuration, and only one "remote-execution" block is ever
considered.
There is no longer any ambiguity here, "remote-execution" applies to an
entire session, one build session cannot be built across multiple
different remote execution build clusters.
Artifact and Source cache configuration
---------------------------------------
Projects are still allowed to provide recommendations for artifact and
source cache servers.
User configuration now has the ability to override them, i.e. disregard
artifact and source cache servers declared in projects.
Also, it is no longer possible to declare an artifact/source cache
server as a dictionary, it MUST be a list.
This choice is simply because it the dict-or-list tactic here does not
buy us any convenience whatsoever, and clarity that it is in fact a
list of dictionaries is more worthwhile.
Consider:
~~~~~~~~~
artifacts:
url: https://pony.moose/zebra:4040
push: true
Versus:
~~~~~~~
artifacts:
- url: https://pony.moose/zebra:4040
push: true
It is exactly the same amount of typing, no point in supporting both
here.
Project Configuration
~~~~~~~~~~~~~~~~~~~~~
#
# This is mostly unchanged, except for the `auth`
#
artifacts:
- url: https://pony.com:9999
type: both
push: false
instance-name: this-shard
auth:
server-cert: server.crt
User Configuration
~~~~~~~~~~~~~~~~~~
We can declare global artifact configuration, which either
overrides or augments project recommended cache servers.
When "augmenting", the user configuration is still at a higher priority
than the project recommendations (as in: user configuration caches will
be consulted *first* when interacting with remotes).
#
# Global artifact configuration
#
artifacts:
#
# Here we decide whether user configuration overrides
# project recommendations.
#
override-project-caches: true
#
# And we declare the global artifact configurations
# under the new "servers" sub-dictionary instead
#
servers:
- url: https://pony.com:9999
type: both
push: true
instance-name: this-shard
auth:
server-cert: server.crt
client-key: client.key
client-cert: client.crt
We can still declare artifact configuration in the overrides, with
exactly the same new configuration:
#
# Artifact configuration for project "foo"
#
projects:
foo:
artifacts:
#
# Lets completely override the cache for only project "foo"
#
override-project-caches: true
#
# And declare the servers here
#
servers:
- url: https://pony.com:9999
type: both
push: true
instance-name: this-shard
auth:
server-cert: server.crt
client-key: client.key
client-cert: client.crt
Use case overview
=================
Here is an overview of the previously discussed desirable use cases.
Inline responses to my initial proposal:
[...]
> Use cases we want
> =================
> Here I will try to provide a birds eye view of what our use cases are,
> what does a BuildStream client application require from these
> services ?
>
> * The ability to store and retrieve artifacts on a remote artifact
> server.
Of course.
> * The ability to store and retrieve staged source packages, indexed
> by source cache key, on remote source cache services.
Of course.
> * The ability to farm out builds to a remote execution service
Depending only on your ability to setup a remote execution build
cluster, of course.
> * The ability to make requirements of worker instances on a remote
> execution service.
>
> - Possibly also the ability to bail out early if the remote
> execution service knows that it cannot provide a worker which
> with the properties which some of the project elements require.
Nothing changes here thus far, although that is not to say we are
perfect in this regard yet, but this patch does not effect this.
> * Ability to have redundancies in configuration of remote servers, in
> case a service is down we usually allow configuration of services
> in list format.
We still have this.
> * Ability to carry artifacts forward from a third party artifact
> cache which was recommended by project configuration across a
> junction boundary.
>
> I.e. for better repeatability, it is often desirable to re-cache
> the artifacts from an upstream project on your own infra in order
> to ensure you have your own copy.
>
> NOTE: This is currently only available in project data and not
> overridable by user configuration in the form of the
> `cache-junction-elements`[2] configuration, which I already
> pointed out was problematic in my original report[0].
With this patch, we can achieve this use case by configuring a global
cache server for pushing, and setting `override-project-caches` to
`false`.
The result will be:
* When pulling, we will:
- First try to pull from the globally defined cache servers
- Fall back on project defined cache servers
* When pushing, we will:
- First push to our globally defined cache servers
- Not push to the project defined cache servers
Why will we not push to the project defined cache servers ?
Well, it is currently only a consequence of the simple fact that
normally people simply do not configure "push" servers in a
project.conf, as that would likely imply that they are publishing the
private key needed to push to their server along with their project, so
that everyone and their dog can push anything they like to the artifact
server.
We could additionally police this in the code and completely disallow
such nonsense configurations, but for now I've just left a fat notice
in the project.conf documentation which points out that is is a very
bad idea to configure a "push" remote from your project.conf (following
the "let people shoot their own feet if they really want to" policy).
> * Ability to avoid downloading artifacts found on third party
> infrastructure.
>
> I.e. for better trustability, you may want to ensure that all of
> the built artifacts you end up consuming were built on
> infrastructure you control, rather than downloaded from an upstream
> project's artifact server.
>
> NOTE: This is currently only available in project data and not
> overridable by user configuration in the form of the
> `ignore-junction-remotes`[2] configuration, which I already
> pointed out was problematic in my original report[0].
This is now possible by simply declaring `override-project-caches` to
`true` in the global configuration, regardless of whether or not you
have provided any remotes in your global configuration.
> * Ability to farm out any local caching work a remote service, to
> reduce uploads and downloads for builds when configured on an RE
> service (by way of specifying the RE service's CAS here),
> configuring multiple build machines which run without RE may also
> be optimized by way of using the same remote CAS for this.
>
> NOTE: This has not yet landed and is a part of Jürg's ongoing
> work[3].
This orthogonal feature has yet to land in master.
> * Ability to clearly override the recommendation of any project.conf
> in the loaded pipeline using user configuration, which should
> always have the last word on any user configuration.
This can be achieved on a per-project bases by setting the new
`override-project-caches` attribute to `true` in the overrides section
of the user configuration.