Re: Master configuration in the registry

[Benjamin Mahler <bm...@apache.org>]

+user

I explicitly didn't mention "update without restart" as a criterion because
you could use a watched configuration file to achieve this, like we did for
the whitelist. If the master or agent observe any changes to the
configuration file then they can apply them. If they cannot apply the
changes without restarting (or at all!), they could exit or ignore/warn.

If we expect there to be natural churn in a particular configuration via
tooling, then the watched file approach adds some friction when wanting to
make interactive changes. Accordingly, the only ones I would have excluded
from your list is the ping timeouts and the max completed frameworks/tasks,
as these don't seem like things that we expect to have natural churn over
time via tooling.

Still, some of these I can imagine operators having a preference for
configuration files or flags over the endpoint + registry persistence:

    e.g. I may want to use a private credentials file rather than storing
it in the master's registry.
    e.g. I may prefer to "deploy" new rate limit rules using my existing
configuration deploy tooling, because hitting an endpoint currently
requires me to find the leader, hit the endpoint, deal with it failing over
while the endpoint is being hit, etc.
    e.g. I want to enforce that the master comes up with these ACLs,
without having to do a two phase procedure which leaves the master
vulnerable with no ACLs while I quickly try to set ACLs!

My thought here is that we may need an approach that allows the operator to
choose between flags/files and endpoints for setting some of the "dynamic"
configuration, or provide a means to solve each specific problem that
arises with using endpoints (examples listed above).

Curious if users/operators have any thoughts on this.

Ben

On Sat, Jun 11, 2016 at 4:20 AM, Adam Bordelon <ad...@mesosphere.io> wrote:

> The motivation for "dynamic" quota/weights was that we wanted operators to
> be able to change these without restarting (all) masters. That means you'd
> update these values via operator endpoints, and if you're doing that, then
> the flag values become outdated/irrelevant, and we must persist these
> values somewhere that they can be restored on master failover (i.e.
> registry). However, we have kept the flags around for bootstrapping and
> backwards compatibility.
>
> Other flags that could benefit from dynamic updates (without restarting
> masters):
> - acls
> - credentials
> - firewall_rules
> - rate_limits
> - ping timeouts
> - max completed frameworks/tasks
> - whitelist (actually reloads the file every so often)
>
> On Fri, Jun 10, 2016 at 4:29 AM, Neil Conway <ne...@gmail.com>
> wrote:
>
> > Makes sense: arguably you could say that "quota" and "weights" are
> > part of the master's (mutable) "state", not its "configuration", which
> > is largely immutable.
> >
> > Another distinction is that some configuration flags control behavior
> > that doesn't need to be consistent between master replicas (e.g.,
> > "--ip", "--port", "--advertise-ip", "--advertise-port", "--hostname",
> > "--hostname-lookup", "--quiet", "--log_dir", "--modules_dir",
> > "--work_dir", etc).
> >
> > Neil
> >
> > On Fri, Jun 10, 2016 at 3:52 AM, Benjamin Mahler <bm...@apache.org>
> > wrote:
> > > I'm curious to hear thoughts on the distinction between using flags and
> > > persisting in the registry for master configuration. This topic had
> come
> > up
> > > in a discussion and our current choices are intuitive but the criteria
> > were
> > > not immediately obvious.
> > >
> > > Two cases seem interesting to me:
> > >
> > > (1) Quota.
> > > (2) Weights.
> > >
> > > These are configuration, but we persist them in the registry. Why is
> > that?
> > >
> > > My intuition is that they reflect the organizational aspects of the
> > > workloads that are running and so we expect administrators and (most
> > > importantly!!) tooling to be view and modify these over time.
> > >
> > > Timeouts, work directories, etc, on the other hand, are rarely modified
> > and
> > > require initial values. There are also sane defaults for these that
> will
> > > work for most users.
> > >
> > > Thought this might be helpful for others that may wonder about this.
> Let
> > me
> > > know if there are any other important criteria that I've missed.
> > >
> > > Ben
> >
>