Re: Config in ZooKeeperInstance

[Michael Berman <mb...@sqrrl.com>]

Ok, I think the "there's no point in client-side detection because it's not
any more valuable a production deployment than insecure" argument convinced
me.  Especially since it would be easy enough to add later if someone does
find a use case for it.

SSL for ZK connections would definitely be a valuable thing.  It sounds
like now that ZK has the option to use netty for NIO connections, SSL
should theoretically be possible, but I can't find any evidence that anyone
has actually tried.  While I think it makes sense as an eventual goal, it's
probably not as important as SSL for client-to-accumulo, since I imagine
typically ZK and accumulo are both on pretty trusted networks, whereas
accumulo clients are more likely to be coming from less trusted networks.

I think having a flag to allow instances to fail fast makes sense.  I'll
take a look at that once I get the mainline scenario built out.


On Wed, Jul 31, 2013 at 4:54 PM, Christopher <ct...@apache.org> wrote:

> I'm not sure I see much point implementing server-side features to
> support a "use SSL, if available" client mode. Such a feature is
> effectively no more secure than the basic "insecure" mode, and as
> such, I think is a waste of resources to implement (and, more
> importantly, to maintain). Besides, clients can implement a "use SSL,
> if available" feature (if they really must) without the server side
> revealing any information about how it is configured (your suggested
> "try, then fallback on failure" would work, and it wouldn't be too
> bad, if you did it once per "Instance"
> (org.apache.accumulo.core.client.Instance).
>
> The "connection string" for Accumulo *is* client configuration, so
> that seems the most sensible route to me.
> For example, one of:
>
> // one of these could respect the JSSE javax.net.ssl system properties
> ZooKeeperInstance(instanceName, zookeepers,
> secureMode).getConnector(user, token);
> SecureZooKeeperInstance(instanceName, zookeepers).getConnector(user,
> token);
> Instance(instanceName, zookeepers).getSslConnector(user, token);
>
> On the other hand, if you mean per-tserver connection strings
> (locations in metadata or server advertisements/locks in ZK), I'd
> steer clear, because you really need such a setting consistent for
> everything related to an entire o.a.a.core.client.Instance. If even
> one tserver is insecure and operating within the cluster, and the
> client is configured to fallback to insecure mode on a per-connection
> basis, then the whole system is insecure.
>
> You could implement a ZK flag that reports which mode (set by the
> Master) the entire instance is running in, but I think this should be
> used only as a fast-failure when constructing a secure client
> connector. Otherwise, if the client is going to change its operation
> mode based on this flag, you're going to have to set up an additional
> layer of trust with ZK before you can trust that it's safe to respect
> this flag in the first place (because the flag has the possibility of
> forcing an insecure mode).
>
> Setting up that trust seems complicated and unnecessary to me (I
> really don't like the idea of an insecure fallback mode), but it might
> be worth it for other reasons anyway. I'm not sure what options are
> available to lock down ZK more. Perhaps an additional shared secret
> for clients (again, distributed as client configuration)? or
> client-certificate authentication for the read-only client areas of
> ZK?
>
> I suggest:
>
> Have the master set a flag in ZK in an area that the client can read,
> to inform the client which mode the server is operating in, for
> fast-failure only (to enable client logic to fall back, if desired).
> Provide an Instance that will use SSL mode for all its RPC calls, and
> will fail if the server's flag doesn't match the specified mode.
> Make clients configurable, to use either the secure Instance or the
> insecure Instance (do this entirely in client code).
> Do additional research on securing/trusting ZK itself.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 31, 2013 at 11:52 AM, Michael Berman <mb...@sqrrl.com>
> wrote:
> > Oh, I absolutely agree we can't trust remote config to tell us where to
> > find the trusted root...that needs to be consciously configured for every
> > client.  I was thinking more along the lines of your "http can
> > automatically redirect to https" example.  If I have an accumulo cluster
> up
> > and running, and I want to switch to SSL, it would be nice to be able to
> > flip the switch on the server, distribute my root cert, and then have all
> > the existing client apps able to figure out that they should make SSL
> > connections to accumulo automatically based on some cue from ZK.  If it's
> > really entirely up to the client to decide, probably that switch would
> > require a recompile of my client apps to add the "use SSL" flag.  I
> > wouldn't want that flag to be in a conf file (although I think that's a
> > great place for the path to the root cert), because on any given machine
> I
> > may want to connect to both SSL and non-SSL accumulos...it seems like a
> > connection-specific setting, not a global machine setting.  I do
> definitely
> > think there's a place in the client interface for "require SSL"...as in,
> > I'm only interested in connecting to accumulo if I can get an SSL
> > connection.  But I think if that's not set, it would be nice for the
> > behavior to be "I don't care" rather than "require non-SSL", in which
> case
> > we do need some hint from ZK.  You have convinced me that we don't want
> the
> > whole config in there, so maybe it should just be a component of the
> > connection string, or a standalone flag.  I suppose another option is to
> > always try SSL first, and if that fails, fall back to non-SSL, but that
> > seems like a lot of overhead every time we try to make a connection,
> > especially if the common case is not SSL.
> >
> >
> > On Tue, Jul 23, 2013 at 5:02 PM, Christopher <ct...@apache.org>
> wrote:
> >
> >> It should really be a conscious decision whether to create a client
> >> that expects SSL enabled or not. It shouldn't just happen
> >> automatically.
> >>
> >> For comparison, imagine if HTTP/SSL worked this way... when I type
> >> https://.... I expect it to be serving over SSL... not just
> >> auto-negotiate to secure or non-secure mode. If it's not in secure
> >> mode when I navigate to https://... , I simply don't want to go there.
> >> Now, browsers do auto-negotiate in the other direction, to secure mode
> >> (via HTTP redirects to HTTPS addresses), but that only works because
> >> your browser has a set of built-in certificate authorities that it
> >> trusts. If it doesn't, your browser gives you an error (if it's a good
> >> browser) or a warning (if it's a weak one). So, even auto-negotiating
> >> to secure mode is problematic if you can't control from the client,
> >> which certificates the client is willing to trust in that secure
> >> communication. We have no sensible way to distribute certificates that
> >> clients should trust. Sysadmins should generate their own certificates
> >> and configure clients appropriately.
> >>
> >> So, I don't see a way around it, you're going to have to make the
> >> client code aware that SSL is the desired mode of communication...
> >> either through a constructor parameter to ZooKeeperInstance, or
> >> through another Instance sub-class... perhaps one that reads a
> >> configuration file from the user's home directory with standard JSSE
> >> properties (
> >>
> http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html
> >> ).
> >> You should not just automatically retrieve this from the server's
> >> configuration in ZooKeeper, because that assumes you already trust the
> >> server, and that undermines the whole point of the authentication
> >> component of SSL.
> >>
> >> There's also the question of enabling SSL on ZK itself. I think you'd
> >> probably want to do that, too. You're still going to need to seed the
> >> SSL attributes in the client configuration, though, before you talk to
> >> any other server. I suppose you could trust ZK, and manage keys there
> >> (still seeding the client, but only with the certs / ssl mode needed
> >> to talk directly to ZK)... but I think that might be overkill for a
> >> first pass (especially since you can probably re-use the same server
> >> cert for ZK as you can for the other servers... since they are
> >> operating as a system). I think a simpler and more reliable solution
> >> is to enable the easy configuration of standard JSSE properties in
> >> both the client and the server.
> >>
> >> Now, what we could do is leave the configuration getter/setter on the
> >> instance, and require setting JSSE options for SSL in that, via
> >> setConfiguration(), though I'm personally a fan of getting them
> >> automatically from a properties file instead (like the
> >> $HOME/.accumulorc or $HOME/.accumulo/config suggestion made on
> >> ACCUMULO-1045, or maybe even /etc/accumulo/clients.conf or similar).
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >>
> >> On Tue, Jul 23, 2013 at 1:38 PM, Michael Berman <mb...@sqrrl.com>
> wrote:
> >> > The reason I'm interested in the config from a ZooKeeperInstance is
> >> because
> >> > a ZKInstance needs to know whether or not to use SSL to try to connect
> >> to a
> >> > given cluster.  I think it would be annoying to require a ZKInstance
> >> client
> >> > to explicitly specify "useSsl" any time they want to connect to an SSL
> >> > tserver (although I do see value in having a "requireSsl" optional
> flag).
> >> >  If configuration isn't the right way to communicate this, do people
> have
> >> > other suggestions?  I suppose I could just throw an ssl:// on the
> front
> >> of
> >> > the connection string or something, but it seems kludgy to do that any
> >> time
> >> > there's a configuration option that we want clients to know about.
> >> >
> >> > As far as removing Instance.getConfiguration() entirely, there are
> about
> >> 30
> >> > references throughout the codebase to getConfiguration() on a bare
> >> > Instance.  Finding some other way to plumb whatever options those
> >> consumers
> >> > care about through the stack sounds like a substantial project.
> >> >
> >> > In my use case, I am only interested in the configuration that's
> stored
> >> in
> >> > ZK, and I already have enough information to connect to ZK, so it
> doesn't
> >> > seem difficult technically, just a question of desired behavior.
> >> >
> >> >
> >> > On Tue, Jul 23, 2013 at 1:11 PM, Eric Newton <er...@gmail.com>
> >> wrote:
> >> >
> >> >> +1 for removing it from Instance
> >> >>
> >> >>
> >> >> On Tue, Jul 23, 2013 at 12:14 PM, Christopher <ct...@apache.org>
> >> wrote:
> >> >>
> >> >> > It only returns the default config if you don't call
> >> >> > setConfiguration(), which appears to be almost always (except
> >> >> > TestIngest).
> >> >> >
> >> >> > I don't know that this API is clearly spelled out, as to its
> intended
> >> >> > purpose. Which configuration is it supposed to be getting, and how
> >> >> > does that relate to the ZooKeeperInstance? The only configuration a
> >> >> > ZooKeeperInstance has is the minimum needed to connect to other
> >> >> > servers. It still has to authenticate to read any other server
> >> >> > configuration.
> >> >> >
> >> >> > Personally, I'd be in favor of removing it from Instance interface,
> >> >> > unless we actually document what it is supposed to be for to
> justify
> >> >> > its utility in, and relationship to, the Instance interface.
> >> >> >
> >> >> > --
> >> >> > Christopher L Tubbs II
> >> >> > http://gravatar.com/ctubbsii
> >> >> >
> >> >> >
> >> >> > On Tue, Jul 23, 2013 at 10:44 AM, Michael Berman <
> mberman@sqrrl.com>
> >> >> > wrote:
> >> >> > > I was surprised to find that ZooKeeperInstance.getConfiguration()
> >> seems
> >> >> > to
> >> >> > > return only the default configuration rather than the
> configuration
> >> >> > stored
> >> >> > > in ZK.  Is this the expected behavior?  I came across this in a
> >> >> MacTest,
> >> >> > so
> >> >> > > if other people are also surprised, it may be a MAC or
> >> MacTest-specific
> >> >> > > issue, in which case I'm happy to track it down.
> >> >> > >
> >> >> > > If this is currently the expected behavior, how would people feel
> >> about
> >> >> > > changing it?  It seems like it would be useful to have a config
> >> channel
> >> >> > to
> >> >> > > ZooKeeperInstance clients (in my case, what I'm specifically
> >> interested
> >> >> > in
> >> >> > > is whether or not SSL is enabled).  There may be a potential for
> >> >> > privileged
> >> >> > > information to escape...table settings, for example, may be
> >> >> sensitive...
> >> >> > > But all the really secret stuff should be in the site.xml which
> >> >> wouldn't
> >> >> > > get exposed.
> >> >> > >
> >> >> > > For reference, ZooKeeperInstance's getConfiguration() is
> implemented
> >> >> as:
> >> >> > > AccumuloConfiguration.getDefaultConfiguration()
> >> >> > >
> >> >> > > whereas HdfsZooInstance's getConfiguration() is:
> >> >> > > ZooConfiguration.getInstance(this, getSiteConfiguration())
> >> >> > >
> >> >> > > My proposal would be to change ZooKeeperInstance's to something
> >> like:
> >> >> > > ZooConfiguration.getInstance(this,
> >> >> > > AccumuloConfiguration.getDefaultConfiguration())
> >> >> > >
> >> >> > > Michael
> >> >> >
> >> >>
> >>
>