Kudu: potential metadata changes

[Andrew Wong <aw...@apache.org>]

Hi Kudu devs!

There've been a few discussions recently regarding changes to Kudu's
metadata storage. There are a number of areas that could benefit from
improving this layer, and I've been coalescing some of these ideas to lock
down what changes make sense in the near future. Here's a list of a few
considerations:

https://docs.google.com/document/d/1jXFqIZvLwkkmSjLC0wy-mDA0l0GmVuZJUMUB_7q5QUE/edit?usp=sharing

To summarize some recent discussions, there are a few general changes that
would be beneficial, in order of implementation difficulty:

1. Moving the metadata to the WAL directory (KUDU-1489)
2. Batch commit of metadata (KUDU-2204)
3. Mirroring of metadata (KUDU-2117)

1. and 3. mainly serve to reduce the points of failure (although 1. also
has performance benefits if the WAL is SSD-mounted), while 2. could solve
potential scalability issues with elections, update-heavy workloads, etc.

Empirically, scalability is not as big of an adoption-bottleneck as it was
before. Additionally, it's not clear that the listed scalability issues are
the biggest bottlenecks to larger data volumes. Moving forward, we should
keep track of user stories that would benefit from such improvements.

There hasn't been a huge amount of noise regarding the two remaining points
of failure (the WALs directory and metadata directory); it's unclear
whether this is due to an acceptance of the fact, or due to the recency of
the release (Kudu 1.6). In any case, the simplicity of 1. makes it the more
likely candidate in the nearer future, provided it doesn't preclude future
work on 2. or 3.

As always, backwards compatibility and downgrades are concerns, and of the
three options, 1. is by far the easiest, as (unlike the others) it will
likely not change any on-disk formatting. That said, we'll have to put in
some thought moving forward, e.g. if we do end up wanting to implement 2.
and 3., the upgrade paths would be a bit trickier with the added
configuration.

With these points in mind, it seems the reasonable path forward is go with
1. and introduce a flag for users to colocate WALs and metadata. If you
have thoughts or comments on the matter, feel free to respond!


Best,
Andrew

Re: Kudu: potential metadata changes

[Andrew Wong <aw...@cloudera.com.INVALID>]

Hi folks,

I'd like to give an update to this thread, noting some metadata changes that
have gone in over the past months, and some further metadata changes that I
thinkare worth making. The considerations from the previous document are
still
valid; I'd like to paint a slightly more concrete picture of the changes I
think can go in.

- the previously-discussed option to allow users to configure the placement
of
  metadata has landed; the new default metadata directory is the WALs
directory
- the optimization to store min/max keys in the rowset metadata is exposed
to
  users via a flag

What's left is, well, the rest of it. We still have no easy way to make
storage
changes to metadata. Nor do we have the abstractions to address all of the
issues mentioned last time. I've put together a doc
<https://docs.google.com/document/d/1ijh3SGFAR2VXAvT5L1stVPgadLLm1aGWCfJjTdijpX0/edit?usp=sharing>
that outlines a couple
potential steps in that direction and some other noteworthy issues with our
current metadata story. I would appreciate some feedback.

Thanks,
Andrew


On Wed, Jan 10, 2018 at 3:11 PM Todd Lipcon <to...@cloudera.com> wrote:

> On Wed, Jan 10, 2018 at 11:57 AM, Adar Lieber-Dembo <ad...@cloudera.com>
> wrote:
>
> >
> > As far as specific bottlenecks are concerned, the spreading of LBM
> > metadata across multiple files is a main contributor to our long
> > startup times, and that's one of the biggest scalability bottlenecks
> > AFAIK.
> >
>
> I think the context of the doc Andrew posted is more about tablet-meta and
> consensus-meta and less about LBM meta. Improving LBM meta to be more
> efficiently stored (perhaps rolled into the same storage as other bits of
> metadata) is a nice idea as well, and maybe there are some shared bits of
> infrastructure here though?
>
> (eg if we imported rocksdb, lmdb, or built a simple KV-store abstraction
> maybe we could use it for all these use cases equally)
>
>
> >
> > > With these points in mind, it seems the reasonable path forward is go
> > with
> > > 1. and introduce a flag for users to colocate WALs and metadata.
> >
> > Makes sense. Will colocation be enabled by default for new clusters?
> > How about using the flag to define an explicit metadata directory,
> > with the default being empty ("use the WAL directory")? That'd make it
> > similar to fs_data_dirs, which is nice. If the metadata directory
> > can't be found in directory specified by this gflag (or in the WAL
> > directory, if blank), we can fall back to looking in the first data
> > directory.
> >
>
> That sounds reasonable to me.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>


-- 
Andrew Wong

Re: Kudu: potential metadata changes

[Todd Lipcon <to...@cloudera.com>]

On Wed, Jan 10, 2018 at 11:57 AM, Adar Lieber-Dembo <ad...@cloudera.com>
wrote:

>
> As far as specific bottlenecks are concerned, the spreading of LBM
> metadata across multiple files is a main contributor to our long
> startup times, and that's one of the biggest scalability bottlenecks
> AFAIK.
>

I think the context of the doc Andrew posted is more about tablet-meta and
consensus-meta and less about LBM meta. Improving LBM meta to be more
efficiently stored (perhaps rolled into the same storage as other bits of
metadata) is a nice idea as well, and maybe there are some shared bits of
infrastructure here though?

(eg if we imported rocksdb, lmdb, or built a simple KV-store abstraction
maybe we could use it for all these use cases equally)


>
> > With these points in mind, it seems the reasonable path forward is go
> with
> > 1. and introduce a flag for users to colocate WALs and metadata.
>
> Makes sense. Will colocation be enabled by default for new clusters?
> How about using the flag to define an explicit metadata directory,
> with the default being empty ("use the WAL directory")? That'd make it
> similar to fs_data_dirs, which is nice. If the metadata directory
> can't be found in directory specified by this gflag (or in the WAL
> directory, if blank), we can fall back to looking in the first data
> directory.
>

That sounds reasonable to me.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Kudu: potential metadata changes

[Andrew Wong <aw...@apache.org>]

>
> Kudu users are assured that their deployments can keep growing as
> Kudu continues to mature.


That's a good point. AFAICT we haven't seen many users complaining about
scalability in their clusters other than maybe slow start times. It's hard
to tell how near or far over the horizon these issues will be, but for now,
keeping an eye out for them should be enough to guide future
decision-making.

How about using the flag to define an explicit metadata directory,
> with the default being empty ("use the WAL directory")?


That sounds reasonable. Having the flag be a string also grants us a bit
more flexibility as the metadata story evolves in the future.

On Wed, Jan 10, 2018 at 11:57 AM, Adar Lieber-Dembo <ad...@cloudera.com>
wrote:

> Thanks for starting a discussion on this topic.
>
> > There've been a few discussions recently regarding changes to Kudu's
> > metadata storage. There are a number of areas that could benefit from
> > improving this layer, and I've been coalescing some of these ideas to
> lock
> > down what changes make sense in the near future. Here's a list of a few
> > considerations:
> >
> > https://docs.google.com/document/d/1jXFqIZvLwkkmSjLC0wy-
> mDA0l0GmVuZJUMUB_7q5QUE/edit?usp=sharing
>
> I left a few comments in the gdoc.
>
> > Empirically, scalability is not as big of an adoption-bottleneck as it
> was
> > before. Additionally, it's not clear that the listed scalability issues
> are
> > the biggest bottlenecks to larger data volumes. Moving forward, we should
> > keep track of user stories that would benefit from such improvements.
>
> I don't really view scalability as an adoption bottleneck; I think
> it's more of a barrier to growing existing clusters. As such, I think
> it's important that we constantly iterate on scalability, so that
> every release is a little bit more scalable than the one before. That
> way Kudu users are assured that their deployments can keep growing as
> Kudu continues to mature.
>
> As far as specific bottlenecks are concerned, the spreading of LBM
> metadata across multiple files is a main contributor to our long
> startup times, and that's one of the biggest scalability bottlenecks
> AFAIK.
>
> > With these points in mind, it seems the reasonable path forward is go
> with
> > 1. and introduce a flag for users to colocate WALs and metadata.
>
> Makes sense. Will colocation be enabled by default for new clusters?
> How about using the flag to define an explicit metadata directory,
> with the default being empty ("use the WAL directory")? That'd make it
> similar to fs_data_dirs, which is nice. If the metadata directory
> can't be found in directory specified by this gflag (or in the WAL
> directory, if blank), we can fall back to looking in the first data
> directory.
>

Re: Kudu: potential metadata changes

[Adar Lieber-Dembo <ad...@cloudera.com>]

Thanks for starting a discussion on this topic.

> There've been a few discussions recently regarding changes to Kudu's
> metadata storage. There are a number of areas that could benefit from
> improving this layer, and I've been coalescing some of these ideas to lock
> down what changes make sense in the near future. Here's a list of a few
> considerations:
>
> https://docs.google.com/document/d/1jXFqIZvLwkkmSjLC0wy-mDA0l0GmVuZJUMUB_7q5QUE/edit?usp=sharing

I left a few comments in the gdoc.

> Empirically, scalability is not as big of an adoption-bottleneck as it was
> before. Additionally, it's not clear that the listed scalability issues are
> the biggest bottlenecks to larger data volumes. Moving forward, we should
> keep track of user stories that would benefit from such improvements.

I don't really view scalability as an adoption bottleneck; I think
it's more of a barrier to growing existing clusters. As such, I think
it's important that we constantly iterate on scalability, so that
every release is a little bit more scalable than the one before. That
way Kudu users are assured that their deployments can keep growing as
Kudu continues to mature.

As far as specific bottlenecks are concerned, the spreading of LBM
metadata across multiple files is a main contributor to our long
startup times, and that's one of the biggest scalability bottlenecks
AFAIK.

> With these points in mind, it seems the reasonable path forward is go with
> 1. and introduce a flag for users to colocate WALs and metadata.

Makes sense. Will colocation be enabled by default for new clusters?
How about using the flag to define an explicit metadata directory,
with the default being empty ("use the WAL directory")? That'd make it
similar to fs_data_dirs, which is nice. If the metadata directory
can't be found in directory specified by this gflag (or in the WAL
directory, if blank), we can fall back to looking in the first data
directory.