Support for large number of keys?

[Jan Kotek <di...@kotek.net>]

Hi,

Zookeeper in current form stores all keys in memory and snapshots them 
periodically. I think it limits size of data Zookeeper can store.

I am investigating feasibility of patching ZK to support large number of keys. It 
would use off-heap storage engine with incremental snapshotting. 
I think this way Zookeeper could store around 100 million keys-value pairs without 
negative impact on performance.  

Technically it is feasible (I already done something similar for Hazelcast). My 
question is if someone would actually use this improvement. Current ZK is 
probably just fine for most uses, it only has problem when you put excessive 
amount of data inside. 

So my questions is: Do you use ZK as a database? And do you have problem with long 
crash recovery time?


Thanks,
Jan Kotek

Re: Support for large number of keys?

[Ted Dunning <te...@gmail.com>]

On Tue, May 27, 2014 at 7:56 AM, Jan Kotek <di...@kotek.net> wrote:

> Zookeeper in current form stores all keys in memory and snapshots them
> periodically. I think it limits size of data Zookeeper can store.
>

Yes. It does.


> I am investigating feasibility of patching ZK to support large number of
> keys. It
> would use off-heap storage engine with incremental snapshotting.
> I think this way Zookeeper could store around 100 million keys-value pairs
> without
> negative impact on performance.
>

ZK already has a progressive snapshot in some form.  What I mean by this is
that the engine does not pause during the creation of a snapshot.  The
reason that this is true is that all transactions that are written to the
transaction log are idempotent.  This means that parts of the memory table
that are written later may actually have been updated after the snapshot
started.  This isn't a problem because after a snapshot is reloaded, all
transactions that were received after the start of the snapshot are
replayed.  If an element was written after a transaction updated it,
replaying that transaction will have no effect.

Technically it is feasible (I already done something similar for
> Hazelcast). My
> question is if someone would actually use this improvement. Current ZK is
> probably just fine for most uses, it only has problem when you put
> excessive
> amount of data inside.
>

And that is, of course, outside the mission that ZK *intends* to support.

So my questions is: Do you use ZK as a database? And do you have problem
> with long
> crash recovery time?
>

This is a question for users.

As a user and as somebody who supports users, I don't and don't know about
anybody using ZK as a large data store.

Re: Support for large number of keys?

[Camille Fournier <ca...@apache.org>]

Well, ZK is not designed to be a database. I wouldn't recommend most people
try to use it as a database. There are many good KV store databases out
there that are better suited to the operations one wants to do with a
database and the consistency models and tradeoffs for a database.
I think it would be interesting to see the work, but I would not personally
recommend using ZK as a large datastore in this way.

C

On Tue, May 27, 2014 at 10:56 AM, Jan Kotek <di...@kotek.net> wrote:

> Hi,
>
> Zookeeper in current form stores all keys in memory and snapshots them
> periodically. I think it limits size of data Zookeeper can store.
>
> I am investigating feasibility of patching ZK to support large number of
> keys. It
> would use off-heap storage engine with incremental snapshotting.
> I think this way Zookeeper could store around 100 million keys-value pairs
> without
> negative impact on performance.
>
> Technically it is feasible (I already done something similar for
> Hazelcast). My
> question is if someone would actually use this improvement. Current ZK is
> probably just fine for most uses, it only has problem when you put
> excessive
> amount of data inside.
>
> So my questions is: Do you use ZK as a database? And do you have problem
> with long
> crash recovery time?
>
>
> Thanks,
> Jan Kotek
>
>