upper bound on znode entries?

[Bill <bi...@dehora.net>]

Hi,

[I had a quick look around and couldn't find any guidance on this]

I was wondering how Zookeeper behaves as the number of entries under a 
znode increases. Specifically I'm looking at a design that tracks which 
servers in a cluster have loaded which search index files into their 
memory, and this would involve a znode folder that keeps an 
index->server mapping as indexes come and go. I have a sense that's 
going to degrade as the number of loaded indexes increases but was 
wondering what typically is considered a sane upper bound of entries 
under a znode (ie is it hundreds, thousands, hundreds of zillions) or is 
there some other naturally limiting factor?

Bill

Re: upper bound on znode entries?

[Ted Dunning <te...@gmail.com>]

There is a bound based on maximum response size.  This typically kicks in
somewhere above 100,000 entries in a directory.

If you stay in the tens of thousands or below and have reasonable znode name
lengths, you should be fine.

Degradation with designs like the one you propose typically comes from
causing broadcast storms of ever increasing object sizes.  Thus, you need to
be cautious about having all clients watch a directory that all clients
register in.

On Fri, Sep 2, 2011 at 9:54 AM, Bill <bi...@dehora.net> wrote:

> Hi,
>
> [I had a quick look around and couldn't find any guidance on this]
>
> I was wondering how Zookeeper behaves as the number of entries under a
> znode increases. Specifically I'm looking at a design that tracks which
> servers in a cluster have loaded which search index files into their memory,
> and this would involve a znode folder that keeps an index->server mapping as
> indexes come and go. I have a sense that's going to degrade as the number of
> loaded indexes increases but was wondering what typically is considered a
> sane upper bound of entries under a znode (ie is it hundreds, thousands,
> hundreds of zillions) or is there some other naturally limiting factor?
>
> Bill
>