zookeeper max no of children of znode

[Shushant Arora <sh...@gmail.com>]

Hi

Is there any max number of children limit of a znode?
As there is a limit of 1MB data of a znode.

can sum of data of  all children of znode be greater than 1MB?

Does in zookeeper noodes which are behind the leader not accept the
connection til they come in sync? If yes and only those nodes which are in
sync with leader accept the connection then How Zookeeper is eventual
consistent, it should be fully consistent then since client will never get
connected to unsync nodes.

Re: zookeeper max no of children of znode

[Ivan Kelly <iv...@apache.org>]

Responses inline

On Sat, Jun 27, 2015 at 4:12 PM Shushant Arora <sh...@gmail.com>
wrote:

> when client reconnects to another node , watches get lost ? Or watches
> stays and in between update notification is lost but client will get next
> update without adding watch again.
>
When a client reconnects, watches are lost. They're not lost silently
though. The watcher will received a disconnected, and once reconnected, a
connected event, which allows the client to check if any changes occurred.

I just wanted to understand how hbase and kafka uses zookeeper reliably
> when it does not have strict consistency ,, Multiple clients can have
> different states of same node at a time.
>
The important thing to note is that zookeeper locks are advisory. Just
because zookeeper has told a client it has a lock on a resource, doesn't
mean that by the time the message with this information reaches the client
in question, that it still has the lock.

Locks have to be enforced by the underlying storage.

In the case of hbase master, the master region writes to a WAL, which is a
HDFS appendable file. The region server does an fsync on the file (which
incidently doesn't guarantee it hits disk) after any write. This fsync
involves a message to the HDFS namenode which guarantees that the WAL only
has one writer at any time. While clients may have an incorrect view of who
is the real hbase master, any attempt to write to the false master will
result in an error which tells them to check their assumptions. In a
non-broken system, they will eventually be informed of the correct master,
and thus be able to continue making progress.

For kafka it's a little more complex because they have a concept of ISR,
and if the required number of acknowledgements for the writes is less that
half the ISR + 1, then indeed writes can be lost. But assuming that the
required number of acknowledgements for a write is greater that (ISR/2) +
1, then no write will be able to succeed without hitting a majority of the
nodes in the ISR. Therefore if there are conflicting writes to a certain
sequence, then the conflicting write should overlap on at least one node,
and thus the system can resolve the conflict like that. I'm sketchy on the
details.

In summary, it's fine for the clients to be out of sync to a degree, since
it's unavoidable until we have some sort of quantum entanglement star trek
computers. But they shouldn't be able to do any damage with out of date
information, and this is why the storage medium needs to be aware of locks.

In Hbase Master selection process, how does a node is 100 % sure that a
> master is created ? Does it has to create the /master node and that node
> already exists will thow node exists excpetion .  Since only by reading (ls
> /) . It may get stale data and gets node does not exists.but in actual
> /master was present.
>
Actually I possibly went into too much detail above.  Node exists
exceptions like this should be expected.

Does these applications read everything on reconnect of session.
>
Nope. It they don't read anything unless the client explicitly asks for it.

And also Is zookeeper not fit for high writes? since write goes via leader
> node then how kafka maintains offsets using zookeeper update operation very
> efficiently.
>
I think zk can do about 10k writes per second. For most kafka usecases this
should be ok. If there are a lot of subscribers you may feel a squeeze
though. Best to ask the Kafka folks about this directly. I'm sure they've
hit across the problem in LinkedIn.

-Ivan

Re: zookeeper max no of children of znode

[Shushant Arora <sh...@gmail.com>]

when client reconnects to another node , watches get lost ? Or watches
stays and in between update notification is lost but client will get next
update without adding watch again.

I just wanted to understand how hbase and kafka uses zookeeper reliably
when it does not have strict consistency ,, Multiple clients can have
different states of same node at a time.

In Hbase Master selection process, how does a node is 100 % sure that a
master is created ? Does it has to create the /master node and that node
already exists will thow node exists excpetion .  Since only by reading (ls
/) . It may get stale data and gets node does not exists.but in actual
/master was present.

Does these applications read everything on reconnect of session.

And also Is zookeeper not fit for high writes? since write goes via leader
node then how kafka maintains offsets using zookeeper update operation very
efficiently.

On Sat, Jun 27, 2015 at 7:06 PM, Ivan Kelly <iv...@apache.org> wrote:

> I assume you are talking about watchers here, because there is no guarantee
> that if a client continually reads a node that it see each and every
> update.
>
> In the case of watchers, you are only guaranteed to see any update from the
> time you apply the watcher to the time you loose the connection to the
> zookeeper session. When you loose the connection, on reconnection you
> should do a read (getData or getChildren) to read any changes that may have
> occurred while you were disconnected.
>
> What design do you have in mind that you want to implement?
>
> -Ivan
>
> On Sat, Jun 27, 2015 at 1:39 PM Shushant Arora <sh...@gmail.com>
> wrote:
>
> > Say a client was connected to follower F1, And F1 was few seconds behind
> > the leader. Then its connection got lost
> > and session is transformed to follwer F2 which is in full sync with
> Leader.
> > Then client will loose some updates ?
> >
> > Since now client will see latest state . All state changes between F1 lag
> > to complete sync (F2)  are lost ?
> >
> > On Sat, Jun 27, 2015 at 4:17 PM, Ivan Kelly <iv...@apache.org> wrote:
> >
> > > Hi Shushant
> > >
> > > Is there any max number of children limit of a znode?
> > > > As there is a limit of 1MB data of a znode.
> > > >
> > > There's no explicit limit to the number of children. However, there is
> a
> > > limit to the size of the packet that can be sent back from the server
> in
> > > response to getChildren which effectly ends up being the limit. This is
> > > controls with a system property, jute.maxbuffer and defaults to 4MB (I
> > > think you can get a around 200,000 znodes into this, though it also
> > depends
> > > on the length of the znode name).
> > >
> > >
> > > > Does in zookeeper noodes which are behind the leader not accept the
> > > > connection til they come in sync? If yes and only those nodes which
> are
> > > in
> > > > sync with leader accept the connection then How Zookeeper is eventual
> > > > consistent, it should be fully consistent then since client will
> never
> > > get
> > > > connected to unsync nodes.
> > > >
> > > Once they have joined the quorum, all zookeeper nodes will accept
> client
> > > connections. Followers are not guaranteed to be 100% in sync with the
> > > leader, due to physics (messages take time to travel over the network).
> > > However, a follower does guarantee that all clients connected will see
> > all
> > > updates in the same sequence that they occur on the server, and they
> > should
> > > only be a number of milliseconds behind. This isn't eventual
> consistency.
> > > Thats what databases like cassandra do where all clients will
> eventually
> > be
> > > able to see all updates though the order it sees them can change.
> > > ZooKeeper provides sequential consistency. However it only provides
> this
> > > consistency within a single client session.
> > >
> > > -Ivan
> > >
> >
>

Re: zookeeper max no of children of znode

[Ivan Kelly <iv...@apache.org>]

I assume you are talking about watchers here, because there is no guarantee
that if a client continually reads a node that it see each and every update.

In the case of watchers, you are only guaranteed to see any update from the
time you apply the watcher to the time you loose the connection to the
zookeeper session. When you loose the connection, on reconnection you
should do a read (getData or getChildren) to read any changes that may have
occurred while you were disconnected.

What design do you have in mind that you want to implement?

-Ivan

On Sat, Jun 27, 2015 at 1:39 PM Shushant Arora <sh...@gmail.com>
wrote:

> Say a client was connected to follower F1, And F1 was few seconds behind
> the leader. Then its connection got lost
> and session is transformed to follwer F2 which is in full sync with Leader.
> Then client will loose some updates ?
>
> Since now client will see latest state . All state changes between F1 lag
> to complete sync (F2)  are lost ?
>
> On Sat, Jun 27, 2015 at 4:17 PM, Ivan Kelly <iv...@apache.org> wrote:
>
> > Hi Shushant
> >
> > Is there any max number of children limit of a znode?
> > > As there is a limit of 1MB data of a znode.
> > >
> > There's no explicit limit to the number of children. However, there is a
> > limit to the size of the packet that can be sent back from the server in
> > response to getChildren which effectly ends up being the limit. This is
> > controls with a system property, jute.maxbuffer and defaults to 4MB (I
> > think you can get a around 200,000 znodes into this, though it also
> depends
> > on the length of the znode name).
> >
> >
> > > Does in zookeeper noodes which are behind the leader not accept the
> > > connection til they come in sync? If yes and only those nodes which are
> > in
> > > sync with leader accept the connection then How Zookeeper is eventual
> > > consistent, it should be fully consistent then since client will never
> > get
> > > connected to unsync nodes.
> > >
> > Once they have joined the quorum, all zookeeper nodes will accept client
> > connections. Followers are not guaranteed to be 100% in sync with the
> > leader, due to physics (messages take time to travel over the network).
> > However, a follower does guarantee that all clients connected will see
> all
> > updates in the same sequence that they occur on the server, and they
> should
> > only be a number of milliseconds behind. This isn't eventual consistency.
> > Thats what databases like cassandra do where all clients will eventually
> be
> > able to see all updates though the order it sees them can change.
> > ZooKeeper provides sequential consistency. However it only provides this
> > consistency within a single client session.
> >
> > -Ivan
> >
>

Re: zookeeper max no of children of znode

[Shushant Arora <sh...@gmail.com>]

Say a client was connected to follower F1, And F1 was few seconds behind
the leader. Then its connection got lost
and session is transformed to follwer F2 which is in full sync with Leader.
Then client will loose some updates ?

Since now client will see latest state . All state changes between F1 lag
to complete sync (F2)  are lost ?

On Sat, Jun 27, 2015 at 4:17 PM, Ivan Kelly <iv...@apache.org> wrote:

> Hi Shushant
>
> Is there any max number of children limit of a znode?
> > As there is a limit of 1MB data of a znode.
> >
> There's no explicit limit to the number of children. However, there is a
> limit to the size of the packet that can be sent back from the server in
> response to getChildren which effectly ends up being the limit. This is
> controls with a system property, jute.maxbuffer and defaults to 4MB (I
> think you can get a around 200,000 znodes into this, though it also depends
> on the length of the znode name).
>
>
> > Does in zookeeper noodes which are behind the leader not accept the
> > connection til they come in sync? If yes and only those nodes which are
> in
> > sync with leader accept the connection then How Zookeeper is eventual
> > consistent, it should be fully consistent then since client will never
> get
> > connected to unsync nodes.
> >
> Once they have joined the quorum, all zookeeper nodes will accept client
> connections. Followers are not guaranteed to be 100% in sync with the
> leader, due to physics (messages take time to travel over the network).
> However, a follower does guarantee that all clients connected will see all
> updates in the same sequence that they occur on the server, and they should
> only be a number of milliseconds behind. This isn't eventual consistency.
> Thats what databases like cassandra do where all clients will eventually be
> able to see all updates though the order it sees them can change.
> ZooKeeper provides sequential consistency. However it only provides this
> consistency within a single client session.
>
> -Ivan
>

Re: zookeeper max no of children of znode

[Ivan Kelly <iv...@apache.org>]

Hi Shushant

Is there any max number of children limit of a znode?
> As there is a limit of 1MB data of a znode.
>
There's no explicit limit to the number of children. However, there is a
limit to the size of the packet that can be sent back from the server in
response to getChildren which effectly ends up being the limit. This is
controls with a system property, jute.maxbuffer and defaults to 4MB (I
think you can get a around 200,000 znodes into this, though it also depends
on the length of the znode name).


> Does in zookeeper noodes which are behind the leader not accept the
> connection til they come in sync? If yes and only those nodes which are in
> sync with leader accept the connection then How Zookeeper is eventual
> consistent, it should be fully consistent then since client will never get
> connected to unsync nodes.
>
Once they have joined the quorum, all zookeeper nodes will accept client
connections. Followers are not guaranteed to be 100% in sync with the
leader, due to physics (messages take time to travel over the network).
However, a follower does guarantee that all clients connected will see all
updates in the same sequence that they occur on the server, and they should
only be a number of milliseconds behind. This isn't eventual consistency.
Thats what databases like cassandra do where all clients will eventually be
able to see all updates though the order it sees them can change.
ZooKeeper provides sequential consistency. However it only provides this
consistency within a single client session.

-Ivan