Verify zookeeper cluster consistency

["Chaloulos, Klearchos (Nokia - GR/Athens)" <kl...@nokia.com>]

Hello,

I recently ran into a case where my zookeeper servers had inconsistent data, that is nodes that existed in one server did not exist in the other two. I used 3 servers, with version 3.4.9. Unfortunately I cannot reproduce the issue, as it occurred a long time ago in an environment outside my control.

However my question is: Is there a way to detect zookeeper cluster inconsistencies? I am not talking about transient inconsistencies that exist for, say, a few seconds, but for permanent inconsistencies. In my case the inconsistencies persisted for a month!

Best regards,

Klearchos

Re: Verify zookeeper cluster consistency

[Patrick Hunt <ph...@apache.org>]

iirc Ben (or was it someone else from Facebook) mentioned at a meetup about
doing some work in this area. I believe they were hashing the tree but I
may be mis-remembering.... Perhaps one of those folks can shed light.

Patrick

On Fri, Jul 27, 2018 at 2:04 AM Andor Molnar <an...@cloudera.com.invalid>
wrote:

> Hi Klearchos,
>
> Some tool which is constantly querying the znode tree on each of every
> participant and tolerates some latency between the inconsistencies, but
> reports everything above the threshold could do the trick.
>
> Personally I'm not aware of anything existing, but feel free to dig github
> or create your own one.
>
> Regards,
> Andor
>
>
>
>
> On Fri, Jul 27, 2018 at 9:16 AM, Chaloulos, Klearchos (Nokia - GR/Athens) <
> klearchos.chaloulos@nokia.com> wrote:
>
> > Hello,
> >
> > I recently ran into a case where my zookeeper servers had inconsistent
> > data, that is nodes that existed in one server did not exist in the other
> > two. I used 3 servers, with version 3.4.9. Unfortunately I cannot
> reproduce
> > the issue, as it occurred a long time ago in an environment outside my
> > control.
> >
> > However my question is: Is there a way to detect zookeeper cluster
> > inconsistencies? I am not talking about transient inconsistencies that
> > exist for, say, a few seconds, but for permanent inconsistencies. In my
> > case the inconsistencies persisted for a month!
> >
> > Best regards,
> >
> > Klearchos
> >
> >
>

Re: Verify zookeeper cluster consistency

[Andor Molnar <an...@cloudera.com.INVALID>]

Hi Klearchos,

Some tool which is constantly querying the znode tree on each of every
participant and tolerates some latency between the inconsistencies, but
reports everything above the threshold could do the trick.

Personally I'm not aware of anything existing, but feel free to dig github
or create your own one.

Regards,
Andor




On Fri, Jul 27, 2018 at 9:16 AM, Chaloulos, Klearchos (Nokia - GR/Athens) <
klearchos.chaloulos@nokia.com> wrote:

> Hello,
>
> I recently ran into a case where my zookeeper servers had inconsistent
> data, that is nodes that existed in one server did not exist in the other
> two. I used 3 servers, with version 3.4.9. Unfortunately I cannot reproduce
> the issue, as it occurred a long time ago in an environment outside my
> control.
>
> However my question is: Is there a way to detect zookeeper cluster
> inconsistencies? I am not talking about transient inconsistencies that
> exist for, say, a few seconds, but for permanent inconsistencies. In my
> case the inconsistencies persisted for a month!
>
> Best regards,
>
> Klearchos
>
>