You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by kishore g <g....@gmail.com> on 2011/09/14 00:53:29 UTC
Dump all changes on ZK to file
Hi,
I want to dump all changes that happen on ZK to a file.
What I am trying to do is following
- Set up some application specific structure on zookeeper.
- Start a listener or something equivalent that watches on all the nodes.
- Multiple processes are connected to Zookeeper and they make some
changes on the ZK as needed
- The listener will detect the changes and dump the changes to a file in
some structure
- Once the run is over there will be a verification tool which will go
over the change log and do some verifications.
I am not sure if there is already a tool to do this or if it is possible to
get all changes from ZK server logs.
I am aware that if we have a separate listener we may not be able to get all
changes since I might miss changes between receiving a notification and a
change happening before i reset the watch. This may not be the case with ZK
logs/snapshot.
This will not be in production but I want to use it to for testing. Idea is
to decouple verification from simulation so that all types of verification
can be done based on the change log.
Any pointers/suggestions ?
thanks,
Kishore G
Re: Dump all changes on ZK to file
Posted by kishore g <g....@gmail.com>.
Will do.
https://issues.apache.org/jira/browse/ZOOKEEPER-1183
thanks
Kishore G
On Wed, Sep 14, 2011 at 10:37 AM, Patrick Hunt <ph...@apache.org> wrote:
> Consider submitting a patch.
>
> Patrick
>
> On Tue, Sep 13, 2011 at 11:44 PM, kishore g <g....@gmail.com> wrote:
> > Hi Ben,
> >
> > I was able to get more information by changing the LogFormatter code.
> >
> > Record txn = SerializeUtils.deserializeTxn(iab, hdr);
> >
> > Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
> > and was able to get all the information needed.
> >
> > thanks,
> > Kishore G
> >
> > On Tue, Sep 13, 2011 at 5:40 PM, kishore g <g....@gmail.com> wrote:
> >
> >> Hi Ben,
> >>
> >> This only gives me the hugh level information something like
> >>
> >> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
> >> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
> >> createSession
> >> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
> >> setData
> >> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
> >> createSession
> >> ...
> >> ..
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
> >> create
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
> >> setData
> >>
> >> How can get more details about the transaction itself. I want to know
> which
> >> node was created deleted what was set etc. Is this possible ?
> >>
> >> thanks,
> >> Kishore G
> >>
> >>
> >>
> >> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <g....@gmail.com> wrote:
> >>
> >>> Thanks Ben, that's exactly what I was looking for.
> >>>
> >>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <br...@apache.org>
> wrote:
> >>>
> >>>> you can do this using the logs. there is a log format utility:
> >>>> LogFormatter that will print the log in a human readable format that
> >>>> you could use to get this information. (actually you might need to
> >>>> enhance the tool a bit depending on the detail that you want.)
> >>>>
> >>>> ben
> >>>>
> >>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com>
> wrote:
> >>>> > Hi,
> >>>> >
> >>>> > I want to dump all changes that happen on ZK to a file.
> >>>> >
> >>>> > What I am trying to do is following
> >>>> >
> >>>> > - Set up some application specific structure on zookeeper.
> >>>> > - Start a listener or something equivalent that watches on all the
> >>>> nodes.
> >>>> > - Multiple processes are connected to Zookeeper and they make some
> >>>> > changes on the ZK as needed
> >>>> > - The listener will detect the changes and dump the changes to a
> file
> >>>> in
> >>>> > some structure
> >>>> > - Once the run is over there will be a verification tool which
> will
> >>>> go
> >>>> > over the change log and do some verifications.
> >>>> >
> >>>> > I am not sure if there is already a tool to do this or if it is
> >>>> possible to
> >>>> > get all changes from ZK server logs.
> >>>> >
> >>>> > I am aware that if we have a separate listener we may not be able to
> >>>> get all
> >>>> > changes since I might miss changes between receiving a notification
> and
> >>>> a
> >>>> > change happening before i reset the watch. This may not be the case
> >>>> with ZK
> >>>> > logs/snapshot.
> >>>> >
> >>>> > This will not be in production but I want to use it to for testing.
> >>>> Idea is
> >>>> > to decouple verification from simulation so that all types of
> >>>> verification
> >>>> > can be done based on the change log.
> >>>> >
> >>>> > Any pointers/suggestions ?
> >>>> >
> >>>> > thanks,
> >>>> > Kishore G
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>
Re: Dump all changes on ZK to file
Posted by Patrick Hunt <ph...@apache.org>.
Consider submitting a patch.
Patrick
On Tue, Sep 13, 2011 at 11:44 PM, kishore g <g....@gmail.com> wrote:
> Hi Ben,
>
> I was able to get more information by changing the LogFormatter code.
>
> Record txn = SerializeUtils.deserializeTxn(iab, hdr);
>
> Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
> and was able to get all the information needed.
>
> thanks,
> Kishore G
>
> On Tue, Sep 13, 2011 at 5:40 PM, kishore g <g....@gmail.com> wrote:
>
>> Hi Ben,
>>
>> This only gives me the hugh level information something like
>>
>> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
>> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
>> createSession
>> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
>> setData
>> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
>> createSession
>> ...
>> ..
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
>> create
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
>> setData
>>
>> How can get more details about the transaction itself. I want to know which
>> node was created deleted what was set etc. Is this possible ?
>>
>> thanks,
>> Kishore G
>>
>>
>>
>> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <g....@gmail.com> wrote:
>>
>>> Thanks Ben, that's exactly what I was looking for.
>>>
>>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <br...@apache.org> wrote:
>>>
>>>> you can do this using the logs. there is a log format utility:
>>>> LogFormatter that will print the log in a human readable format that
>>>> you could use to get this information. (actually you might need to
>>>> enhance the tool a bit depending on the detail that you want.)
>>>>
>>>> ben
>>>>
>>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > I want to dump all changes that happen on ZK to a file.
>>>> >
>>>> > What I am trying to do is following
>>>> >
>>>> > - Set up some application specific structure on zookeeper.
>>>> > - Start a listener or something equivalent that watches on all the
>>>> nodes.
>>>> > - Multiple processes are connected to Zookeeper and they make some
>>>> > changes on the ZK as needed
>>>> > - The listener will detect the changes and dump the changes to a file
>>>> in
>>>> > some structure
>>>> > - Once the run is over there will be a verification tool which will
>>>> go
>>>> > over the change log and do some verifications.
>>>> >
>>>> > I am not sure if there is already a tool to do this or if it is
>>>> possible to
>>>> > get all changes from ZK server logs.
>>>> >
>>>> > I am aware that if we have a separate listener we may not be able to
>>>> get all
>>>> > changes since I might miss changes between receiving a notification and
>>>> a
>>>> > change happening before i reset the watch. This may not be the case
>>>> with ZK
>>>> > logs/snapshot.
>>>> >
>>>> > This will not be in production but I want to use it to for testing.
>>>> Idea is
>>>> > to decouple verification from simulation so that all types of
>>>> verification
>>>> > can be done based on the change log.
>>>> >
>>>> > Any pointers/suggestions ?
>>>> >
>>>> > thanks,
>>>> > Kishore G
>>>> >
>>>>
>>>
>>>
>>
>
Re: Dump all changes on ZK to file
Posted by kishore g <g....@gmail.com>.
Hi Ben,
I was able to get more information by changing the LogFormatter code.
Record txn = SerializeUtils.deserializeTxn(iab, hdr);
Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
and was able to get all the information needed.
thanks,
Kishore G
On Tue, Sep 13, 2011 at 5:40 PM, kishore g <g....@gmail.com> wrote:
> Hi Ben,
>
> This only gives me the hugh level information something like
>
> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
> createSession
> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
> setData
> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
> createSession
> ...
> ..
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
> create
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
> setData
>
> How can get more details about the transaction itself. I want to know which
> node was created deleted what was set etc. Is this possible ?
>
> thanks,
> Kishore G
>
>
>
> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <g....@gmail.com> wrote:
>
>> Thanks Ben, that's exactly what I was looking for.
>>
>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <br...@apache.org> wrote:
>>
>>> you can do this using the logs. there is a log format utility:
>>> LogFormatter that will print the log in a human readable format that
>>> you could use to get this information. (actually you might need to
>>> enhance the tool a bit depending on the detail that you want.)
>>>
>>> ben
>>>
>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I want to dump all changes that happen on ZK to a file.
>>> >
>>> > What I am trying to do is following
>>> >
>>> > - Set up some application specific structure on zookeeper.
>>> > - Start a listener or something equivalent that watches on all the
>>> nodes.
>>> > - Multiple processes are connected to Zookeeper and they make some
>>> > changes on the ZK as needed
>>> > - The listener will detect the changes and dump the changes to a file
>>> in
>>> > some structure
>>> > - Once the run is over there will be a verification tool which will
>>> go
>>> > over the change log and do some verifications.
>>> >
>>> > I am not sure if there is already a tool to do this or if it is
>>> possible to
>>> > get all changes from ZK server logs.
>>> >
>>> > I am aware that if we have a separate listener we may not be able to
>>> get all
>>> > changes since I might miss changes between receiving a notification and
>>> a
>>> > change happening before i reset the watch. This may not be the case
>>> with ZK
>>> > logs/snapshot.
>>> >
>>> > This will not be in production but I want to use it to for testing.
>>> Idea is
>>> > to decouple verification from simulation so that all types of
>>> verification
>>> > can be done based on the change log.
>>> >
>>> > Any pointers/suggestions ?
>>> >
>>> > thanks,
>>> > Kishore G
>>> >
>>>
>>
>>
>
Re: Dump all changes on ZK to file
Posted by kishore g <g....@gmail.com>.
Hi Ben,
This only gives me the hugh level information something like
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
createSession
8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
setData
8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
createSession
...
..
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c create
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
setData
How can get more details about the transaction itself. I want to know which
node was created deleted what was set etc. Is this possible ?
thanks,
Kishore G
On Tue, Sep 13, 2011 at 4:47 PM, kishore g <g....@gmail.com> wrote:
> Thanks Ben, that's exactly what I was looking for.
>
> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <br...@apache.org> wrote:
>
>> you can do this using the logs. there is a log format utility:
>> LogFormatter that will print the log in a human readable format that
>> you could use to get this information. (actually you might need to
>> enhance the tool a bit depending on the detail that you want.)
>>
>> ben
>>
>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com> wrote:
>> > Hi,
>> >
>> > I want to dump all changes that happen on ZK to a file.
>> >
>> > What I am trying to do is following
>> >
>> > - Set up some application specific structure on zookeeper.
>> > - Start a listener or something equivalent that watches on all the
>> nodes.
>> > - Multiple processes are connected to Zookeeper and they make some
>> > changes on the ZK as needed
>> > - The listener will detect the changes and dump the changes to a file
>> in
>> > some structure
>> > - Once the run is over there will be a verification tool which will go
>> > over the change log and do some verifications.
>> >
>> > I am not sure if there is already a tool to do this or if it is possible
>> to
>> > get all changes from ZK server logs.
>> >
>> > I am aware that if we have a separate listener we may not be able to get
>> all
>> > changes since I might miss changes between receiving a notification and
>> a
>> > change happening before i reset the watch. This may not be the case with
>> ZK
>> > logs/snapshot.
>> >
>> > This will not be in production but I want to use it to for testing. Idea
>> is
>> > to decouple verification from simulation so that all types of
>> verification
>> > can be done based on the change log.
>> >
>> > Any pointers/suggestions ?
>> >
>> > thanks,
>> > Kishore G
>> >
>>
>
>
Re: Dump all changes on ZK to file
Posted by kishore g <g....@gmail.com>.
Thanks Ben, that's exactly what I was looking for.
On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <br...@apache.org> wrote:
> you can do this using the logs. there is a log format utility:
> LogFormatter that will print the log in a human readable format that
> you could use to get this information. (actually you might need to
> enhance the tool a bit depending on the detail that you want.)
>
> ben
>
> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com> wrote:
> > Hi,
> >
> > I want to dump all changes that happen on ZK to a file.
> >
> > What I am trying to do is following
> >
> > - Set up some application specific structure on zookeeper.
> > - Start a listener or something equivalent that watches on all the
> nodes.
> > - Multiple processes are connected to Zookeeper and they make some
> > changes on the ZK as needed
> > - The listener will detect the changes and dump the changes to a file
> in
> > some structure
> > - Once the run is over there will be a verification tool which will go
> > over the change log and do some verifications.
> >
> > I am not sure if there is already a tool to do this or if it is possible
> to
> > get all changes from ZK server logs.
> >
> > I am aware that if we have a separate listener we may not be able to get
> all
> > changes since I might miss changes between receiving a notification and a
> > change happening before i reset the watch. This may not be the case with
> ZK
> > logs/snapshot.
> >
> > This will not be in production but I want to use it to for testing. Idea
> is
> > to decouple verification from simulation so that all types of
> verification
> > can be done based on the change log.
> >
> > Any pointers/suggestions ?
> >
> > thanks,
> > Kishore G
> >
>
Re: Dump all changes on ZK to file
Posted by Benjamin Reed <br...@apache.org>.
you can do this using the logs. there is a log format utility:
LogFormatter that will print the log in a human readable format that
you could use to get this information. (actually you might need to
enhance the tool a bit depending on the detail that you want.)
ben
On Tue, Sep 13, 2011 at 3:53 PM, kishore g <g....@gmail.com> wrote:
> Hi,
>
> I want to dump all changes that happen on ZK to a file.
>
> What I am trying to do is following
>
> - Set up some application specific structure on zookeeper.
> - Start a listener or something equivalent that watches on all the nodes.
> - Multiple processes are connected to Zookeeper and they make some
> changes on the ZK as needed
> - The listener will detect the changes and dump the changes to a file in
> some structure
> - Once the run is over there will be a verification tool which will go
> over the change log and do some verifications.
>
> I am not sure if there is already a tool to do this or if it is possible to
> get all changes from ZK server logs.
>
> I am aware that if we have a separate listener we may not be able to get all
> changes since I might miss changes between receiving a notification and a
> change happening before i reset the watch. This may not be the case with ZK
> logs/snapshot.
>
> This will not be in production but I want to use it to for testing. Idea is
> to decouple verification from simulation so that all types of verification
> can be done based on the change log.
>
> Any pointers/suggestions ?
>
> thanks,
> Kishore G
>