You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Zod <zo...@gmail.com> on 2018/02/28 07:26:51 UTC
Zookeeper general log or “monitor” command
I want see all commands issued to my Zookeeper cluster, something like
general log in MySQL or "monitor" command in Redis. How can this be done?
The intent is to see how Storm uses Zookeeper (for state management &
ack-ing) and a good way to would be to run a sample topology and see all
the commands issued to Zookeeper by Storm.
I tried enabling debug log for Zookeeper, but that is insufficient has a
lot of noise. For example, issue a create /node prints
2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
-
zodvik
Re: Zookeeper general log or “monitor” command
Posted by Nikhil Bafna <zo...@gmail.com>.
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
Yes, that would work.
You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
Checked the PR. IMO, the actual proposed change there doesn't solve the
intent of the mentioned problem (though it solves mine) and have left a
comment there with my viewpoint.
-
zodvik
On Fri, Mar 9, 2018 at 12:36 AM, Abraham Fine <af...@apache.org> wrote:
> You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
>
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
>
> Thanks,
> Abe
>
> On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> > Cool, monitoring the transaction makes much sense I think.
> >
> > Interesting to see this tool, I didn't know about it yet. For some reason
> > it replicates most of the logic in FileSnap and FileTxnLog classes, but
> at
> > the same time it uses ZooKeeper as a dependency.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
> >
> > > As you mentioned, enabling trace/debug might impact performance.
> > >
> > >
> > > Here's the approach I'm using now.
> > >
> > > -
> > >
> > > Parse the Zookeeper transaction logs. The best repo I've found to do
> > > this is https://github.com/alenca/zklogtool. It's feature packed.
> The
> > > only downside is that it won't record the read queries.
> > >
> > >
> > > -
> > > zodvik
> > >
> > > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > Have you checked TRACE logging of client requests? (
> > > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > > html#sc_authOptions
> > > > )
> > > > Not sure how much different is from what you refer as 'debug logs',
> but
> > > it
> > > > might worth a try. It generates log messages like:
> > > >
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> > > >
> > > > Which looks like quite easy to parse and contains all required
> > > information.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> > > >
> > > > > Unfortunately, parsing debug log is no-go since it doesn't print
> the
> > > > > complete command.
> > > > >
> > > > > I've 2 approached in mind as of now
> > > > >
> > > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > > "tailing".
> > > > > 2. Use ngrep on my local to monitor the network traffic to remote
> ZK
> > > > host &
> > > > > port 2181
> > > > >
> > > > > Will get back with what works better.
> > > > >
> > > > > -
> > > > > zodvik
> > > > >
> > > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Zod,
> > > > > >
> > > > > > I think there's no such thing in ZooKeeper currently other than
> debug
> > > > > logs.
> > > > > > You might want to attach some processing engine to the debug log
> feed
> > > > and
> > > > > > parse and output data which is important for you into MySQL or
> Redis
> > > > > > database.
> > > > > >
> > > > > > The downside of this approach is debug logging has significant
> > > > > performance
> > > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > > analysis.
> > > > > >
> > > > > > Regards,
> > > > > > Andor
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > > > >
> > > > > > > I want see all commands issued to my Zookeeper cluster,
> something
> > > > like
> > > > > > > general log in MySQL or "monitor" command in Redis. How can
> this be
> > > > > done?
> > > > > > >
> > > > > > > The intent is to see how Storm uses Zookeeper (for state
> > > management &
> > > > > > > ack-ing) and a good way to would be to run a sample topology
> and
> > > see
> > > > > all
> > > > > > > the commands issued to Zookeeper by Storm.
> > > > > > >
> > > > > > > I tried enabling debug log for Zookeeper, but that is
> insufficient
> > > > has
> > > > > a
> > > > > > > lot of noise. For example, issue a create /node prints
> > > > > > >
> > > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > > -
> > > > > > > zodvik
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: Zookeeper general log or “monitor” command
Posted by Abraham Fine <af...@apache.org>.
You may also want to consider taking a look and reviewing the change proposed here: https://github.com/apache/zookeeper/pull/307
If I understand correctly, a slow operation log with the threshold for logging set to 0 should accomplish what you are trying to accomplish.
Thanks,
Abe
On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> Cool, monitoring the transaction makes much sense I think.
>
> Interesting to see this tool, I didn't know about it yet. For some reason
> it replicates most of the logic in FileSnap and FileTxnLog classes, but at
> the same time it uses ZooKeeper as a dependency.
>
> Regards,
> Andor
>
>
>
> On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
>
> > As you mentioned, enabling trace/debug might impact performance.
> >
> >
> > Here's the approach I'm using now.
> >
> > -
> >
> > Parse the Zookeeper transaction logs. The best repo I've found to do
> > this is https://github.com/alenca/zklogtool. It's feature packed. The
> > only downside is that it won't record the read queries.
> >
> >
> > -
> > zodvik
> >
> > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
> >
> > > Hi Zod,
> > >
> > > Have you checked TRACE logging of client requests? (
> > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > html#sc_authOptions
> > > )
> > > Not sure how much different is from what you refer as 'debug logs', but
> > it
> > > might worth a try. It generates log messages like:
> > >
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> > >
> > > Which looks like quite easy to parse and contains all required
> > information.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> > >
> > > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > > complete command.
> > > >
> > > > I've 2 approached in mind as of now
> > > >
> > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > "tailing".
> > > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > > host &
> > > > port 2181
> > > >
> > > > Will get back with what works better.
> > > >
> > > > -
> > > > zodvik
> > > >
> > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> > wrote:
> > > >
> > > > > Hi Zod,
> > > > >
> > > > > I think there's no such thing in ZooKeeper currently other than debug
> > > > logs.
> > > > > You might want to attach some processing engine to the debug log feed
> > > and
> > > > > parse and output data which is important for you into MySQL or Redis
> > > > > database.
> > > > >
> > > > > The downside of this approach is debug logging has significant
> > > > performance
> > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > analysis.
> > > > >
> > > > > Regards,
> > > > > Andor
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > > >
> > > > > > I want see all commands issued to my Zookeeper cluster, something
> > > like
> > > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > > done?
> > > > > >
> > > > > > The intent is to see how Storm uses Zookeeper (for state
> > management &
> > > > > > ack-ing) and a good way to would be to run a sample topology and
> > see
> > > > all
> > > > > > the commands issued to Zookeeper by Storm.
> > > > > >
> > > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > > has
> > > > a
> > > > > > lot of noise. For example, issue a create /node prints
> > > > > >
> > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > -
> > > > > > zodvik
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Zookeeper general log or “monitor” command
Posted by Andor Molnar <an...@cloudera.com>.
Cool, monitoring the transaction makes much sense I think.
Interesting to see this tool, I didn't know about it yet. For some reason
it replicates most of the logic in FileSnap and FileTxnLog classes, but at
the same time it uses ZooKeeper as a dependency.
Regards,
Andor
On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
> As you mentioned, enabling trace/debug might impact performance.
>
>
> Here's the approach I'm using now.
>
> -
>
> Parse the Zookeeper transaction logs. The best repo I've found to do
> this is https://github.com/alenca/zklogtool. It's feature packed. The
> only downside is that it won't record the read queries.
>
>
> -
> zodvik
>
> On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
>
> > Hi Zod,
> >
> > Have you checked TRACE logging of client requests? (
> > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> html#sc_authOptions
> > )
> > Not sure how much different is from what you refer as 'debug logs', but
> it
> > might worth a try. It generates log messages like:
> >
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> >
> > Which looks like quite easy to parse and contains all required
> information.
> >
> > Regards,
> > Andor
> >
> >
> > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> >
> > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > complete command.
> > >
> > > I've 2 approached in mind as of now
> > >
> > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > "tailing".
> > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > host &
> > > port 2181
> > >
> > > Will get back with what works better.
> > >
> > > -
> > > zodvik
> > >
> > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > I think there's no such thing in ZooKeeper currently other than debug
> > > logs.
> > > > You might want to attach some processing engine to the debug log feed
> > and
> > > > parse and output data which is important for you into MySQL or Redis
> > > > database.
> > > >
> > > > The downside of this approach is debug logging has significant
> > > performance
> > > > impact on the entire ensemble, but it might be acceptable for you
> > > analysis.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > >
> > > > > I want see all commands issued to my Zookeeper cluster, something
> > like
> > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > done?
> > > > >
> > > > > The intent is to see how Storm uses Zookeeper (for state
> management &
> > > > > ack-ing) and a good way to would be to run a sample topology and
> see
> > > all
> > > > > the commands issued to Zookeeper by Storm.
> > > > >
> > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > has
> > > a
> > > > > lot of noise. For example, issue a create /node prints
> > > > >
> > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > -
> > > > > zodvik
> > > > >
> > > >
> > >
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Nikhil Bafna <zo...@gmail.com>.
As you mentioned, enabling trace/debug might impact performance.
Here's the approach I'm using now.
-
Parse the Zookeeper transaction logs. The best repo I've found to do
this is https://github.com/alenca/zklogtool. It's feature packed. The
only downside is that it won't record the read queries.
-
zodvik
On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
> Hi Zod,
>
> Have you checked TRACE logging of client requests? (
> http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions
> )
> Not sure how much different is from what you refer as 'debug logs', but it
> might worth a try. It generates log messages like:
>
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
>
> Which looks like quite easy to parse and contains all required information.
>
> Regards,
> Andor
>
>
> On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
>
> > Unfortunately, parsing debug log is no-go since it doesn't print the
> > complete command.
> >
> > I've 2 approached in mind as of now
> >
> > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > "tailing".
> > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> host &
> > port 2181
> >
> > Will get back with what works better.
> >
> > -
> > zodvik
> >
> > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
> >
> > > Hi Zod,
> > >
> > > I think there's no such thing in ZooKeeper currently other than debug
> > logs.
> > > You might want to attach some processing engine to the debug log feed
> and
> > > parse and output data which is important for you into MySQL or Redis
> > > database.
> > >
> > > The downside of this approach is debug logging has significant
> > performance
> > > impact on the entire ensemble, but it might be acceptable for you
> > analysis.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > >
> > > > I want see all commands issued to my Zookeeper cluster, something
> like
> > > > general log in MySQL or "monitor" command in Redis. How can this be
> > done?
> > > >
> > > > The intent is to see how Storm uses Zookeeper (for state management &
> > > > ack-ing) and a good way to would be to run a sample topology and see
> > all
> > > > the commands issued to Zookeeper by Storm.
> > > >
> > > > I tried enabling debug log for Zookeeper, but that is insufficient
> has
> > a
> > > > lot of noise. For example, issue a create /node prints
> > > >
> > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > -
> > > > zodvik
> > > >
> > >
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Andor Molnar <an...@cloudera.com>.
Hi Zod,
Have you checked TRACE logging of client requests? (
http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions)
Not sure how much different is from what you refer as 'debug logs', but it
might worth a try. It generates log messages like:
TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
zxid:0xfffffffffffffffe txntype:unknown reqpath:/
TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
Which looks like quite easy to parse and contains all required information.
Regards,
Andor
On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> Unfortunately, parsing debug log is no-go since it doesn't print the
> complete command.
>
> I've 2 approached in mind as of now
>
> 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> "tailing".
> 2. Use ngrep on my local to monitor the network traffic to remote ZK host &
> port 2181
>
> Will get back with what works better.
>
> -
> zodvik
>
> On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
>
> > Hi Zod,
> >
> > I think there's no such thing in ZooKeeper currently other than debug
> logs.
> > You might want to attach some processing engine to the debug log feed and
> > parse and output data which is important for you into MySQL or Redis
> > database.
> >
> > The downside of this approach is debug logging has significant
> performance
> > impact on the entire ensemble, but it might be acceptable for you
> analysis.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> >
> > > I want see all commands issued to my Zookeeper cluster, something like
> > > general log in MySQL or "monitor" command in Redis. How can this be
> done?
> > >
> > > The intent is to see how Storm uses Zookeeper (for state management &
> > > ack-ing) and a good way to would be to run a sample topology and see
> all
> > > the commands issued to Zookeeper by Storm.
> > >
> > > I tried enabling debug log for Zookeeper, but that is insufficient has
> a
> > > lot of noise. For example, issue a create /node prints
> > >
> > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > -
> > > zodvik
> > >
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Zod <zo...@gmail.com>.
Unfortunately, parsing debug log is no-go since it doesn't print the
complete command.
I've 2 approached in mind as of now
1. Modify of org.apache.zookeeper.server.LogFormatter to support "tailing".
2. Use ngrep on my local to monitor the network traffic to remote ZK host &
port 2181
Will get back with what works better.
-
zodvik
On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
> Hi Zod,
>
> I think there's no such thing in ZooKeeper currently other than debug logs.
> You might want to attach some processing engine to the debug log feed and
> parse and output data which is important for you into MySQL or Redis
> database.
>
> The downside of this approach is debug logging has significant performance
> impact on the entire ensemble, but it might be acceptable for you analysis.
>
> Regards,
> Andor
>
>
>
> On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
>
> > I want see all commands issued to my Zookeeper cluster, something like
> > general log in MySQL or "monitor" command in Redis. How can this be done?
> >
> > The intent is to see how Storm uses Zookeeper (for state management &
> > ack-ing) and a good way to would be to run a sample topology and see all
> > the commands issued to Zookeeper by Storm.
> >
> > I tried enabling debug log for Zookeeper, but that is insufficient has a
> > lot of noise. For example, issue a create /node prints
> >
> > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > -
> > zodvik
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Andor Molnar <an...@cloudera.com>.
Hi Zod,
I think there's no such thing in ZooKeeper currently other than debug logs.
You might want to attach some processing engine to the debug log feed and
parse and output data which is important for you into MySQL or Redis
database.
The downside of this approach is debug logging has significant performance
impact on the entire ensemble, but it might be acceptable for you analysis.
Regards,
Andor
On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> I want see all commands issued to my Zookeeper cluster, something like
> general log in MySQL or "monitor" command in Redis. How can this be done?
>
> The intent is to see how Storm uses Zookeeper (for state management &
> ack-ing) and a good way to would be to run a sample topology and see all
> the commands issued to Zookeeper by Storm.
>
> I tried enabling debug log for Zookeeper, but that is insufficient has a
> lot of noise. For example, issue a create /node prints
>
> 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> -
> zodvik
>