You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Zod <zo...@gmail.com> on 2018/03/01 04:33:24 UTC
Re: Zookeeper general log or “monitor” command
Unfortunately, parsing debug log is no-go since it doesn't print the
complete command.
I've 2 approached in mind as of now
1. Modify of org.apache.zookeeper.server.LogFormatter to support "tailing".
2. Use ngrep on my local to monitor the network traffic to remote ZK host &
port 2181
Will get back with what works better.
-
zodvik
On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
> Hi Zod,
>
> I think there's no such thing in ZooKeeper currently other than debug logs.
> You might want to attach some processing engine to the debug log feed and
> parse and output data which is important for you into MySQL or Redis
> database.
>
> The downside of this approach is debug logging has significant performance
> impact on the entire ensemble, but it might be acceptable for you analysis.
>
> Regards,
> Andor
>
>
>
> On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
>
> > I want see all commands issued to my Zookeeper cluster, something like
> > general log in MySQL or "monitor" command in Redis. How can this be done?
> >
> > The intent is to see how Storm uses Zookeeper (for state management &
> > ack-ing) and a good way to would be to run a sample topology and see all
> > the commands issued to Zookeeper by Storm.
> >
> > I tried enabling debug log for Zookeeper, but that is insufficient has a
> > lot of noise. For example, issue a create /node prints
> >
> > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > -
> > zodvik
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Nikhil Bafna <zo...@gmail.com>.
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
Yes, that would work.
You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
Checked the PR. IMO, the actual proposed change there doesn't solve the
intent of the mentioned problem (though it solves mine) and have left a
comment there with my viewpoint.
-
zodvik
On Fri, Mar 9, 2018 at 12:36 AM, Abraham Fine <af...@apache.org> wrote:
> You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
>
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
>
> Thanks,
> Abe
>
> On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> > Cool, monitoring the transaction makes much sense I think.
> >
> > Interesting to see this tool, I didn't know about it yet. For some reason
> > it replicates most of the logic in FileSnap and FileTxnLog classes, but
> at
> > the same time it uses ZooKeeper as a dependency.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
> >
> > > As you mentioned, enabling trace/debug might impact performance.
> > >
> > >
> > > Here's the approach I'm using now.
> > >
> > > -
> > >
> > > Parse the Zookeeper transaction logs. The best repo I've found to do
> > > this is https://github.com/alenca/zklogtool. It's feature packed.
> The
> > > only downside is that it won't record the read queries.
> > >
> > >
> > > -
> > > zodvik
> > >
> > > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > Have you checked TRACE logging of client requests? (
> > > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > > html#sc_authOptions
> > > > )
> > > > Not sure how much different is from what you refer as 'debug logs',
> but
> > > it
> > > > might worth a try. It generates log messages like:
> > > >
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> > > >
> > > > Which looks like quite easy to parse and contains all required
> > > information.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> > > >
> > > > > Unfortunately, parsing debug log is no-go since it doesn't print
> the
> > > > > complete command.
> > > > >
> > > > > I've 2 approached in mind as of now
> > > > >
> > > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > > "tailing".
> > > > > 2. Use ngrep on my local to monitor the network traffic to remote
> ZK
> > > > host &
> > > > > port 2181
> > > > >
> > > > > Will get back with what works better.
> > > > >
> > > > > -
> > > > > zodvik
> > > > >
> > > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Zod,
> > > > > >
> > > > > > I think there's no such thing in ZooKeeper currently other than
> debug
> > > > > logs.
> > > > > > You might want to attach some processing engine to the debug log
> feed
> > > > and
> > > > > > parse and output data which is important for you into MySQL or
> Redis
> > > > > > database.
> > > > > >
> > > > > > The downside of this approach is debug logging has significant
> > > > > performance
> > > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > > analysis.
> > > > > >
> > > > > > Regards,
> > > > > > Andor
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > > > >
> > > > > > > I want see all commands issued to my Zookeeper cluster,
> something
> > > > like
> > > > > > > general log in MySQL or "monitor" command in Redis. How can
> this be
> > > > > done?
> > > > > > >
> > > > > > > The intent is to see how Storm uses Zookeeper (for state
> > > management &
> > > > > > > ack-ing) and a good way to would be to run a sample topology
> and
> > > see
> > > > > all
> > > > > > > the commands issued to Zookeeper by Storm.
> > > > > > >
> > > > > > > I tried enabling debug log for Zookeeper, but that is
> insufficient
> > > > has
> > > > > a
> > > > > > > lot of noise. For example, issue a create /node prints
> > > > > > >
> > > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > > -
> > > > > > > zodvik
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: Zookeeper general log or “monitor” command
Posted by Abraham Fine <af...@apache.org>.
You may also want to consider taking a look and reviewing the change proposed here: https://github.com/apache/zookeeper/pull/307
If I understand correctly, a slow operation log with the threshold for logging set to 0 should accomplish what you are trying to accomplish.
Thanks,
Abe
On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> Cool, monitoring the transaction makes much sense I think.
>
> Interesting to see this tool, I didn't know about it yet. For some reason
> it replicates most of the logic in FileSnap and FileTxnLog classes, but at
> the same time it uses ZooKeeper as a dependency.
>
> Regards,
> Andor
>
>
>
> On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
>
> > As you mentioned, enabling trace/debug might impact performance.
> >
> >
> > Here's the approach I'm using now.
> >
> > -
> >
> > Parse the Zookeeper transaction logs. The best repo I've found to do
> > this is https://github.com/alenca/zklogtool. It's feature packed. The
> > only downside is that it won't record the read queries.
> >
> >
> > -
> > zodvik
> >
> > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
> >
> > > Hi Zod,
> > >
> > > Have you checked TRACE logging of client requests? (
> > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > html#sc_authOptions
> > > )
> > > Not sure how much different is from what you refer as 'debug logs', but
> > it
> > > might worth a try. It generates log messages like:
> > >
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> > >
> > > Which looks like quite easy to parse and contains all required
> > information.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> > >
> > > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > > complete command.
> > > >
> > > > I've 2 approached in mind as of now
> > > >
> > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > "tailing".
> > > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > > host &
> > > > port 2181
> > > >
> > > > Will get back with what works better.
> > > >
> > > > -
> > > > zodvik
> > > >
> > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> > wrote:
> > > >
> > > > > Hi Zod,
> > > > >
> > > > > I think there's no such thing in ZooKeeper currently other than debug
> > > > logs.
> > > > > You might want to attach some processing engine to the debug log feed
> > > and
> > > > > parse and output data which is important for you into MySQL or Redis
> > > > > database.
> > > > >
> > > > > The downside of this approach is debug logging has significant
> > > > performance
> > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > analysis.
> > > > >
> > > > > Regards,
> > > > > Andor
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > > >
> > > > > > I want see all commands issued to my Zookeeper cluster, something
> > > like
> > > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > > done?
> > > > > >
> > > > > > The intent is to see how Storm uses Zookeeper (for state
> > management &
> > > > > > ack-ing) and a good way to would be to run a sample topology and
> > see
> > > > all
> > > > > > the commands issued to Zookeeper by Storm.
> > > > > >
> > > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > > has
> > > > a
> > > > > > lot of noise. For example, issue a create /node prints
> > > > > >
> > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > -
> > > > > > zodvik
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Zookeeper general log or “monitor” command
Posted by Andor Molnar <an...@cloudera.com>.
Cool, monitoring the transaction makes much sense I think.
Interesting to see this tool, I didn't know about it yet. For some reason
it replicates most of the logic in FileSnap and FileTxnLog classes, but at
the same time it uses ZooKeeper as a dependency.
Regards,
Andor
On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zo...@gmail.com> wrote:
> As you mentioned, enabling trace/debug might impact performance.
>
>
> Here's the approach I'm using now.
>
> -
>
> Parse the Zookeeper transaction logs. The best repo I've found to do
> this is https://github.com/alenca/zklogtool. It's feature packed. The
> only downside is that it won't record the read queries.
>
>
> -
> zodvik
>
> On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
>
> > Hi Zod,
> >
> > Have you checked TRACE logging of client requests? (
> > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> html#sc_authOptions
> > )
> > Not sure how much different is from what you refer as 'debug logs', but
> it
> > might worth a try. It generates log messages like:
> >
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> >
> > Which looks like quite easy to parse and contains all required
> information.
> >
> > Regards,
> > Andor
> >
> >
> > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> >
> > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > complete command.
> > >
> > > I've 2 approached in mind as of now
> > >
> > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > "tailing".
> > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > host &
> > > port 2181
> > >
> > > Will get back with what works better.
> > >
> > > -
> > > zodvik
> > >
> > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > I think there's no such thing in ZooKeeper currently other than debug
> > > logs.
> > > > You might want to attach some processing engine to the debug log feed
> > and
> > > > parse and output data which is important for you into MySQL or Redis
> > > > database.
> > > >
> > > > The downside of this approach is debug logging has significant
> > > performance
> > > > impact on the entire ensemble, but it might be acceptable for you
> > > analysis.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > > >
> > > > > I want see all commands issued to my Zookeeper cluster, something
> > like
> > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > done?
> > > > >
> > > > > The intent is to see how Storm uses Zookeeper (for state
> management &
> > > > > ack-ing) and a good way to would be to run a sample topology and
> see
> > > all
> > > > > the commands issued to Zookeeper by Storm.
> > > > >
> > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > has
> > > a
> > > > > lot of noise. For example, issue a create /node prints
> > > > >
> > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > -
> > > > > zodvik
> > > > >
> > > >
> > >
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Nikhil Bafna <zo...@gmail.com>.
As you mentioned, enabling trace/debug might impact performance.
Here's the approach I'm using now.
-
Parse the Zookeeper transaction logs. The best repo I've found to do
this is https://github.com/alenca/zklogtool. It's feature packed. The
only downside is that it won't record the read queries.
-
zodvik
On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <an...@cloudera.com> wrote:
> Hi Zod,
>
> Have you checked TRACE logging of client requests? (
> http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions
> )
> Not sure how much different is from what you refer as 'debug logs', but it
> might worth a try. It generates log messages like:
>
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
>
> Which looks like quite easy to parse and contains all required information.
>
> Regards,
> Andor
>
>
> On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
>
> > Unfortunately, parsing debug log is no-go since it doesn't print the
> > complete command.
> >
> > I've 2 approached in mind as of now
> >
> > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > "tailing".
> > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> host &
> > port 2181
> >
> > Will get back with what works better.
> >
> > -
> > zodvik
> >
> > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
> >
> > > Hi Zod,
> > >
> > > I think there's no such thing in ZooKeeper currently other than debug
> > logs.
> > > You might want to attach some processing engine to the debug log feed
> and
> > > parse and output data which is important for you into MySQL or Redis
> > > database.
> > >
> > > The downside of this approach is debug logging has significant
> > performance
> > > impact on the entire ensemble, but it might be acceptable for you
> > analysis.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> > >
> > > > I want see all commands issued to my Zookeeper cluster, something
> like
> > > > general log in MySQL or "monitor" command in Redis. How can this be
> > done?
> > > >
> > > > The intent is to see how Storm uses Zookeeper (for state management &
> > > > ack-ing) and a good way to would be to run a sample topology and see
> > all
> > > > the commands issued to Zookeeper by Storm.
> > > >
> > > > I tried enabling debug log for Zookeeper, but that is insufficient
> has
> > a
> > > > lot of noise. For example, issue a create /node prints
> > > >
> > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > -
> > > > zodvik
> > > >
> > >
> >
>
Re: Zookeeper general log or “monitor” command
Posted by Andor Molnar <an...@cloudera.com>.
Hi Zod,
Have you checked TRACE logging of client requests? (
http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions)
Not sure how much different is from what you refer as 'debug logs', but it
might worth a try. It generates log messages like:
TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
zxid:0xfffffffffffffffe txntype:unknown reqpath:/
TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
Which looks like quite easy to parse and contains all required information.
Regards,
Andor
On Thu, Mar 1, 2018 at 5:33 AM, Zod <zo...@gmail.com> wrote:
> Unfortunately, parsing debug log is no-go since it doesn't print the
> complete command.
>
> I've 2 approached in mind as of now
>
> 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> "tailing".
> 2. Use ngrep on my local to monitor the network traffic to remote ZK host &
> port 2181
>
> Will get back with what works better.
>
> -
> zodvik
>
> On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <an...@cloudera.com> wrote:
>
> > Hi Zod,
> >
> > I think there's no such thing in ZooKeeper currently other than debug
> logs.
> > You might want to attach some processing engine to the debug log feed and
> > parse and output data which is important for you into MySQL or Redis
> > database.
> >
> > The downside of this approach is debug logging has significant
> performance
> > impact on the entire ensemble, but it might be acceptable for you
> analysis.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zo...@gmail.com> wrote:
> >
> > > I want see all commands issued to my Zookeeper cluster, something like
> > > general log in MySQL or "monitor" command in Redis. How can this be
> done?
> > >
> > > The intent is to see how Storm uses Zookeeper (for state management &
> > > ack-ing) and a good way to would be to run a sample topology and see
> all
> > > the commands issued to Zookeeper by Storm.
> > >
> > > I tried enabling debug log for Zookeeper, but that is insufficient has
> a
> > > lot of noise. For example, issue a create /node prints
> > >
> > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > -
> > > zodvik
> > >
> >
>