You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Ivan Kelly <iv...@yahoo-inc.com> on 2010/03/29 19:07:49 UTC

Solitication for logging/debugging requirements

Hi,

Im going to be using Zookeeper quite extensively for a project in a  
few weeks, but development hasn't kicked off yet. This means I have  
some time on my hands and I'd like to get familiar with zookeeper  
beforehand by perhaps writing some tools to make debugging problems  
with it easier so as to save myself some time in the future. Problem  
is I haven't had to debug many zookeeper problems yet, so I don't know  
where the pain points are.

So, without further ado,
	- Are there any places that logging is deficient that sorely needs  
improvement?
	- Could current logs be improved any amount or presented in a more  
readable fashion?
	- Would some form of log visualisation be useful (for example in  
something approximating a sequence diagram)?

Feel free to suggest anything which the list above doesn't allude to  
which you think would be helpful.

Cheers,
Ivan
	

Re: Solitication for logging/debugging requirements

Posted by Patrick Hunt <ph...@apache.org>.
Take a look at the logging page in the docs:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperInternals.html#sc_logging

Some good guidelines in there. Basically we log things at info level 
that are interesting/informational but not logged so frequently that 
they fill the log. WARN is for things that are bad but that we can 
handle (like network connectivity failure). ERROR is generally for 
things we don't expect and are unlikely we can handle. FATAL means 
really bad, we shutdown the server. Many end users log only at WARN 
level or higher in production, so typically we err on the side of WARN 
for issues (so that we have a shot at debugging after the fact). Over 
time, as we gain confidence in production environments, we've been 
pushing more things that were WARN down to INFO.

I fixed a number of JIRAs for 3.3 related to logging. In particular I 
cleaned up the client session logging significantly. The most fertile 
area right now to cleanup logging is in the quorum code. That code in 
particular has issues wrt providing sufficient information to debug 
error conditions. You can easily see this by starting an ensemble of 
greater than 1 machine and try killing one/more of the servers. There 
are many places where the logging is insufficient (eg. "got vote", which 
doesn't say what the vote was or what the effect of such a vote is, 
etc...) Having improved logging in this area would really help.

Try searching on the JIRA
https://issues.apache.org/jira/browse/ZOOKEEPER
of open/closed issues re "log4j" or "logging" or "log" for further insight.

Patrick

Benjamin Reed wrote:
> awesome! that would be great ivan. i'm sure pat has some more concrete 
> suggestions, but one simple thing to do is to run the unit tests and 
> look at the log messages that get output. there are a couple of 
> categories of things that need to be fixed (this is in no way exhaustive):
> 
> 1) messages that have useful information, but only if you look in the 
> code to figure out what it means. there are some leader election 
> messages that fall into this category. it would be nice to clarify them.
> 2) there are error messages that really aren't errors. when shutting 
> down there are a bunch of errors that are expected, but still logged, 
> for example.
> 3) misclassified error levels
> 
> welcome aboard!
> 
> ben
> 
> On 03/29/2010 10:07 AM, Ivan Kelly wrote:
>> Hi,
>>
>> Im going to be using Zookeeper quite extensively for a project in a
>> few weeks, but development hasn't kicked off yet. This means I have
>> some time on my hands and I'd like to get familiar with zookeeper
>> beforehand by perhaps writing some tools to make debugging problems
>> with it easier so as to save myself some time in the future. Problem
>> is I haven't had to debug many zookeeper problems yet, so I don't know
>> where the pain points are.
>>
>> So, without further ado,
>>     - Are there any places that logging is deficient that sorely needs
>> improvement?
>>     - Could current logs be improved any amount or presented in a more
>> readable fashion?
>>     - Would some form of log visualisation be useful (for example in
>> something approximating a sequence diagram)?
>>
>> Feel free to suggest anything which the list above doesn't allude to
>> which you think would be helpful.
>>
>> Cheers,
>> Ivan
>>     
>>    
> 

Re: Solitication for logging/debugging requirements

Posted by Patrick Hunt <ph...@apache.org>.
Take a look at the logging page in the docs:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperInternals.html#sc_logging

Some good guidelines in there. Basically we log things at info level 
that are interesting/informational but not logged so frequently that 
they fill the log. WARN is for things that are bad but that we can 
handle (like network connectivity failure). ERROR is generally for 
things we don't expect and are unlikely we can handle. FATAL means 
really bad, we shutdown the server. Many end users log only at WARN 
level or higher in production, so typically we err on the side of WARN 
for issues (so that we have a shot at debugging after the fact). Over 
time, as we gain confidence in production environments, we've been 
pushing more things that were WARN down to INFO.

I fixed a number of JIRAs for 3.3 related to logging. In particular I 
cleaned up the client session logging significantly. The most fertile 
area right now to cleanup logging is in the quorum code. That code in 
particular has issues wrt providing sufficient information to debug 
error conditions. You can easily see this by starting an ensemble of 
greater than 1 machine and try killing one/more of the servers. There 
are many places where the logging is insufficient (eg. "got vote", which 
doesn't say what the vote was or what the effect of such a vote is, 
etc...) Having improved logging in this area would really help.

Try searching on the JIRA
https://issues.apache.org/jira/browse/ZOOKEEPER
of open/closed issues re "log4j" or "logging" or "log" for further insight.

I'd also suggest you move this thread to zookeeper-dev mailing list, 
that's the proper forum for development discussion.

Regards,

Patrick

Benjamin Reed wrote:
> awesome! that would be great ivan. i'm sure pat has some more concrete 
> suggestions, but one simple thing to do is to run the unit tests and 
> look at the log messages that get output. there are a couple of 
> categories of things that need to be fixed (this is in no way exhaustive):
> 
> 1) messages that have useful information, but only if you look in the 
> code to figure out what it means. there are some leader election 
> messages that fall into this category. it would be nice to clarify them.
> 2) there are error messages that really aren't errors. when shutting 
> down there are a bunch of errors that are expected, but still logged, 
> for example.
> 3) misclassified error levels
> 
> welcome aboard!
> 
> ben
> 
> On 03/29/2010 10:07 AM, Ivan Kelly wrote:
>> Hi,
>>
>> Im going to be using Zookeeper quite extensively for a project in a
>> few weeks, but development hasn't kicked off yet. This means I have
>> some time on my hands and I'd like to get familiar with zookeeper
>> beforehand by perhaps writing some tools to make debugging problems
>> with it easier so as to save myself some time in the future. Problem
>> is I haven't had to debug many zookeeper problems yet, so I don't know
>> where the pain points are.
>>
>> So, without further ado,
>>     - Are there any places that logging is deficient that sorely needs
>> improvement?
>>     - Could current logs be improved any amount or presented in a more
>> readable fashion?
>>     - Would some form of log visualisation be useful (for example in
>> something approximating a sequence diagram)?
>>
>> Feel free to suggest anything which the list above doesn't allude to
>> which you think would be helpful.
>>
>> Cheers,
>> Ivan
>>     
>>    
> 

Re: Solitication for logging/debugging requirements

Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Moving the discussion to zookeeper-dev.

It is perhaps important to distinguish between message logging and  
transaction logging. Ben seems to be referring to message logging, the  
ones we output with log4j. I'm not sure if Ivan is talking about the  
same.

In any case, I think point 3 is interesting: looking at ways to  
visualize logs that could help with spotting problems.

-Flavio

On Mar 29, 2010, at 7:14 PM, Benjamin Reed wrote:

> awesome! that would be great ivan. i'm sure pat has some more concrete
> suggestions, but one simple thing to do is to run the unit tests and
> look at the log messages that get output. there are a couple of
> categories of things that need to be fixed (this is in no way  
> exhaustive):
>
> 1) messages that have useful information, but only if you look in the
> code to figure out what it means. there are some leader election
> messages that fall into this category. it would be nice to clarify  
> them.
> 2) there are error messages that really aren't errors. when shutting
> down there are a bunch of errors that are expected, but still logged,
> for example.
> 3) misclassified error levels
>
> welcome aboard!
>
> ben
>
> On 03/29/2010 10:07 AM, Ivan Kelly wrote:
>> Hi,
>>
>> Im going to be using Zookeeper quite extensively for a project in a
>> few weeks, but development hasn't kicked off yet. This means I have
>> some time on my hands and I'd like to get familiar with zookeeper
>> beforehand by perhaps writing some tools to make debugging problems
>> with it easier so as to save myself some time in the future. Problem
>> is I haven't had to debug many zookeeper problems yet, so I don't  
>> know
>> where the pain points are.
>>
>> So, without further ado,
>> 	- Are there any places that logging is deficient that sorely needs
>> improvement?
>> 	- Could current logs be improved any amount or presented in a more
>> readable fashion?
>> 	- Would some form of log visualisation be useful (for example in
>> something approximating a sequence diagram)?
>>
>> Feel free to suggest anything which the list above doesn't allude to
>> which you think would be helpful.
>>
>> Cheers,
>> Ivan
>> 	
>>
>


Re: Solitication for logging/debugging requirements

Posted by Benjamin Reed <br...@yahoo-inc.com>.
awesome! that would be great ivan. i'm sure pat has some more concrete 
suggestions, but one simple thing to do is to run the unit tests and 
look at the log messages that get output. there are a couple of 
categories of things that need to be fixed (this is in no way exhaustive):

1) messages that have useful information, but only if you look in the 
code to figure out what it means. there are some leader election 
messages that fall into this category. it would be nice to clarify them.
2) there are error messages that really aren't errors. when shutting 
down there are a bunch of errors that are expected, but still logged, 
for example.
3) misclassified error levels

welcome aboard!

ben

On 03/29/2010 10:07 AM, Ivan Kelly wrote:
> Hi,
>
> Im going to be using Zookeeper quite extensively for a project in a
> few weeks, but development hasn't kicked off yet. This means I have
> some time on my hands and I'd like to get familiar with zookeeper
> beforehand by perhaps writing some tools to make debugging problems
> with it easier so as to save myself some time in the future. Problem
> is I haven't had to debug many zookeeper problems yet, so I don't know
> where the pain points are.
>
> So, without further ado,
> 	- Are there any places that logging is deficient that sorely needs
> improvement?
> 	- Could current logs be improved any amount or presented in a more
> readable fashion?
> 	- Would some form of log visualisation be useful (for example in
> something approximating a sequence diagram)?
>
> Feel free to suggest anything which the list above doesn't allude to
> which you think would be helpful.
>
> Cheers,
> Ivan
> 	
>