LogFormatter

[Daniel Feinstein <da...@rawsugar.com>]

Hi,
 
This is my first email to the community. In our (rawsugar.com) project we integrated multiple lucene indexes and a nutch
package. We use the packages for about two years. Lucene is working great and we could use it as a "black box".
Unfortunately, we had to modify Nutch package for our usage because of some leaks and a design problem in LogFormatter
class. This fact force us to use a customized version of Nutch and avoid to update versions. At the last week we decided
to move to the latest version of Nutch but discovered that the LogFormatter still have the same problem (leaks I believe
has been fixed).
 
The problem:
In Nutch project LogFormatter has duplicated functionality:
1) Logger records format and
2) Severe error handler
 
The first usage is standard and usually could be overwritten by a user of the package by modifying logging.properties
file.
The second usage is much more problematic because it affects the behavior of the whole application (not only Nutch
package). To support the error handling LogFormatter enforce usage of the formatter class by all classes of the whole
application which uses Nutch package. This is done by overwriting all the system handlers (class
java.util.logging.Handler). This operation prevents the application to use its own log formatter. Also this cause
LogFormatter.hasLoggedSevere() to be sensitive to all severe records in the big system but not only to relevant. More
than that this flag, LogFormatter.loggedSevere is never cleaned what means if an application had one, even unrelated
severe record, tools like Fetcher will never run until the application will be restarted.
 
I would like to suggest the following solutions:
1) To separate the functionality of log formatting and error handling or
2) Change LogFormatter class to be affected only by nutch package functions
 
For my opinion the first solution is much better especially if error handling will be encapsulated for each task. I have
found the following usages of LogFormatter.hasLoggedSevere():
- Fetcher
- URLFilterChecker
- ParseSegment
Unfortunately I'm not familiar enough with the usages above to implement this solution that why I suggest the second
one. The attached file is our implementation of LogFormatter class which we use for more than a year. I hope this change
will be accepted by the community.
 
Daniel
_____________________
Daniel Feinstein
Lead Software Engineer
www.rawsugar.com <http://www.rawsugar.com/> 
 
__________________________

CONFIDENTIALITY NOTICE

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review or
distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and
delete all copies.

 

Re: LogFormatter

[Erik Hatcher <er...@ehatchersolutions.com>]

Daniel,

Please post this to the Nutch list, not the Lucene one.

	Erik


On Jan 3, 2006, at 5:42 AM, Daniel Feinstein wrote:

> Hi,
>
> This is my first email to the community. In our (rawsugar.com)  
> project we integrated multiple lucene indexes and a nutch package.  
> We use the packages for about two years. Lucene is working great  
> and we could use it as a "black box". Unfortunately, we had to  
> modify Nutch package for our usage because of some leaks and a  
> design problem in LogFormatter class. This fact force us to use a  
> customized version of Nutch and avoid to update versions. At the  
> last week we decided to move to the latest version of Nutch but  
> discovered that the LogFormatter still have the same problem (leaks  
> I believe has been fixed).
>
> The problem:
> In Nutch project LogFormatter has duplicated functionality:
> 1) Logger records format and
> 2) Severe error handler
>
> The first usage is standard and usually could be overwritten by a  
> user of the package by modifying logging.properties file.
> The second usage is much more problematic because it affects the  
> behavior of the whole application (not only Nutch package). To  
> support the error handling LogFormatter enforce usage of the  
> formatter class by all classes of the whole application which uses  
> Nutch package. This is done by overwriting all the system handlers  
> (class java.util.logging.Handler). This operation prevents the  
> application to use its own log formatter. Also this cause  
> LogFormatter.hasLoggedSevere() to be sensitive to all severe  
> records in the big system but not only to relevant. More than that  
> this flag, LogFormatter.loggedSevere is never cleaned what means if  
> an application had one, even unrelated severe record, tools like  
> Fetcher will never run until the application will be restarted.
>
> I would like to suggest the following solutions:
> 1) To separate the functionality of log formatting and error  
> handling or
> 2) Change LogFormatter class to be affected only by nutch package  
> functions
>
> For my opinion the first solution is much better especially if  
> error handling will be encapsulated for each task. I have found the  
> following usages of LogFormatter.hasLoggedSevere():
> - Fetcher
> - URLFilterChecker
> - ParseSegment
> Unfortunately I'm not familiar enough with the usages above to  
> implement this solution that why I suggest the second one. The  
> attached file is our implementation of LogFormatter class which we  
> use for more than a year. I hope this change will be accepted by  
> the community.
>
> Daniel
> _____________________
> Daniel Feinstein
> Lead Software Engineer
> www.rawsugar.com
>
> __________________________
> CONFIDENTIALITY NOTICE
> This email may contain confidential and privileged material for the  
> sole use of the intended recipient. Any review or distribution by  
> others is strictly prohibited. If you are not the intended  
> recipient please contact the sender and delete all copies.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org