You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Scott Ganyo (JIRA)" <ji...@apache.org> on 2006/04/28 22:39:37 UTC
[jira] Created: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Once Nutch logs a SEVERE log item, Nutch fails forevermore
----------------------------------------------------------
Key: NUTCH-258
URL: http://issues.apache.org/jira/browse/NUTCH-258
Project: Nutch
Type: Bug
Components: fetcher
Versions: 0.8-dev
Environment: All
Reporter: Scott Ganyo
Priority: Critical
Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
public void run() {
synchronized (Fetcher.this) {activeThreads++;} // count threads
try {
UTF8 key = new UTF8();
CrawlDatum datum = new CrawlDatum();
while (true) {
if (LogFormatter.hasLoggedSevere()) // something bad happened
break; // exit
Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
(Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann resolved NUTCH-258:
-------------------------------------
Resolution: Won't Fix
The use of LOG.severe in the fetcher indicates an unrecoverable error: thus, this issue is not a bug, and in fact describes the actual intended behavior of the system.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore
Posted by Stefan Groschupf <sg...@media-style.com>.
> I have a proposal for a simple solution: set a flag in the current
> Configuration instance, and check for this flag. The Configuration
> instance provides a task-specific context persisting throughout the
> lifetime of a task - but limited only to that task. Voila - problem
> solved. We get rid of the dubious use of LogFormatter (I hope Chris
> that even you would agree that this pattern is slightly ..
> unusual ;) ), and we gain flexible mechanism limited in scope to
> the current task, which ensures isolation from other tasks in the
> same JVM. How about that?
Wonderful idea!!!! :-D
+ 1
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by Andrzej Bialecki <ab...@getopt.org>.
Chris Mattmann wrote:
> +1
>
> So, to summarize, the proposed resolution is:
>
> * add flag field in Configuration instance to signify whether or not a
> SEVERE error has been logged within a task's context
>
Yes, preferably define this as a public static final String-s in
NutchConfiguration, both the field name and this special field value.
The value should be a (short) string too, to minimize conversions
from/to other formats.
> * check this field within the fetcher to determine whether or not to stop
> the fetcher, just for that fetching task identified by its Configuration
> (and no others)
>
Yes.
> Is this representative of what you're proposing Andrzej? If so, I'd like to
> take the lead on contributing a small patch that handles this, and then it
> would be great if people like Scott could test this out in their existing
> environments where this error was manifesting itself.
>
> Thanks!
>
> Cheers,
> Chris
>
> (BTW: would you like me to re-open the JIRA issue, or do you want to do it?)
>
Sure, feel free to follow-up on this to its conclusion :)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore
Posted by Scott Ganyo <sc...@apache.org>.
Thanks, Chris! (And thank you, Andrzej for interpreting my rantings!)
That plan sounds fantastic and I would be happy to help out.
Scott
On Jun 5, 2006, at 1:01 PM, Chris Mattmann wrote:
> Hi Andrzej,
>
>>
>> The main problem, as Scott observed, is that the static flag
>> affects all
>> instances of the task executing inside the same JVM. If there are
>> several Fetcher tasks (or any other tasks that check for SEVERE
>> flag!),
>> belonging to different jobs, all of them will quit. This is certainly
>> not the intended behavior.
>>
>
> Got it.
>
>>>
>>>> In fact, I believe that this would make a fantastic anti-
>>>> pattern. If this
>>>> kind of behavior is *really* wanted (and I argue that it should
>>>> not be
>>>> below),
>>>> it should be done through an explicit mechanism, not as a side-
>>>> effect.
>>>>
>>>
>>>
>>
>> I have a proposal for a simple solution: set a flag in the current
>> Configuration instance, and check for this flag. The Configuration
>> instance provides a task-specific context persisting throughout the
>> lifetime of a task - but limited only to that task. Voila - problem
>> solved. We get rid of the dubious use of LogFormatter (I hope
>> Chris that
>> even you would agree that this pattern is slightly .. unusual ;) )
>
> What, "unusual"? Huh? :-)
>
>> and
>> we gain flexible mechanism limited in scope to the current task,
>> which
>> ensures isolation from other tasks in the same JVM. How about that?
>
> +1
>
> I like your proposed solution. I haven't used multiple fetchers really
> inside the same process too, much however, I do have an application
> that
> calls fetches in more of a sequential way in the same JVM. So, I
> guess I
> just never ran across the behavior. The thing I like about the
> proposed
> solution is its separation and isolation of a task context, which I
> think
> that Nutch (now relying on Hadoop as the underlying architectural
> computing
> platform) needed to address.
>
> So, to summarize, the proposed resolution is:
>
> * add flag field in Configuration instance to signify whether or not a
> SEVERE error has been logged within a task's context
>
> * check this field within the fetcher to determine whether or not
> to stop
> the fetcher, just for that fetching task identified by its
> Configuration
> (and no others)
>
> Is this representative of what you're proposing Andrzej? If so, I'd
> like to
> take the lead on contributing a small patch that handles this, and
> then it
> would be great if people like Scott could test this out in their
> existing
> environments where this error was manifesting itself.
>
> Thanks!
>
> Cheers,
> Chris
>
> (BTW: would you like me to re-open the JIRA issue, or do you want
> to do it?)
>
> ______________________________________________
> Chris A. Mattmann
> Chris.Mattmann@jpl.nasa.gov
> Staff Member
> Modeling and Data Management Systems Section (387)
> Data Management Systems and Technologies Group
>
> _________________________________________________
> Jet Propulsion Laboratory Pasadena, CA
> Office: 171-266B Mailstop: 171-246
> _______________________________________________________
>
> Disclaimer: The opinions presented within are my own and do not
> reflect
> those of either NASA, JPL, or the California Institute of Technology.
>
>
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log
item, Nutch fails forevermore
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
Hi Andrzej,
>
> The main problem, as Scott observed, is that the static flag affects all
> instances of the task executing inside the same JVM. If there are
> several Fetcher tasks (or any other tasks that check for SEVERE flag!),
> belonging to different jobs, all of them will quit. This is certainly
> not the intended behavior.
>
Got it.
>>
>>> In fact, I believe that this would make a fantastic anti-pattern. If this
>>> kind of behavior is *really* wanted (and I argue that it should not be
>>> below),
>>> it should be done through an explicit mechanism, not as a side-effect.
>>>
>>
>>
>
> I have a proposal for a simple solution: set a flag in the current
> Configuration instance, and check for this flag. The Configuration
> instance provides a task-specific context persisting throughout the
> lifetime of a task - but limited only to that task. Voila - problem
> solved. We get rid of the dubious use of LogFormatter (I hope Chris that
> even you would agree that this pattern is slightly .. unusual ;) )
What, "unusual"? Huh? :-)
> and
> we gain flexible mechanism limited in scope to the current task, which
> ensures isolation from other tasks in the same JVM. How about that?
+1
I like your proposed solution. I haven't used multiple fetchers really
inside the same process too, much however, I do have an application that
calls fetches in more of a sequential way in the same JVM. So, I guess I
just never ran across the behavior. The thing I like about the proposed
solution is its separation and isolation of a task context, which I think
that Nutch (now relying on Hadoop as the underlying architectural computing
platform) needed to address.
So, to summarize, the proposed resolution is:
* add flag field in Configuration instance to signify whether or not a
SEVERE error has been logged within a task's context
* check this field within the fetcher to determine whether or not to stop
the fetcher, just for that fetching task identified by its Configuration
(and no others)
Is this representative of what you're proposing Andrzej? If so, I'd like to
take the lead on contributing a small patch that handles this, and then it
would be great if people like Scott could test this out in their existing
environments where this error was manifesting itself.
Thanks!
Cheers,
Chris
(BTW: would you like me to re-open the JIRA issue, or do you want to do it?)
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by Andrzej Bialecki <ab...@getopt.org>.
Chris Mattmann wrote:
> Folks,
>
> Before I (or someone else) reopens the issue, I think it's important to
> understand the implications:
>
I vote for re-opening. See below.
>
>> 1) Having a *side-effect* of the entire system stop processing after merely
>> logging a message at a certain event level is a poor practice.
>>
>
> I'm not sure that the Fetcher quitting is a * side-effect * as you call it.
> In fact, I think it's clearly stated as the behavior of the system, both
> within the code, and in several mailing list conversations I've seen over
> the course of the past two years (I can dig these up, if needed).
>
The main problem, as Scott observed, is that the static flag affects all
instances of the task executing inside the same JVM. If there are
several Fetcher tasks (or any other tasks that check for SEVERE flag!),
belonging to different jobs, all of them will quit. This is certainly
not the intended behavior.
>
>> In fact, I believe that this would make a fantastic anti-pattern. If this
>> kind of behavior is *really* wanted (and I argue that it should not be below),
>> it should be done through an explicit mechanism, not as a side-effect.
>>
>
>
I have a proposal for a simple solution: set a flag in the current
Configuration instance, and check for this flag. The Configuration
instance provides a task-specific context persisting throughout the
lifetime of a task - but limited only to that task. Voila - problem
solved. We get rid of the dubious use of LogFormatter (I hope Chris that
even you would agree that this pattern is slightly .. unusual ;) ), and
we gain flexible mechanism limited in scope to the current task, which
ensures isolation from other tasks in the same JVM. How about that?
> I've been using Nutch in a server environment (JSPs and Tomcat) within a
> large-scale data system at NASA for the course of the past year, and have
> never been impeded by the behavior of the fetcher. Can you be more specific
>
Have you ever tried to run several different crawls inside the same JVM?
That's a common requirement if you want to use Nutch as a "crawler
component" inside a larger application. I have, and as a result of my
bad experiences I initiated the discussion, which led to the "dynamic
NutchConf" patches implemented by Stefan. The issue of LogFormatter has
been discussed also about that time, but since we hadn't had "dynamic
NutchConf" yet it was postponed, because there was no clear idea how to
solve it cleanly. I believe there is now.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log
item, Nutch fails forevermore
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
Folks,
Before I (or someone else) reopens the issue, I think it's important to
understand the implications:
>1) Having a *side-effect* of the entire system stop processing after merely
> logging a message at a certain event level is a poor practice.
I'm not sure that the Fetcher quitting is a * side-effect * as you call it.
In fact, I think it's clearly stated as the behavior of the system, both
within the code, and in several mailing list conversations I've seen over
the course of the past two years (I can dig these up, if needed).
> In fact, I believe that this would make a fantastic anti-pattern. If this
> kind of behavior is *really* wanted (and I argue that it should not be below),
> it should be done through an explicit mechanism, not as a side-effect.
Again, the use of side-effect here is strange to me: how is an explicit
check for any LOG messages to the SEVERE level before quitting a
"side-effect"?
> For example, did you realize that since Hadoop hijacks and reassigns all log
> formatters (also a bad practice!) in the org.apache.hadoop.util.LogFormatter
> static constructor that anyone using Nutch as a library and logs a SEVERE\
> error will suffer by having Nutch stop fetching?
I'm not convinced that having Nutch stop fetching when a SEVERE error is
logged is the wrong behavior. Let's think about what possible SEVERE errors
may typically be logged: Out of Memory error, potentially,
InterruptedExceptions in Threads (possibly), failure in any of the plugin
libraries critical to the fetch running (possibly), the list goes on and on.
So, in this case, you argue that the Fetcher should continue operating?
> 2) Moreover, having the system stop processing forever more by use of a
> static(!) flag makes the use of the Nutch system as a library within a server
> or service environment impossible. Once this logging is done, no more Fetcher
> processing in this run *or any other* can take place.
I've been using Nutch in a server environment (JSPs and Tomcat) within a
large-scale data system at NASA for the course of the past year, and have
never been impeded by the behavior of the fetcher. Can you be more specific
here as to the exact use-case that's failing in your scenario? I've also
been watching the mailing lists for the better course of almost 2 years, and
have seen little traffic (outside of the aforementioned clarifications/etc.
above) about this issue. I may be out on an island here, but again, I'm not
convinced that this is a core issue.
Just my 2 cents. If the votes continue that this is an issue, however, I'll
have no problem opening it up (or one of the committers can do it as well).
Cheers,
Chris
On 6/5/06 7:11 AM, "Stefan Groschupf (JIRA)" <ji...@apache.org> wrote:
> [
> http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12414763 ]
>
> Stefan Groschupf commented on NUTCH-258:
> ----------------------------------------
>
> Scott,
> I agree with you. However we need a clean patch to solve the problem, we can
> not just comment things out of the code.
> So I vote for the issue and I vote to reopen this issue.
>
>> Once Nutch logs a SEVERE log item, Nutch fails forevermore
>> ----------------------------------------------------------
>>
>> Key: NUTCH-258
>> URL: http://issues.apache.org/jira/browse/NUTCH-258
>> Project: Nutch
>> Type: Bug
>
>> Components: fetcher
>> Versions: 0.8-dev
>> Environment: All
>> Reporter: Scott Ganyo
>> Priority: Critical
>> Attachments: dumbfix.patch
>>
>> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore.
>> This is from the run() method in Fetcher.java:
>> public void run() {
>> synchronized (Fetcher.this) {activeThreads++;} // count threads
>>
>> try {
>> UTF8 key = new UTF8();
>> CrawlDatum datum = new CrawlDatum();
>>
>> while (true) {
>> if (LogFormatter.hasLoggedSevere()) // something bad happened
>> break; // exit
>>
>> Notice the last 2 lines. This will prevent Nutch from ever Fetching again
>> once this is hit as LogFormatter is storing this data as a static.
>> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in
>> org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
>> This must be fixed or Nutch cannot be run as any kind of long-running
>> service. Furthermore, I believe it is a poor decision to rely on a logging
>> event to determine the state of the application - this could have any number
>> of side-effects that would be extremely difficult to track down. (As it has
>> already for me.)
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
Phone: 818-354-8810
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12414763 ]
Stefan Groschupf commented on NUTCH-258:
----------------------------------------
Scott,
I agree with you. However we need a clean patch to solve the problem, we can not just comment things out of the code.
So I vote for the issue and I vote to reopen this issue.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12416379 ]
Chris A. Mattmann commented on NUTCH-258:
-----------------------------------------
> Thanks for this patch Chris - even if now it is outdate by NUTCH-303 :-(
> Since Nutch no more use the deprecated Hadoop LogFormatter, there is no more logSevere check in the code.
Oh Jerome. You're always trying to scoop me on stuff! ;)
> But I'm not sure all these log severe should be marked as severe (fatal level is used now).
Agreed. Let's review the places in the patch where severe errors are logged, and then remove/add as deemed necessary.
> So, what I suggest is to review all the fatal logs and check if they are really fatal for the whole process.
Agreed. I'll get on this right away.
> And finally, why not simply throwing a RuntimeException that will by catched the Fetcher if something wrong really occurs?
Because we don't want one RuntimeException killing all subsequent fetching tasks. See the previous discussions on this by Andrzej, Scott, and I. Basically it boils down to ensuring that LOG.severe and its associated checking mechanism is associated within the context of a particular fetching task that executes: we believed that the best way to do that would be to use the Hadoop Configuration (which is task specific). Make sense?
Okey dokey, I'll work on an updated patch and submit for review soon (I won't specify an exact date, because I'm always late ;) ).
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assignee: Chris A. Mattmann
> Priority: Critical
> Attachments: NUTCH-258.Mattmann.060906.patch.txt, dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann updated NUTCH-258:
------------------------------------
Attachment: NUTCH-258.Mattmann.080406.patch.txt
Hi Folks,
Sorry I'm a little later than I expected on this one. Attached is a patch that implements the suggested fix for NUTCH-258 by Jerome. Everywhere that a SEVERE (or I guess it's called "Fatal" now) error is logged, a RuntimeException is thrown that is caught in the outermost loop of the Fetcher, which should stop only the current Fetcher thread that catches the Exception. Additionally, I have tried to clean up places where I felt a LOG.fatal wasn't warranted, so please do confirm that everywhere that I cleaned up is in fact places that we don't want LOG.fatal, otherwise, feel free to move them back to LOG.fatal. The only caveat is, there has to be some discipline by the programmer ensuring that everywhere LOG.fatal is used, a RuntimeException should be thrown if your code is touched by the Fetcher during fetching.
I tested this patch against the latest Nutch SVN and all unit tests are passing. It would be great if someone could test this in a distributed environment.
Thanks,
Chris
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8
> Environment: All
> Reporter: Scott Ganyo
> Assigned To: Chris A. Mattmann
> Priority: Critical
> Fix For: 0.9.0
>
> Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt, NUTCH-258.Mattmann.080406.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12429035 ]
Chris A. Mattmann commented on NUTCH-258:
-----------------------------------------
Hi Folks,
A patch is available on this issue. Has anyone who was experiencing the original problem tried out the latest trunk with this patch applied? Does this patch resolve your issue?
Thanks,
Chris
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8
> Environment: All
> Reporter: Scott Ganyo
> Assigned To: Chris A. Mattmann
> Priority: Critical
> Fix For: 0.9.0
>
> Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt, NUTCH-258.Mattmann.080406.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12422962 ]
Chris A. Mattmann commented on NUTCH-258:
-----------------------------------------
Guys,
This issue slipped off my radar for a bt, but I'll have some free time this week to work on it. If there are no objections, I will implement what Jerome suggested: that is, throwing a RuntimeException instead of setting a flag in the NutchConfiguration, as I believe that it has the same effect.
Thanks,
Chris
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assigned To: Chris A. Mattmann
> Priority: Critical
> Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann updated NUTCH-258:
------------------------------------
Fix Version/s: 0.8-dev
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assigned To: Chris A. Mattmann
> Priority: Critical
> Fix For: 0.8-dev
>
> Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Stefan Neufeind (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Stefan Neufeind updated NUTCH-258:
----------------------------------
Attachment: dumbfix.patch
I know this is a dumb fix :-) But it solves the problem for the moment ...
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch
fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann closed NUTCH-258:
-----------------------------------
Won't fix: issue describes intended behavior of system (fetcher component).
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12416526 ]
Jerome Charron commented on NUTCH-258:
--------------------------------------
> Because we don't want one RuntimeException killing all subsequent fetching tasks
Chris, a RuntimeException will not kill all subsequent fetching tasks. It will acts exactly the same way you do with the configuration: Since there is a try ... catch statement, only the Thread that received the exception will be impacted.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assignee: Chris A. Mattmann
> Priority: Critical
> Attachments: NUTCH-258.Mattmann.060906.patch.txt, dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12415984 ]
Jerome Charron commented on NUTCH-258:
--------------------------------------
Thanks for this patch Chris - even if now it is outdate by NUTCH-303 :-(
Since Nutch no more use the deprecated Hadoop LogFormatter, there is no more logSevere check in the code.
So we quickly need to have a patch for this issue in order to have the same behaviors.
In your patch Chris, you set a severe flag each time a log severe is called.
But I'm not sure all these log severe should be marked as severe (fatal level is used now).
For instance, is it really fatal for the fetcher that the conf file for RegexUrlNormalizer is wrong?
Is it really fatal for the fetcher if the language identifier raise an exception while loading ngrams profiles?
Is it really fatal for the fetcher if the ontology plugin failed on reading an ontology?
But sure it is fatal if the user-agent is not correctly setted in http plugins!
So, what I suggest is to review all the fatal logs and check if they are really fatal for the whole process.
And finally, why not simply throwing a RuntimeException that will by catched the Fetcher if something wrong really occurs?
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assignee: Chris A. Mattmann
> Priority: Critical
> Attachments: NUTCH-258.Mattmann.060906.patch.txt, dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Stefan Neufeind (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12412705 ]
Stefan Neufeind commented on NUTCH-258:
---------------------------------------
Beware of simply silencing the error! It helped me at one place - but at another it really caused an infinite loop not to end.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12414598 ]
Chris A. Mattmann commented on NUTCH-258:
-----------------------------------------
Hi there,
I believe that the fetcher halting on a LOG.Severe is the intended behavior of the system. The use of this SEVERE error in Nutch is pretty consistent with Sun's documentation (http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/overview.html#1.2), including its javadoc for JDK 5 (http://java.sun.com/j2se/1.5.0/docs/api/java/util/logging/Level.html). A SEVERE error is defined as "a message level indicating a serious failure.". So I think that in the case of the Fetcher, that this behavior is actually warranted, considering if anything got logged to the SEVERE level, then there was some serious, un-recoverable error while fetching.
If you believe that there is an inappropriate use of LOG.severe, however, in the Fetcher, for instance, if an informational message is being logged to the SEVERE level, then that's a separate issue, and please indicate where this is happening, However, as I stated, I believe SEVERE errors causing the fetcher to halt is indeed the intended behavior of Nutch, so, if there are no objections, I would like to close this issue.
Thanks,
Chris
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Stefan Neufeind (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12414646 ]
Stefan Neufeind commented on NUTCH-258:
---------------------------------------
Agreed. The root-causee of the loop should be identified. So I'd suggest turning this into a wont-fix-bug - and if it occurs again somewhere, we should try to track down the root cause.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Sami Siren (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Sami Siren updated NUTCH-258:
-----------------------------
Fix Version/s: 0.9
(was: 0.8)
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8
> Environment: All
> Reporter: Scott Ganyo
> Assigned To: Chris A. Mattmann
> Priority: Critical
> Fix For: 0.9
>
> Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann updated NUTCH-258:
------------------------------------
Attachment: NUTCH-258.Mattmann.060906.patch.txt
Hi Folks,
Attached is a patch that implements the suggested two fixes to this issue. I had to go through the Nutch code and look for LOG.severe calls, and then add an additional:
conf.set(NutchConfiguration.LOG_SEVERE_FIELD, NutchConfiguration.LOG_SEVERE);
at the bottom of it. I had to go through several places in the code too where SEVERE errors were being logged and make sure that those pieces of code had access to the Configuration object. I ran unit-level tests and compilation, but no system level tests. Could Scott or someone else who was experiencing this problem test out this patch and then let me know if this fixes the issue?
Thanks!
Cheers,
Chris
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assignee: Chris A. Mattmann
> Priority: Critical
> Attachments: NUTCH-258.Mattmann.060906.patch.txt, dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log
item, Nutch fails forevermore
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
> It's at least out-of-date and perhaps obsolete. A quick read of
> Fetcher.java looks like there might be a case where a "fatal" error is
> logged but the fetcher doesn't exit, in FetcherThread#output().
>
So this raises an interesting question:
People (such as Scott G.) out there -- are you folks still experiencing
similar problems? Do the recent Hadoop changes alleviate the bad behavior
you were experiencing? If so, then maybe this issue should be closed...
Cheers,
Chris
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by Doug Cutting <cu...@apache.org>.
Chris Mattmann wrote:
> So, does this render the patch that I wrote obsolete?
It's at least out-of-date and perhaps obsolete. A quick read of
Fetcher.java looks like there might be a case where a "fatal" error is
logged but the fetcher doesn't exit, in FetcherThread#output().
Doug
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log
item, Nutch fails forevermore
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
Hi Doug,
So, does this render the patch that I wrote obsolete?
Cheers,
Chris
On 1/25/07 10:08 AM, "Doug Cutting" <cu...@apache.org> wrote:
> Scott Ganyo (JIRA) wrote:
>> ... since Hadoop hijacks and reassigns all log formatters (also a bad
>> practice!) in the org.apache.hadoop.util.LogFormatter static constructor ...
>
> FYI, Hadoop no longer does this.
>
> Doug
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
Re: [jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by Doug Cutting <cu...@apache.org>.
Scott Ganyo (JIRA) wrote:
> ... since Hadoop hijacks and reassigns all log formatters (also a bad practice!) in the org.apache.hadoop.util.LogFormatter static constructor ...
FYI, Hadoop no longer does this.
Doug
[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Scott Ganyo (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=comments#action_12414762 ]
Scott Ganyo commented on NUTCH-258:
-----------------------------------
For the record: I strongly object to closing this issue for the following reasons:
1) Having a *side-effect* of the entire system stop processing after merely logging a message at a certain event level is a poor practice. In fact, I believe that this would make a fantastic anti-pattern. If this kind of behavior is *really* wanted (and I argue that it should not be below), it should be done through an explicit mechanism, not as a side-effect. For example, did you realize that since Hadoop hijacks and reassigns all log formatters (also a bad practice!) in the org.apache.hadoop.util.LogFormatter static constructor that anyone using Nutch as a library and logs a SEVERE error will suffer by having Nutch stop fetching?
2) Moreover, having the system stop processing forever more by use of a static(!) flag makes the use of the Nutch system as a library within a server or service environment impossible. Once this logging is done, no more Fetcher processing in this run *or any other* can take place. This is inappropriate. You might as well call System.exit() at this point! In fact, I could even argue that the current behavior is worse than a System.exit(), as it can actually obfuscate why the system has ceased being operational even though it is still ostensibly "running."
Thus, while there definitely *are* instances of inappropriate logging levels being used and I could document them, I believe that this issue is more endemic to the system and it's architecture than the utilization of a particular logging level for a certain event.
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Reopened: (NUTCH-258) Once Nutch logs a SEVERE log item,
Nutch fails forevermore
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-258?page=all ]
Chris A. Mattmann reopened NUTCH-258:
-------------------------------------
Assign To: Chris A. Mattmann
Issue found to in fact be a real issue with the Fetcher: here's the proposed solution:
* add flag field (preferably a public static final short String) in Configuration instance to signify whether or not a
SEVERE error has been logged within a task's context
* check this field within the fetcher to determine whether or not to stop
the fetcher, just for that fetching task identified by its Configuration
(and no others)
> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
> Key: NUTCH-258
> URL: http://issues.apache.org/jira/browse/NUTCH-258
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Environment: All
> Reporter: Scott Ganyo
> Assignee: Chris A. Mattmann
> Priority: Critical
> Attachments: dumbfix.patch
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore. This is from the run() method in Fetcher.java:
> public void run() {
> synchronized (Fetcher.this) {activeThreads++;} // count threads
>
> try {
> UTF8 key = new UTF8();
> CrawlDatum datum = new CrawlDatum();
>
> while (true) {
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break; // exit
>
> Notice the last 2 lines. This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service. Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down. (As it has already for me.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira