Fetcher timeout

[Jonathan Reichhold <jd...@speakeasy.net>]

While testing out Nutch, I've discovered several issues with hangs 
inside of specific parsers, and realized that the Fetcher code has no 
concept of timeout on a thread.  From experience in doing whole web 
crawls, I've found this to be an essential feature for long-term 
stability (read hands-off production crawling for large indices)

As I'm coming into this codebase new, does the idea of a Fetch thread 
timeout exist (not just HTTP timeout) for a bad parser?  If so, how 
would I use set it?  If not, and looking at the code I believe this to 
be true, any issue with adding it?

Saw mentions from Doug Cutting on nutch-general on Oct 29th 2005

"Also, the mapred fetcher has been changed to succeed even when threads  
hang.  Perhaps we should change the 0.7 fetcher similarly?  I think we  
should probably go even farther, and kill threads which take longer than  
a timeout to process a url.  Thread.stop() is theoretically unsafe, but  
I've used it in the past for this sort of thing and never traced  
subsequent problems back to it...  "

Would agree with doug on this being "unsafe" but used it on large 
sites.  At the very least restarting the fetcher (can this be done) 
after this point would help get through the list.

Jonathan Reichhold