Fetching even after timeout

[og...@yahoo.com]

Hello,

Here is one puzzling thing.  Nutch tries to fetch robots.txt from a site X (foobar.com in example). It fails due to a timeout.  Then, a few moments later, it decides to fetch a page from that site X *anyway*.  Example from the log:

2008-04-08 15:30:27,480 INFO  api.RobotRulesParser - Couldn't get robots.txt for http://www.foobar.com/8.php: java.net.SocketTimeoutExceptio
n: connect timed out
2008-04-08 15:30:27,500 INFO  fetcher.Fetcher - fetching http://www.....
2008-04-08 15:30:27,535 INFO  fetcher.Fetcher - fetching http://www......
2008-04-08 15:30:27,560 INFO  fetcher.Fetcher - fetching http://www......
2008-04-08 15:30:28,124 INFO  fetcher.Fetcher - fetching http://www.foobar.com/14429.php


Is this the intended behaviour?  This doesn't look good to me for a few reasons:

1) if robots.txt was not retrieved, why is Nutch assuming it can fetch pages from the site anyway?  It sounds like a potentially bad assumption - robots.txt could be denying access to everything, so maybe Nutch should be conservative and skip that host all together until it can fetch robots.txt successfully (or get a 404 for it).

2) if robots.txt could not be retrieved due to a *timeout*, then it is likely (though not certain) that Nutch won't be able to fetch other pages from that site either.  Wouldn't it make more sense to "deprecate" that host, or at least have a config option to do so after robots.txt fetching fails? (assumption here is that robots.txt is typically a very small and static file, so if that times out, the chances are high that everything else from that site will time out)

Why does Nutch do this?
 
Thanks,
Otis

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch