ProtocolStatus: redirected and blocked for robots

[Otis Gospodnetic <ot...@yahoo.com>]

Hello,

I'm trying to detect redirected Urls and Urls blocked by robots.txt,
but it appears ProtocolStatus values for those cases are not set.

This is how I'm trying to check if the Url was redirected:
(_currentOutput is FetcherOutput)

    public boolean isFetchRedirect() {
        int code = _currentOutput.getProtocolStatus().getCode();
        return (code == ProtocolStatus.MOVED)
        || (code == ProtocolStatus.TEMP_MOVED)
        || (code == ProtocolStatus.REDIR_EXCEEDED);
    }

This is how I'm trying to check if the Url was blocked by robots.txt:

    public boolean isFetchBlockedForRobots() {
        return _currentOutput.getProtocolStatus().getCode() ==
ProtocolStatus.ROBOTS_DENIED;
    }


Is this the right way to do it?

... after a bit of debugging I found that the latter case (Urls blocked
by robots) is inconsistent.  For example:

http://del.icio.us/merlinmann
STATUS: notfound(14), lastModified=0

http://del.icio.us/biketrouble
STATUS: exception(16), lastModified=0:
org.apache.nutch.protocol.ResourceGone: Blocked by robots.txt


That STATUS line is ProtocolStatus.toString();

I'm using the freshly baked Nutch 0.7-dev from SVN.  Baked this
morning.

Thanks,
Otis

____________________________________________________________________
Simpy -- simpy.com -- tags, social bookmarks, personal search engine