You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Sami Siren (JIRA)" <ji...@apache.org> on 2006/07/25 21:40:26 UTC
[jira] Updated: (NUTCH-247) robot parser to restrict.
[ http://issues.apache.org/jira/browse/NUTCH-247?page=all ]
Sami Siren updated NUTCH-247:
-----------------------------
Fix Version/s: 0.9-dev
(was: 0.8-dev)
> robot parser to restrict.
> -------------------------
>
> Key: NUTCH-247
> URL: http://issues.apache.org/jira/browse/NUTCH-247
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8-dev
> Reporter: Stefan Groschupf
> Priority: Minor
> Fix For: 0.9-dev
>
>
> If the agent name and the robots agents are not proper configure the Robot rule parser uses LOG.severe to log the problem but solve it also.
> Later on the fetcher thread checks for severe errors and stop if there is one.
> RobotRulesParser:
> if (agents.size() == 0) {
> agents.add(agentName);
> LOG.severe("No agents listed in 'http.robots.agents' property!");
> } else if (!((String)agents.get(0)).equalsIgnoreCase(agentName)) {
> agents.add(0, agentName);
> LOG.severe("Agent we advertise (" + agentName
> + ") not listed first in 'http.robots.agents' property!");
> }
> Fetcher.FetcherThread:
> if (LogFormatter.hasLoggedSevere()) // something bad happened
> break;
> I suggest to use warn or something similar instead of severe to log this problem.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira