You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ed Whittaker <ed...@ewdw.com> on 2006/01/07 00:19:10 UTC
Urgent help requested regarding Nutch obeying instructions
Does Nutch obey the following instruction:
<META NAME="ROBOTS" CONTENT="NOARCHIVE">
It seems the crawler I ran has not obeyed them and I have a large number of
cached pages that may affect page writers' copyright. Some users are already
complaining!! Presumably I am not the only one unless I did something
incorrectly during the crawl although I ran Nutch out of the box as per the
tutorial. Is anyone else using Nutch in a live system that's publicly
accessible? Is not Nutch not yet intended for such public use?
FYI it seems Google and Yahoo do obey this directive.
-Ed
Re: Urgent help requested regarding Nutch obeying instructions
Posted by Jérôme Charron <je...@gmail.com>.
> Many thanks for the info. Any suggestions as to how I should respond to
> complainants?
Short time workaround : Remove access and link to the nutch cache page.
> However non-standard the directive is it does seem that Google, Yahoo and
> MSN support it which makes it a somewhat de facto standard doesn't it?
Create a JIRA issue so that we can discuss about integrating this in nutch.
http://nagoya.apache.org/jira/browse/Nutch
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
Re: Urgent help requested regarding Nutch obeying instructions
Posted by Ed Whittaker <ed...@ewdw.com>.
Many thanks for the info. Any suggestions as to how I should respond to
complainants?
However non-standard the directive is it does seem that Google, Yahoo and
MSN support it which makes it a somewhat de facto standard doesn't it?
-Ed
----- Original Message -----
From: "Jérôme Charron" <je...@gmail.com>
To: <nu...@lucene.apache.org>
Sent: Saturday, January 07, 2006 8:37 AM
Subject: Re: Urgent help requested regarding Nutch obeying <META
CONTENT="NOARCHIVE"> instructions
Nutch Html Parser support standard directives:
<meta name="robots" content="nofollow"/>
<meta name="robots" content="noindex"/>
<meta name="robots" content="none"/>
<meta name="robots" content="all"/>
noarchive is not a standard meta robots exclusion (it is a Google specific
one)
(see http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4.1.2 or
http://www.robotstxt.org/wc/meta-user.html)
Regards
Jérôme
On 1/7/06, Ed Whittaker <ed...@ewdw.com> wrote:
>
> Does Nutch obey the following instruction:
>
> <META NAME="ROBOTS" CONTENT="NOARCHIVE">
>
> It seems the crawler I ran has not obeyed them and I have a large number
> of
> cached pages that may affect page writers' copyright. Some users are
> already
> complaining!! Presumably I am not the only one unless I did something
> incorrectly during the crawl although I ran Nutch out of the box as per
> the
> tutorial. Is anyone else using Nutch in a live system that's publicly
> accessible? Is not Nutch not yet intended for such public use?
>
> FYI it seems Google and Yahoo do obey this directive.
>
> -Ed
>
>
--
http://motrech.free.fr/
http://www.frutch.org/
Re: Urgent help requested regarding Nutch obeying instructions
Posted by Jérôme Charron <je...@gmail.com>.
Nutch Html Parser support standard directives:
<meta name="robots" content="nofollow"/>
<meta name="robots" content="noindex"/>
<meta name="robots" content="none"/>
<meta name="robots" content="all"/>
noarchive is not a standard meta robots exclusion (it is a Google specific
one)
(see http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4.1.2 or
http://www.robotstxt.org/wc/meta-user.html)
Regards
Jérôme
On 1/7/06, Ed Whittaker <ed...@ewdw.com> wrote:
>
> Does Nutch obey the following instruction:
>
> <META NAME="ROBOTS" CONTENT="NOARCHIVE">
>
> It seems the crawler I ran has not obeyed them and I have a large number
> of
> cached pages that may affect page writers' copyright. Some users are
> already
> complaining!! Presumably I am not the only one unless I did something
> incorrectly during the crawl although I ran Nutch out of the box as per
> the
> tutorial. Is anyone else using Nutch in a live system that's publicly
> accessible? Is not Nutch not yet intended for such public use?
>
> FYI it seems Google and Yahoo do obey this directive.
>
> -Ed
>
>
--
http://motrech.free.fr/
http://www.frutch.org/