meta robots directive

[Tim Pease <ti...@gmail.com>]

Currently Nutch supports the <meta name="robots" content="noindex"> directive in the head of individual pages. I would like to extend this feature to allow the "http.agent.name" as a valid name in addition to the "robots" directive. For example, in your nutch-site.xml file if you have the property

  <property>
    <name>http.agent.name</name>
    <value>examplebot</value>
  </property>

then any pages with the meta directive in the head

  <meta name="examplebot" content="noindex,nofollow">

then this directive would cause Nutch to not index the page and to not follow links on the page.

So, the place to put this code is in the parse-html plugin and the parse-tika plugin. Both of these have an HTMLMetaProcessor class that would need to be updated.

1) Has someone already written this patch?

2) Should the property "http.robots.agents" also be included as is done in the lib-http RobotRulesParser class?


Blessings,
TwP

P.S.  There is precedence for such a change as the googlebot already parses these type of meta directives <http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=93710&from=61050&rd=1>

Re: meta robots directive

[Markus Jelsma <ma...@openindex.io>]

I see. You can write an indexing filter that drops (returns null) the document 
if it encounters noindex. I'm not sure the meta tag is by default present in 
the meta data, if not, you must either make a HTML parse filter to add the 
value of that meta tag to the meta data or try an existing meta data plugin.

> On Jul 11, 2011, at 4:22 PM, Markus Jelsma wrote:
> > Can't you use a robots.txt instead?
> 
> I have no control over the robots.txt
> 
> This is a client of ours with poor URL schemas. They would have to put 6
> million entries into their robots.txt in order to exclude our crawler.
> That leaves <meta> directives as the next logical place to exclude
> specific pages (CMS templates make this easy).
> 
> I would love to not have this problem. But alas, since the client's URL
> schema is very poor we cannot effectively use robots.txt or any of the
> regular expression filters.
> 
> I am comfortable patching parse-html for our needs. Was just wondering if
> someone else has done this and would it be a worthwhile contribution back
> to the Nutch community at large.
> 
> Blessings,
> TwP
> 
> >> Currently Nutch supports the > directive in the head of individual
> >> pages. I
> > 
> > would like to extend this > feature to allow the "http.agent.name" as a
> > valid name in addition to the > "robots" directive. For example, in your
> > nutch- site.xml file if you have > the property > > > http.agent.name >
> > examplebot > >
> > 
> >> then any pages with the meta directive in the head > > > > then this
> > 
> > directive would cause Nutch to not index the page and to not > follow
> > links on the page. > > So, the place to put this code is in the
> > parse-html plugin and the > parse-tika plugin. Both of these have an
> > HTMLMetaProcessor class that > would need to be updated. > > 1) Has
> > someone already written this patch? > > 2) Should the property
> > "http.robots.agents" also be included as is done in > the lib-http
> > RobotRulesParser class? > > > Blessings, > TwP > > P.S. There is
> > precedence for such a change as the googlebot already parses > these
> > type of meta directives > &from=61050&rd=1>

Re: meta robots directive

[Tim Pease <ti...@gmail.com>]

On Jul 11, 2011, at 4:22 PM, Markus Jelsma wrote:

> Can't you use a robots.txt instead?
> 

I have no control over the robots.txt

This is a client of ours with poor URL schemas. They would have to put 6 million entries into their robots.txt in order to exclude our crawler. That leaves <meta> directives as the next logical place to exclude specific pages (CMS templates make this easy).

I would love to not have this problem. But alas, since the client's URL schema is very poor we cannot effectively use robots.txt or any of the regular expression filters.

I am comfortable patching parse-html for our needs. Was just wondering if someone else has done this and would it be a worthwhile contribution back to the Nutch community at large.

Blessings,
TwP


>> Currently Nutch supports the > directive in the head of individual pages. I 
> would like to extend this > feature to allow the "http.agent.name" as a valid 
> name in addition to the > "robots" directive. For example, in your nutch-
> site.xml file if you have > the property > > > http.agent.name > examplebot > > 
>> then any pages with the meta directive in the head > > > > then this 
> directive would cause Nutch to not index the page and to not > follow links on 
> the page. > > So, the place to put this code is in the parse-html plugin and 
> the > parse-tika plugin. Both of these have an HTMLMetaProcessor class that > 
> would need to be updated. > > 1) Has someone already written this patch? > > 
> 2) Should the property "http.robots.agents" also be included as is done in > 
> the lib-http RobotRulesParser class? > > > Blessings, > TwP > > P.S. There is 
> precedence for such a change as the googlebot already parses > these type of 
> meta directives > &from=61050&rd=1>

Re: meta robots directive

[Markus Jelsma <ma...@openindex.io>]

Can't you use a robots.txt instead?

> Currently Nutch supports the > directive in the head of individual pages. I 
would like to extend this > feature to allow the "http.agent.name" as a valid 
name in addition to the > "robots" directive. For example, in your nutch-
site.xml file if you have > the property > > > http.agent.name > examplebot > > 
> then any pages with the meta directive in the head > > > > then this 
directive would cause Nutch to not index the page and to not > follow links on 
the page. > > So, the place to put this code is in the parse-html plugin and 
the > parse-tika plugin. Both of these have an HTMLMetaProcessor class that > 
would need to be updated. > > 1) Has someone already written this patch? > > 
2) Should the property "http.robots.agents" also be included as is done in > 
the lib-http RobotRulesParser class? > > > Blessings, > TwP > > P.S. There is 
precedence for such a change as the googlebot already parses > these type of 
meta directives > &from=61050&rd=1>