You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Michael Lissner <ml...@michaeljaylissner.com> on 2012/01/22 01:01:26 UTC

Support for x-robots-tag

Hi,

I'm doing some research on what technologies various crawlers support 
for crawl exclusion. Without installing and figuring out Nutch, I can't 
figure out if it supports the x-robots-tag HTTP header?

Does anybody know?

Thanks,

Mike

Re: Support for x-robots-tag

Posted by Michael Lissner <ml...@michaeljaylissner.com>.

OK, issue created: https://issues.apache.org/jira/browse/NUTCH-1257

Thanks again.

Mike

On 01/24/2012 11:44 PM, Markus Jelsma wrote:
> Not that i'm aware of. You can create an issue in Jira if you like and provide
> a patch if possible.
>
>> Thanks for the follow up. Very good to know. Seems like this would be
>> pretty easy to add, given that the robots meta tag is supported.
>>
>> Are there any plans to do so?
>>
>> Mike
>>
>> On Mon 23 Jan 2012 02:27:02 AM PST, Markus Jelsma wrote:
>>> There is currently no built-in support for the x-robots-tag header.
>>>
>>> On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
>>>> Hi,
>>>>
>>>> I'm doing some research on what technologies various crawlers support
>>>> for crawl exclusion. Without installing and figuring out Nutch, I can't
>>>> figure out if it supports the x-robots-tag HTTP header?
>>>>
>>>> Does anybody know?
>>>>
>>>> Thanks,
>>>>
>>>> Mike

Re: Support for x-robots-tag

Posted by Markus Jelsma <ma...@openindex.io>.

Not that i'm aware of. You can create an issue in Jira if you like and provide 
a patch if possible.

> Thanks for the follow up. Very good to know. Seems like this would be
> pretty easy to add, given that the robots meta tag is supported.
> 
> Are there any plans to do so?
> 
> Mike
> 
> On Mon 23 Jan 2012 02:27:02 AM PST, Markus Jelsma wrote:
> > There is currently no built-in support for the x-robots-tag header.
> > 
> > On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
> >> Hi,
> >> 
> >> I'm doing some research on what technologies various crawlers support
> >> for crawl exclusion. Without installing and figuring out Nutch, I can't
> >> figure out if it supports the x-robots-tag HTTP header?
> >> 
> >> Does anybody know?
> >> 
> >> Thanks,
> >> 
> >> Mike

Re: Support for x-robots-tag

Posted by Michael Lissner <ml...@michaeljaylissner.com>.

Thanks for the follow up. Very good to know. Seems like this would be 
pretty easy to add, given that the robots meta tag is supported.

Are there any plans to do so? 

Mike

On Mon 23 Jan 2012 02:27:02 AM PST, Markus Jelsma wrote:
> There is currently no built-in support for the x-robots-tag header.
>
> On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
>> Hi,
>>
>> I'm doing some research on what technologies various crawlers support
>> for crawl exclusion. Without installing and figuring out Nutch, I can't
>> figure out if it supports the x-robots-tag HTTP header?
>>
>> Does anybody know?
>>
>> Thanks,
>>
>> Mike
>

Re: Support for x-robots-tag

Posted by Markus Jelsma <ma...@openindex.io>.

There is currently no built-in support for the x-robots-tag header.

On Sunday 22 January 2012 01:01:26 Michael Lissner wrote:
> Hi,
> 
> I'm doing some research on what technologies various crawlers support
> for crawl exclusion. Without installing and figuring out Nutch, I can't
> figure out if it supports the x-robots-tag HTTP header?
> 
> Does anybody know?
> 
> Thanks,
> 
> Mike

-- 
Markus Jelsma - CTO - Openindex