You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Markus Schuch <ma...@web.de> on 2017/02/25 22:02:20 UTC

[Webcrawler Connector] Feature for ignoring meta/rel robots tags/attributes

Hi,

what do you think about adding the possibility to ignore meta/rel robots
tags/attributes?

I know, such a thing is an unpolite behavior for a webcrawler, but we
already have the feature to ignore the robots.txt and for me it was
unexpected, when i configured the crawler to ignore robots.txt but it
still respected the meta/rel robots tags/attibutes.

I will open a ticket, if there are no objections.

Thanks in advance.
Markus

Re: [Webcrawler Connector] Feature for ignoring meta/rel robots tags/attributes

Posted by Markus Schuch <ma...@web.de>.
raised https://issues.apache.org/jira/browse/CONNECTORS-1392

Cheers,
Markus

Am 26.02.2017 um 00:43 schrieb Karl Wright:
> I certainly have no objection.  I would recommend, however, that the
> default setting of this configuration option be set to "follow
> metadata/rel", and that the implementation be backwards compatible.
> 
> Thanks,
> Karl
> 
> 
> On Sat, Feb 25, 2017 at 5:02 PM, Markus Schuch <markus_schuch@web.de
> <ma...@web.de>> wrote:
> 
>     Hi,
> 
>     what do you think about adding the possibility to ignore meta/rel robots
>     tags/attributes?
> 
>     I know, such a thing is an unpolite behavior for a webcrawler, but we
>     already have the feature to ignore the robots.txt and for me it was
>     unexpected, when i configured the crawler to ignore robots.txt but it
>     still respected the meta/rel robots tags/attibutes.
> 
>     I will open a ticket, if there are no objections.
> 
>     Thanks in advance.
>     Markus
> 
>

Re: [Webcrawler Connector] Feature for ignoring meta/rel robots tags/attributes

Posted by Karl Wright <da...@gmail.com>.
I certainly have no objection.  I would recommend, however, that the
default setting of this configuration option be set to "follow
metadata/rel", and that the implementation be backwards compatible.

Thanks,
Karl


On Sat, Feb 25, 2017 at 5:02 PM, Markus Schuch <ma...@web.de> wrote:

> Hi,
>
> what do you think about adding the possibility to ignore meta/rel robots
> tags/attributes?
>
> I know, such a thing is an unpolite behavior for a webcrawler, but we
> already have the feature to ignore the robots.txt and for me it was
> unexpected, when i configured the crawler to ignore robots.txt but it
> still respected the meta/rel robots tags/attibutes.
>
> I will open a ticket, if there are no objections.
>
> Thanks in advance.
> Markus
>