You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Alexis Hope <al...@cvofhope.com> on 2015/02/14 07:56:14 UTC

domain vs regexurl filter

Hi All,

I just want to clarify the differences between domain-urlfilter vs
regex-urlfilter.

Fundamentally these are the same? Both used to limit generated URLs.

If I include both in plugins.include will both be used? And if so in what
order will the be executed? Im unsure about how plugins work.

I can see regex-urlfilter will offer a little more control, especially over
subdirectories like /blog or /forum.
But domain-urlfilter offers an easier format to configure. I was wondering
if I could limit via domains then have some regex to control what paths are
accepted /posts /archive etc.

Regards,
Lex

Re: domain vs regexurl filter

Posted by Alexis Hope <al...@cvofhope.com>.
Thanks a bunch for your reply Sebastian.

That clears things up. I was unaware of the urlfilter.order property.
Good point to consider performance in filter order.

Lex.

On Sat, Feb 14, 2015 at 1:40 PM, Sebastian Nagel <wastl.nagel@googlemail.com
> wrote:

> Hi Lex,
>
> > Fundamentally these are the same? Both used to limit generated URLs.
> All kinds of URL filters are the same in this point: explicitly include
> or exclude URLs from being crawled/followed.
>
> > If I include both in plugins.include will both be used?
> Yes, both will be used.
>
> > And if so in what order will the be executed?
> See property "urlfilter.order".
> But ordering should not matter except for performance.
> It's more efficient to apply simple exclusion rules
> and fast exclusion filters (domain, prefix, suffix)
> at the beginning.
>
> > I can see regex-urlfilter will offer a little more control, especially
> over
> > subdirectories like /blog or /forum.
> Yes, regex filter is definitely the most powerful.
> But it can be slow with many rules and/or complex (long) URLs.
>
> > But domain-urlfilter offers an easier format to configure. I was
> wondering
> > if I could limit via domains then have some regex to control what paths
> are
> > accepted /posts /archive etc.
> It's probably a good way to use the filters for the tasks they are designed
> for (domains, file suffixes, etc.), esp. if the rules are more complex.
>
> Bye,
> Sebastian
>
>
> On 02/14/2015 07:56 AM, Alexis Hope wrote:
> > Hi All,
> >
> > I just want to clarify the differences between domain-urlfilter vs
> > regex-urlfilter.
> >
> > Fundamentally these are the same? Both used to limit generated URLs.
> >
> > If I include both in plugins.include will both be used? And if so in what
> > order will the be executed? Im unsure about how plugins work.
> >
> > I can see regex-urlfilter will offer a little more control, especially
> over
> > subdirectories like /blog or /forum.
> > But domain-urlfilter offers an easier format to configure. I was
> wondering
> > if I could limit via domains then have some regex to control what paths
> are
> > accepted /posts /archive etc.
> >
> > Regards,
> > Lex
> >
>
>

Re: domain vs regexurl filter

Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Lex,

> Fundamentally these are the same? Both used to limit generated URLs.
All kinds of URL filters are the same in this point: explicitly include
or exclude URLs from being crawled/followed.

> If I include both in plugins.include will both be used?
Yes, both will be used.

> And if so in what order will the be executed?
See property "urlfilter.order".
But ordering should not matter except for performance.
It's more efficient to apply simple exclusion rules
and fast exclusion filters (domain, prefix, suffix)
at the beginning.

> I can see regex-urlfilter will offer a little more control, especially over
> subdirectories like /blog or /forum.
Yes, regex filter is definitely the most powerful.
But it can be slow with many rules and/or complex (long) URLs.

> But domain-urlfilter offers an easier format to configure. I was wondering
> if I could limit via domains then have some regex to control what paths are
> accepted /posts /archive etc.
It's probably a good way to use the filters for the tasks they are designed
for (domains, file suffixes, etc.), esp. if the rules are more complex.

Bye,
Sebastian


On 02/14/2015 07:56 AM, Alexis Hope wrote:
> Hi All,
> 
> I just want to clarify the differences between domain-urlfilter vs
> regex-urlfilter.
> 
> Fundamentally these are the same? Both used to limit generated URLs.
> 
> If I include both in plugins.include will both be used? And if so in what
> order will the be executed? Im unsure about how plugins work.
> 
> I can see regex-urlfilter will offer a little more control, especially over
> subdirectories like /blog or /forum.
> But domain-urlfilter offers an easier format to configure. I was wondering
> if I could limit via domains then have some regex to control what paths are
> accepted /posts /archive etc.
> 
> Regards,
> Lex
>