URL filter plugins for nutch

[Madan Patil <ma...@usc.edu>]

Hi,

I am working on assignment where I am supposed to use nutch to crawl
antractic data.
I am writing a plugin which extends URLFilter to not crawl duplicate (exact
and near duplicate) URLs. All the plugins, the defaults ones and others on
web, have only one URL. They decide what to do or not to do based on
content of one URL.

Could any one point me to resources which would help me compare content of
one URL with the ones already crawled?

Thanks in advance.

Regards,
Madan Patil

Re: [MASSMAIL]URL filter plugins for nutch

[Jorge Luis Betancourt González <jl...@uci.cu>]

The idea behind the URL filter plugins is to decide weather the current URL (string) should be allowed to be fetched or not, in your particular case I think that you could try to read the LinkDB and then decide if you want to fetch or not this particular URL, keep in mind that this logic should be something fast, because is going to be executed a lot of times (one for each URL). 

I don't know of any plugin that does this, typically this is kind of hard to do right (if possible at all), but you can check out the LinkDbReader for a way to read from LinkDB to do your check. One more detail if you only filter by the URL you can find resources on the Web where the content has changed and in this case you will discard the fetching of the resource. 

Regards,

----- Original Message -----
From: "Madan Patil" <ma...@usc.edu>
To: "user" <us...@nutch.apache.org>
Sent: Wednesday, February 18, 2015 3:09:00 PM
Subject: [MASSMAIL]URL filter plugins for nutch

Hi,

I am working on assignment where I am supposed to use nutch to crawl
antractic data.
I am writing a plugin which extends URLFilter to not crawl duplicate (exact
and near duplicate) URLs. All the plugins, the defaults ones and others on
web, have only one URL. They decide what to do or not to do based on
content of one URL.

Could any one point me to resources which would help me compare content of
one URL with the ones already crawled?

Thanks in advance.

Regards,
Madan Patil