Ignore external Links

[David Odmark <da...@moongatetech.com>]

Hi,

We are using 0.8, and I see a property called db.ignore.internal.links 
that is used by LinkDB to, well, ignore internal links. What we need is 
a runtime-switchable option that allows the opposite -something like 
db.ignore.external links. This is to say, for a given page, we don't 
want to follow any inlinks that call out to a different host from the 
containing page.

Is there any way to accomplish this without a) modifying Nutch itself, 
or b) using an URLFilter and therefore having to maintain a static 
file-based list of hosts to exclude?

Thanks in advance,

David Odmark

Re: Ignore external Links

[Jeff Ritchie <jr...@netwurklabs.com>]

David Odmark wrote:
> Hi,
>
> We are using 0.8, and I see a property called db.ignore.internal.links 
> that is used by LinkDB to, well, ignore internal links. What we need 
> is a runtime-switchable option that allows the opposite -something 
> like db.ignore.external links. This is to say, for a given page, we 
> don't want to follow any inlinks that call out to a different host 
> from the containing page.
>
> Is there any way to accomplish this without a) modifying Nutch itself, 
> or b) using an URLFilter and therefore having to maintain a static 
> file-based list of hosts to exclude?
>
> Thanks in advance,
>
> David Odmark
>
+1

That would be really useful in one of the projects I am using nutch for.

It would be a mess to implement I think...  Humm... maybe not.

LinkDb.java
line 44 insert
private boolean ignoreExternalLinks;

line 55 insert
ignoreExternalLinks = job.getBoolean("db.ignore.external.links",false);

line 82
insert something like
if(ignoreExternalLinks) {
String toHost = getHost(toUrl);
if (toHost == null || !toHost.equals(fromHost)) { // external link
continue; // skip it
}
}


Would that code be almost correct?  David you want to try that?

Jeff