Can I only add url in a specified div to the fetch list with nutch?

[刘 <lc...@gmail.com>]

Such as the title, I want crawl a page with many urls, but only the ones in
a specified div are meaningful to me. So I want to write a plugin to filter
it, but I don't know which extension point should I choose.

The htmlparser filter can get the html content, but seems like process
after the "add to fetch list" operation. And the urlfilter can control the
fetch list, but I cant get the html content in it.

Look forward to any helpful replies, thx.

RE: Can I only add url in a specified div to the fetch list with nutch?

[Markus Jelsma <ma...@openindex.io>]

Hi,

Outlinks are added to the ParseData object before being passed to a HTMLParseFilter. In a HTMLParseFilter plugin you can obtain the Outlinks and remove those you don't want.

   Outlinks[] outlinks = parseResult.get(content.getUrl()).getData().getOutlinks();

Use the setOutlinks() method to write your processed list to the ParseData.

Cheers,
 
 
-----Original message-----
> From:刘?? <lc...@gmail.com>
> Sent: Fri 03-Aug-2012 15:45
> To: user@nutch.apache.org
> Subject: Can I only add url in a specified div to the fetch list with nutch?
> 
> Such as the title, I want crawl a page with many urls, but only the ones in
> a specified div are meaningful to me. So I want to write a plugin to filter
> it, but I don't know which extension point should I choose.
> 
> The htmlparser filter can get the html content, but seems like process
> after the "add to fetch list" operation. And the urlfilter can control the
> fetch list, but I cant get the html content in it.
> 
> Look forward to any helpful replies, thx.
>