Nutch Content Filtering

[mausmust <ma...@hotmail.com>]

While Apache Nutch 1.3 crawling pages, i want to analyze the content of 
the page and if the content contains some keywords then adding page for 
next steps, say indexing. If the content do not contains at least one 
key, then just getting links from that page and ignoring it. How can i 
do that? Is there any filtering plugin available for this purpose? Thnx.

RE: Nutch Content Filtering

[maus must <ma...@hotmail.com>]

Thank you much.


> Subject: RE: Nutch Content Filtering
> From: markus.jelsma@openindex.io
> To: user@nutch.apache.org
> Date: Tue, 17 Jul 2012 07:55:28 +0000
> 
> HtmlParseFilter or IndexingFilter. If you do want to parse and extract outlinks use an indexing filter to deny pages from being indexed. If you just want to throw away the whole page and it's outlinks if it does not contain your terms then implement HtmlParseFilter. See plugins for examples.
> 
>  
>  
 		 	   		  

RE: Nutch Content Filtering

[Markus Jelsma <ma...@openindex.io>]

HtmlParseFilter or IndexingFilter. If you do want to parse and extract outlinks use an indexing filter to deny pages from being indexed. If you just want to throw away the whole page and it's outlinks if it does not contain your terms then implement HtmlParseFilter. See plugins for examples.

 
 
-----Original message-----
> From:mausmust <ma...@hotmail.com>
> Sent: Tue 17-Jul-2012 09:53
> To: user@nutch.apache.org
> Subject: Re: Nutch Content Filtering
> 
> Which interface i should use for implementing?
> 
> 
> On 07/17/2012 10:45 AM, Markus Jelsma wrote:
> > Hi,
> >
> > You can create a simple parse or index filter implementation, check for words in the content and act appropriately.
> >
> > Cheers
> >   
> 
> 
> 

Re: Nutch Content Filtering

[mausmust <ma...@hotmail.com>]

Which interface i should use for implementing?


On 07/17/2012 10:45 AM, Markus Jelsma wrote:
> Hi,
>
> You can create a simple parse or index filter implementation, check for words in the content and act appropriately.
>
> Cheers
>   

RE: Nutch Content Filtering

[Markus Jelsma <ma...@openindex.io>]

Hi,

You can create a simple parse or index filter implementation, check for words in the content and act appropriately.

Cheers 
 
-----Original message-----
> From:mausmust <ma...@hotmail.com>
> Sent: Tue 17-Jul-2012 09:43
> To: user@nutch.apache.org
> Subject: Nutch Content Filtering
> 
> While Apache Nutch 1.3 crawling pages, i want to analyze the content of 
> the page and if the content contains some keywords then adding page for 
> next steps, say indexing. If the content do not contains at least one 
> key, then just getting links from that page and ignoring it. How can i 
> do that? Is there any filtering plugin available for this purpose? Thnx.
> 

Re: Nutch Content Filtering

[Alexander Chepurnoy <ku...@yahoo.com>]

I don't know about available plugin, but you can develop your own, it's quite easy. First, read this article on how to develop custom plugin [http://sujitpal.blogspot.com/2009/07/nutch-custom-plugin-to-parse-and-add.html][1] . You need for plugin like that. Check key in HtmlParseFilter extending class (if you want to check it in HTML code) or IndexingFilter extending class(if you want to check whether key is presented in "visible text", in that case you can omit HtmlParseFilter extending class). In first case, get HTML code as follows:

    String url = content.getUrl();
    Parse parse = parseResult.get(url);
    String pageText = parse.getData().toString();

then check for key in it, set boolean field to metadata, check for it on indexing stage and if not set, return null. Sample code from my project where crawler checks for some code on page:

    if(metadata.getValues(RgPluginParser.NO_INDEX).length!=0){
      LOG.debug("No code found on "+url+" page - indexing has been cancelled");
      return null;
    }


In second case, just do such a thing on indexing stage:

    String text = parse.getText();
    if(!text.contains(key)){
      return null;
    }

  [1]: http://sujitpal.blogspot.com/2009/07/nutch-custom-plugin-to-parse-and-add.html

WBR, Alexander Chepurnoy.

--- 17.7.12, mausmust <ma...@hotmail.com> :

От: mausmust <ma...@hotmail.com>
Тема: Nutch Content Filtering
Кому: user@nutch.apache.org
Дата: Вторник, 17 июль 2012, 11:41

While Apache Nutch 1.3 crawling pages, i want to analyze the content of the page and if the content contains some keywords then adding page for next steps, say indexing. If the content do not contains at least one key, then just getting links from that page and ignoring it. How can i do that? Is there any filtering plugin available for this purpose? Thnx.