You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by jotta <so...@gmail.com> on 2011/04/29 11:26:17 UTC

Getting content from crawling site's

Hi!

I have to crawl couple of sites (each of them is different). Problem is that
most of the crawled content is rubbish (page headers, menus, adverts, footer
etc).
I want to ask you in what way are you getting right content from this? Is it
a good way to parse html in own plugin and get content only from this html's
tags which I want? And then index this content to a new field.

Regards,
Jotta--
View this message in context: http://lucene.472066.n3.nabble.com/Getting-content-from-crawling-site-s-tp2878602p2878602.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Getting content from crawling site's

Posted by Julien Nioche <li...@gmail.com>.
See https://issues.apache.org/jira/browse/NUTCH-961

On 29 April 2011 15:46, Hannes Carl Meyer <ha...@googlemail.com> wrote:

> Hi,
>
> have look at boilerpipe http://code.google.com/p/boilerpipe/
>
> BR
>
> Hannes
>
> On Fri, Apr 29, 2011 at 11:26 AM, jotta <so...@gmail.com> wrote:
>
> > Hi!
> >
> > I have to crawl couple of sites (each of them is different). Problem is
> > that
> > most of the crawled content is rubbish (page headers, menus, adverts,
> > footer
> > etc).
> > I want to ask you in what way are you getting right content from this? Is
> > it
> > a good way to parse html in own plugin and get content only from this
> > html's
> > tags which I want? And then index this content to a new field.
> >
> > Regards,
> > Jotta--
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Getting-content-from-crawling-site-s-tp2878602p2878602.html
> > Sent from the Nutch - User mailing list archive at Nabble.com.
> >
>



-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com

Re: Getting content from crawling site's

Posted by Hannes Carl Meyer <ha...@googlemail.com>.
Hi,

have look at boilerpipe http://code.google.com/p/boilerpipe/

BR

Hannes

On Fri, Apr 29, 2011 at 11:26 AM, jotta <so...@gmail.com> wrote:

> Hi!
>
> I have to crawl couple of sites (each of them is different). Problem is
> that
> most of the crawled content is rubbish (page headers, menus, adverts,
> footer
> etc).
> I want to ask you in what way are you getting right content from this? Is
> it
> a good way to parse html in own plugin and get content only from this
> html's
> tags which I want? And then index this content to a new field.
>
> Regards,
> Jotta--
> View this message in context:
> http://lucene.472066.n3.nabble.com/Getting-content-from-crawling-site-s-tp2878602p2878602.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>