You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Ian M. Evans" <ia...@digitalhit.com> on 2010/02/28 18:42:09 UTC
Update on ignoring menu divs
Using Nutch as a crawler for solr.
I've been digging around the nutch-user archives a bit and have seen
some people discussing how to ignore menu items or other unnecessary div
areas like common footers, etc. I still haven't come across a full
answer yet.
Is there a to define a div by id that nutch will strip out before
tossing the content into solr?
Thanks.
Re: Update on ignoring menu divs
Posted by Ian Evans <ia...@digitalhit.com>.
On Mon, March 1, 2010 1:24 am, Sami Siren wrote:
> Andrzej Bialecki wrote:
>> On 2010-02-28 18:42, Ian M. Evans wrote:
>>> I've been digging around the nutch-user archives a bit and have seen
>>> some people discussing how to ignore menu items or other unnecessary
>>> div areas like common footers, etc. I still haven't come across a full
>>> answer yet.
>>>
>> There is no such functionality out of the box. One direction that is
>> worth pursuing would be to create an HtmlParseFilter plugin that wraps
>> the Boilerpipe library http://code.google.com/p/boilerpipe/ .
>>
> Andrzej, have you tested that lib? If the result is of decent quality it
> would be nice to have that wrapped as a plugin in Nutch.
It would be great. I imagine that for many sites with a standard template
being able to ignore menu, sidebar and footer divs would make for much
cleaner results.
I wish I could do it myself, but I'm a writer/photographer who sysadmins
by voraciously reading support lists like this. Java programming? Maybe
one day. :-)
Re: Update on ignoring menu divs
Posted by Ken Krugler <kk...@transpac.com>.
On Feb 28, 2010, at 10:24pm, Sami Siren wrote:
> Andrzej Bialecki wrote:
>> On 2010-02-28 18:42, Ian M. Evans wrote:
>>> Using Nutch as a crawler for solr.
>>>
>>> I've been digging around the nutch-user archives a bit and have seen
>>> some people discussing how to ignore menu items or other
>>> unnecessary div
>>> areas like common footers, etc. I still haven't come across a full
>>> answer yet.
>>>
>> There is no such functionality out of the box. One direction that
>> is worth pursuing would be to create an HtmlParseFilter plugin that
>> wraps the Boilerpipe library http://code.google.com/p/boilerpipe/ .
> Andrzej, have you tested that lib? If the result is of decent
> quality it would be nice to have that wrapped as a plugin in Nutch.
We've done some testing of it with general web crawls, and it seems
promising.
I've got a patch that modifies Boilerpipe so it works better as a plug-
in to Tika, so if/when that gets rolled into the source then it should
be easier to use the project with Nutch.
-- Ken
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
Re: Update on ignoring menu divs
Posted by Sami Siren <ss...@gmail.com>.
Andrzej Bialecki wrote:
> On 2010-02-28 18:42, Ian M. Evans wrote:
>> Using Nutch as a crawler for solr.
>>
>> I've been digging around the nutch-user archives a bit and have seen
>> some people discussing how to ignore menu items or other unnecessary div
>> areas like common footers, etc. I still haven't come across a full
>> answer yet.
>>
> There is no such functionality out of the box. One direction that is
> worth pursuing would be to create an HtmlParseFilter plugin that wraps
> the Boilerpipe library http://code.google.com/p/boilerpipe/ .
>
Andrzej, have you tested that lib? If the result is of decent quality it
would be nice to have that wrapped as a plugin in Nutch.
--
Sami Siren
Re: Update on ignoring menu divs
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-02-28 18:42, Ian M. Evans wrote:
> Using Nutch as a crawler for solr.
>
> I've been digging around the nutch-user archives a bit and have seen
> some people discussing how to ignore menu items or other unnecessary div
> areas like common footers, etc. I still haven't come across a full
> answer yet.
>
> Is there a to define a div by id that nutch will strip out before
> tossing the content into solr?
There is no such functionality out of the box. One direction that is
worth pursuing would be to create an HtmlParseFilter plugin that wraps
the Boilerpipe library http://code.google.com/p/boilerpipe/ .
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com