You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Rushikesh K <ru...@gmail.com> on 2017/11/13 19:58:30 UTC
Removing header,Footer and left menus while crawling
Hello Sebastian,
we are most excited in using the Nutch 1.3 (with solr 6.4) for crawling
our website and we are happy with the search results but we had
requirement to skip the header footer and left menus and some other parts
of the page, can you please guide how can we exclude those parts.i was
trying various ways on google but nothing works for me.
Appreciate for your help in Advance!
--
Regards
Rushikesh M
.Net Developer
RE: Removing header,Footer and left menus while crawling
Posted by Mark Vega <ve...@uci.edu>.
Michael,
I don't know if it's compatible with v1.13, but I've been using an extractor plug-in from Bayan Group (https://github.com/BayanGroup/nutch-custom-search) with v1.10 to strip content that repeats on every page (header, footer, toc/nav) and index only the main content section into the default search field. The plug-in is easy to configure and use and allows you to specify multiple elements to remove from the indexable content by element type, id, name or css class. It also allows you to map multiple elements from different sites with different element naming/classing conventions into the same field, helpful if you've got multiple sites that each call or class their main content section something different. I've been using it without issue for about four years now.
--
Mark F. Vega
Programmer/Analyst
UC Irvine Libraries - Web Services
vegamf@uci.edu
949.824.9872
--
-----Original Message-----
From: Michael Coffey [mailto:mcoffey@yahoo.com.INVALID]
Sent: Tuesday, November 14, 2017 11:25 AM
To: user@nutch.apache.org
Cc: wastl.nagel@googlemail.com
Subject: Re: Removing header,Footer and left menus while crawling
That is a very interesting note. I have been wanting something like that. I use the python-based "newspaper" package but it is not directly compatible with the nutch/hadoop infrastructure.
From: Jorge Betancourt <be...@gmail.com>
To: user@nutch.apache.org
Cc: wastl.nagel@googlemail.com
Sent: Tuesday, November 14, 2017 5:35 AM
Subject: Re: Removing header,Footer and left menus while crawling
Hello Rushikesh,
Are you using Nutch 1.3 or Nutch 1.13? If you're using Nutch 1.13, then you could use the Tika boilerpipe implementation, on the nutch-site.xml you need to enable this feature with:
<property>
<name>tika.extractor</name>
<value>boilerpipe</value>
<description>
Which text extraction algorithm to use. Valid values are: boilerpipe or none.
</description>
</property>
And configure the proper extractor with
the tika.extractor.boilerpipe.algorithm setting.
This is not a perfect solution, but I've used it successfully in the past, of course, your results will depend on how is the structure (markup of the website).
Other option could be to implement your own parser if you need to have more control over what to include/exclude from the HTML. You can take a look at this issue https://issues.apache.org/jira/browse/NUTCH-585 which contains some info and old patches.
Best Regards,
Jorge
On Mon, Nov 13, 2017 at 8:58 PM Rushikesh K <ru...@gmail.com>
wrote:
> Hello Sebastian,
> we are most excited in using the Nutch 1.3 (with solr 6.4) for
> crawling our website and we are happy with the search results but we
> had requirement to skip the header footer and left menus and some
> other parts of the page, can you please guide how can we exclude those
> parts.i was trying various ways on google but nothing works for me.
>
> Appreciate for your help in Advance!
> --
> Regards
> Rushikesh M
> .Net Developer
>
Re: Removing header,Footer and left menus while crawling
Posted by Michael Coffey <mc...@yahoo.com.INVALID>.
That is a very interesting note. I have been wanting something like that. I use the python-based "newspaper" package but it is not directly compatible with the nutch/hadoop infrastructure.
From: Jorge Betancourt <be...@gmail.com>
To: user@nutch.apache.org
Cc: wastl.nagel@googlemail.com
Sent: Tuesday, November 14, 2017 5:35 AM
Subject: Re: Removing header,Footer and left menus while crawling
Hello Rushikesh,
Are you using Nutch 1.3 or Nutch 1.13? If you're using Nutch 1.13, then you
could use the Tika boilerpipe implementation, on the nutch-site.xml you
need to enable this feature with:
<property>
<name>tika.extractor</name>
<value>boilerpipe</value>
<description>
Which text extraction algorithm to use. Valid values are: boilerpipe or
none.
</description>
</property>
And configure the proper extractor with
the tika.extractor.boilerpipe.algorithm setting.
This is not a perfect solution, but I've used it successfully in the past,
of course, your results will depend on how is the structure (markup of the
website).
Other option could be to implement your own parser if you need to have more
control over what to include/exclude from the HTML. You can take a look at
this issue https://issues.apache.org/jira/browse/NUTCH-585 which contains
some info and old patches.
Best Regards,
Jorge
On Mon, Nov 13, 2017 at 8:58 PM Rushikesh K <ru...@gmail.com>
wrote:
> Hello Sebastian,
> we are most excited in using the Nutch 1.3 (with solr 6.4) for crawling
> our website and we are happy with the search results but we had
> requirement to skip the header footer and left menus and some other parts
> of the page, can you please guide how can we exclude those parts.i was
> trying various ways on google but nothing works for me.
>
> Appreciate for your help in Advance!
> --
> Regards
> Rushikesh M
> .Net Developer
>
Re: Removing header,Footer and left menus while crawling
Posted by Rushikesh K <ru...@gmail.com>.
Hello,
*Jorge*
Thanks for response,Sorry for confusion i am using Nutch 1.13 but also i
tried configuring Tika boilerpipe with this version but this doesn't work
for me.As you suggested to use own parser ,i am not a java developer by
chance.
By chance if you or anyone in the community has a working file ,it would be
great if you can share it because there are many people facing with this
issue (i came to know this when i googled).
Mark Vega
we also tried Bayan Group extractor plugin with Nutch 1.13 but this is also
not working.we followed the same steps.I can share the changes if you want
to take a look.
I appreciate for your quick suggestions!
Thanks
Rushikesh
On Tue, Nov 14, 2017 at 8:34 AM, Jorge Betancourt <
betancourt.jorge@gmail.com> wrote:
> Hello Rushikesh,
>
> Are you using Nutch 1.3 or Nutch 1.13? If you're using Nutch 1.13, then you
> could use the Tika boilerpipe implementation, on the nutch-site.xml you
> need to enable this feature with:
>
> <property>
> <name>tika.extractor</name>
> <value>boilerpipe</value>
> <description>
> Which text extraction algorithm to use. Valid values are: boilerpipe or
> none.
> </description>
> </property>
>
> And configure the proper extractor with
> the tika.extractor.boilerpipe.algorithm setting.
>
> This is not a perfect solution, but I've used it successfully in the past,
> of course, your results will depend on how is the structure (markup of the
> website).
>
> Other option could be to implement your own parser if you need to have more
> control over what to include/exclude from the HTML. You can take a look at
> this issue https://issues.apache.org/jira/browse/NUTCH-585 which contains
> some info and old patches.
>
> Best Regards,
> Jorge
>
> On Mon, Nov 13, 2017 at 8:58 PM Rushikesh K <ru...@gmail.com>
> wrote:
>
> > Hello Sebastian,
> > we are most excited in using the Nutch 1.3 (with solr 6.4) for crawling
> > our website and we are happy with the search results but we had
> > requirement to skip the header footer and left menus and some other parts
> > of the page, can you please guide how can we exclude those parts.i was
> > trying various ways on google but nothing works for me.
> >
> > Appreciate for your help in Advance!
> > --
> > Regards
> > Rushikesh M
> > .Net Developer
> >
>
--
Regards
Rushikesh M
.Net Developer
Re: Removing header,Footer and left menus while crawling
Posted by Jorge Betancourt <be...@gmail.com>.
Hello Rushikesh,
Are you using Nutch 1.3 or Nutch 1.13? If you're using Nutch 1.13, then you
could use the Tika boilerpipe implementation, on the nutch-site.xml you
need to enable this feature with:
<property>
<name>tika.extractor</name>
<value>boilerpipe</value>
<description>
Which text extraction algorithm to use. Valid values are: boilerpipe or
none.
</description>
</property>
And configure the proper extractor with
the tika.extractor.boilerpipe.algorithm setting.
This is not a perfect solution, but I've used it successfully in the past,
of course, your results will depend on how is the structure (markup of the
website).
Other option could be to implement your own parser if you need to have more
control over what to include/exclude from the HTML. You can take a look at
this issue https://issues.apache.org/jira/browse/NUTCH-585 which contains
some info and old patches.
Best Regards,
Jorge
On Mon, Nov 13, 2017 at 8:58 PM Rushikesh K <ru...@gmail.com>
wrote:
> Hello Sebastian,
> we are most excited in using the Nutch 1.3 (with solr 6.4) for crawling
> our website and we are happy with the search results but we had
> requirement to skip the header footer and left menus and some other parts
> of the page, can you please guide how can we exclude those parts.i was
> trying various ways on google but nothing works for me.
>
> Appreciate for your help in Advance!
> --
> Regards
> Rushikesh M
> .Net Developer
>