You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by sprabhu_PN <sh...@pinakilabs.com> on 2009/10/21 09:47:11 UTC
Plug-ins during Nutch Crawl
We have added a few plug-ins such as date parsing plug-in that get exercised
during a Nutch crawl and update a field in each index record. Now we find
that we need to improve the plug-in and re-run it. Is the only option to
crawl the whole index once again ? Is there any way we can do a recrawl
which will just exercise newer versions of plug-ins and take less time to do
it ?
Thanks in advance.
Regards
Shreekanth Prabhu
--
View this message in context: http://www.nabble.com/Plug-ins-during-Nutch-Crawl-tp25987956p25987956.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Plug-ins during Nutch Crawl
Posted by reinhard schwab <re...@aon.at>.
if you try
bin/nutch
without any arguments and options, it will show you
Usage: nutch [-core] COMMAND
where COMMAND is one of:
...
parse parse a segment's pages
invertlinks create a linkdb from parsed segments
index run the indexer on parsed segments and linkdb
there is no need to redo the whole crawl.
only reparse and reindex.
may be you have to delete some of the directories in the segments you want
to reparse(i guess parse_data and parse_text)
reinhard@thord:>ls crawl/segments/20091021095928/
content crawl_fetch crawl_generate crawl_parse parse_data parse_text
regards
reinhard
sprabhu_PN schrieb:
> We have added a few plug-ins such as date parsing plug-in that get exercised
> during a Nutch crawl and update a field in each index record. Now we find
> that we need to improve the plug-in and re-run it. Is the only option to
> crawl the whole index once again ? Is there any way we can do a recrawl
> which will just exercise newer versions of plug-ins and take less time to do
> it ?
>
> Thanks in advance.
>
> Regards
> Shreekanth Prabhu
>
Re: Plug-ins during Nutch Crawl
Posted by Eran Zinman <zz...@gmail.com>.
Hi Shreekanth Prabhu,
I was facing the same problem.
You don't need to recrawl all the urls from scratch.
Remember - you have the segments already in your harddisk. It really depends
where you've done the date parsing. If you done in as part of the parser you
can run the parser again.
If you done it just before indexing you job is even easier - you can just
run the indexer again. Remember - each Nutch component can run as a
standalone application.
BTW - I'm really looking for a JAVA based date parser myself that can parse
dates written by humans (or website owners) - is it possible for you to
share the date parser or if you are using an open-source one, give me a
recommendation on one I can use?
Thanks,
Eran
On Wed, Oct 21, 2009 at 9:47 AM, sprabhu_PN <
shreekanth.prabhu@pinakilabs.com> wrote:
>
>
> We have added a few plug-ins such as date parsing plug-in that get
> exercised
> during a Nutch crawl and update a field in each index record. Now we find
> that we need to improve the plug-in and re-run it. Is the only option to
> crawl the whole index once again ? Is there any way we can do a recrawl
> which will just exercise newer versions of plug-ins and take less time to
> do
> it ?
>
> Thanks in advance.
>
> Regards
> Shreekanth Prabhu
> --
> View this message in context:
> http://www.nabble.com/Plug-ins-during-Nutch-Crawl-tp25987956p25987956.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>