You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by sprabhu_PN <sh...@pinakilabs.com> on 2009/10/21 09:47:11 UTC

Plug-ins during Nutch Crawl


We have added a few plug-ins such as date parsing plug-in that get exercised
during a Nutch crawl and update a field in each index record. Now we find
that we need to improve the plug-in and re-run it. Is the only option to
crawl the whole index once again ? Is there any way we can do a recrawl
which will just exercise newer versions of plug-ins and take less time to do
it ?

Thanks in advance.

Regards
Shreekanth Prabhu
-- 
View this message in context: http://www.nabble.com/Plug-ins-during-Nutch-Crawl-tp25987956p25987956.html
Sent from the Nutch - User mailing list archive at Nabble.com.


Re: Plug-ins during Nutch Crawl

Posted by reinhard schwab <re...@aon.at>.
if you try

bin/nutch

without any arguments and options, it will show you

Usage: nutch [-core] COMMAND
where COMMAND is one of:
...
  parse             parse a segment's pages
  invertlinks       create a linkdb from parsed segments
  index             run the indexer on parsed segments and linkdb

there is no need to redo the whole crawl.
only reparse and reindex.
may be you have to delete some of the directories in the segments you want
to reparse(i guess parse_data and parse_text)

reinhard@thord:>ls crawl/segments/20091021095928/
content  crawl_fetch  crawl_generate  crawl_parse  parse_data  parse_text

regards
reinhard

sprabhu_PN schrieb:
> We have added a few plug-ins such as date parsing plug-in that get exercised
> during a Nutch crawl and update a field in each index record. Now we find
> that we need to improve the plug-in and re-run it. Is the only option to
> crawl the whole index once again ? Is there any way we can do a recrawl
> which will just exercise newer versions of plug-ins and take less time to do
> it ?
>
> Thanks in advance.
>
> Regards
> Shreekanth Prabhu
>   


Re: Plug-ins during Nutch Crawl

Posted by Eran Zinman <zz...@gmail.com>.
Hi Shreekanth Prabhu,

I was facing the same problem.

You don't need to recrawl all the urls from scratch.

Remember - you have the segments already in your harddisk. It really depends
where you've done the date parsing. If you done in as part of the parser you
can run the parser again.

If you done it just before indexing you job is even easier - you can just
run the indexer again. Remember - each Nutch component can run as a
standalone application.

BTW - I'm really looking for a JAVA based date parser myself that can parse
dates written by humans (or website owners) - is it possible for you to
share the date parser or if you are using an open-source one, give me a
recommendation on one I can use?

Thanks,
Eran

On Wed, Oct 21, 2009 at 9:47 AM, sprabhu_PN <
shreekanth.prabhu@pinakilabs.com> wrote:

>
>
> We have added a few plug-ins such as date parsing plug-in that get
> exercised
> during a Nutch crawl and update a field in each index record. Now we find
> that we need to improve the plug-in and re-run it. Is the only option to
> crawl the whole index once again ? Is there any way we can do a recrawl
> which will just exercise newer versions of plug-ins and take less time to
> do
> it ?
>
> Thanks in advance.
>
> Regards
> Shreekanth Prabhu
> --
> View this message in context:
> http://www.nabble.com/Plug-ins-during-Nutch-Crawl-tp25987956p25987956.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>