You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by kiran chitturi <ch...@gmail.com> on 2012/11/18 18:32:48 UTC
Best practices for running Nutch
Hi!
I have been running crawls using Nutch for 13000 documents (protocol http)
on a single machine and it goes on to take 2-3 days to get finished. I am
using 2.x version of Nutch.
I use a depth of 20 and topN of 1000 (2000) when i initiate the 'sh
bin/nutch crawl -depth 20 -topN 1000'.
I keep running in to Exceptions after one day. Sometimes its
- Memory Exception : Heap Space (after the parsing of the documents)
- Mysql Connection Error (because the crawler went on to fetch 10,000
documents after the command 'sh bin/nutch crawl -continue -depth 10 -topN
700' as the crawl failed because
I increased the heap space and increased the timeout.
I am wondering what are the best practices to run Nutch crawls. Is a full
crawl a good thing to do or should i do it in steps (generate, fetch,
parse, updatedb) ?
Also how do i choose the value of the parameters, even if i give topN as
700 the fetcher goes to fetch 3000 documents. What parameters have high
impact on the running time of the crawl ?
All these options might be system based and need not have general values
which work for everyone.
I am wondering what are things that Nutch Users and Developers follow here
when running big crawls ?
Some of the exceptions come after 1 or 2 days of running the crawler, so
its getting hard to know how to fix them before hand. Are there any common
exceptions that Nutch can run in to frequently ?
Is there any documentation for Nutch practices ? I have seen people crawls
go for a long time because of the filtering sometimes.
Sorry for the long email.
Thank you,
--
Kiran Chitturi
RE: Best practices for running Nutch
Posted by Markus Jelsma <ma...@openindex.io>.
A long running fetcher:
- allows possible memory leak to accumulate until disaster;
- looses more records if it terminates;
- can run even longer because more records to shuffle in map reduce;
- is harder to debug, log files can become too large to easily grep through
- is more likely to have a hanging fetcher thread;
- ....
The fetcher.timelimit and throughput settings are your friend. There is no problem in terminating a fetcher gracefully via these settings, just update and generate again and unfetched URL's will be fetched eventually.
The downside is that you have to update and index more often and therefore read the crawldb more often.
-----Original message-----
> From:Manu Reddy <ma...@gmail.com>
> Sent: Fri 14-Dec-2012 18:15
> To: user@nutch.apache.org
> Subject: Re: Best practices for running Nutch
>
> Could you tell us some problems that can arise if we run the fetcher for
> long?? Can you guide to us some place where we can find more information on
> the same.
>
> Thanks in advance.
> Manu
>
>
>
>
> On Mon, Nov 19, 2012 at 8:08 PM, kiran chitturi
> <ch...@gmail.com>wrote:
>
> > Hi
> >
> > I'm not sure this applies to you because i don't know what you mean by
> > > `running crawler`; never run the fetcher for longer than an hour orso.
> > >
> >
> > Thank you for the reply. I have seen fetcher run for more time like 3-5
> > hours and more.
> >
> > I think based on both of your suggestions, i will try to see if i can
> > switch my database and not to use the crawl class but use the crawl
> > script.
> >
> > I will start a new script and try to see if it makes any changes in the
> > performance.
> >
> > Thank you,
> > Kiran.
> >
> > --
> > Kiran Chitturi
> >
>
>
>
> --
> Manu Reddy,
> mobile no: 091-8143405225,
> Hyderabad.
>
Re: Best practices for running Nutch
Posted by Manu Reddy <ma...@gmail.com>.
Could you tell us some problems that can arise if we run the fetcher for
long?? Can you guide to us some place where we can find more information on
the same.
Thanks in advance.
Manu
On Mon, Nov 19, 2012 at 8:08 PM, kiran chitturi
<ch...@gmail.com>wrote:
> Hi
>
> I'm not sure this applies to you because i don't know what you mean by
> > `running crawler`; never run the fetcher for longer than an hour orso.
> >
>
> Thank you for the reply. I have seen fetcher run for more time like 3-5
> hours and more.
>
> I think based on both of your suggestions, i will try to see if i can
> switch my database and not to use the crawl class but use the crawl
> script.
>
> I will start a new script and try to see if it makes any changes in the
> performance.
>
> Thank you,
> Kiran.
>
> --
> Kiran Chitturi
>
--
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.
Re: Best practices for running Nutch
Posted by kiran chitturi <ch...@gmail.com>.
Hi
I'm not sure this applies to you because i don't know what you mean by
> `running crawler`; never run the fetcher for longer than an hour orso.
>
Thank you for the reply. I have seen fetcher run for more time like 3-5
hours and more.
I think based on both of your suggestions, i will try to see if i can
switch my database and not to use the crawl class but use the crawl
script.
I will start a new script and try to see if it makes any changes in the
performance.
Thank you,
Kiran.
--
Kiran Chitturi
Re: Best practices for running Nutch
Posted by Julien Nioche <li...@gmail.com>.
>
> > I use a depth of 20 and topN of 1000 (2000) when i initiate the 'sh
> > bin/nutch crawl -depth 20 -topN 1000'.
> >
> > I keep running in to Exceptions after one day. Sometimes its
> >
> >
> > - Memory Exception : Heap Space (after the parsing of the documents)
>
> After parsing the documents? That should be during updatedb but are you
> sure? That job hardly ever runs out of memory.
>
The crawl class is deprecated and you should use the crawl script instead
or write your own script to call the commands individually. The crawl class
can indeed have memory issues with runaway parse threads (i.e. time outs).
This is mentioned on this list on a regular basis.
>
> > - Mysql Connection Error (because the crawler went on to fetch 10,000
> > documents after the command 'sh bin/nutch crawl -continue -depth 10
> -topN
> > 700' as the crawl failed because
> >
>
We have numerous bugs filed in JIRA for the SQL backend. My advice would be
to use a more stable one like HBase
>
> >
> > Also how do i choose the value of the parameters, even if i give topN as
> > 700 the fetcher goes to fetch 3000 documents. What parameters have high
> > impact on the running time of the crawl ?
>
> Are you sure? The generator (at least in trunk) honors the topN parameter
> and will not generate more than specified. Keep in mind that using the
> crawl script and the depth parameter you're multiplying topN by depth.
>
See my comment above. The SQL backend should be considered broken
Julien
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
RE: Best practices for running Nutch
Posted by Markus Jelsma <ma...@openindex.io>.
Hi
-----Original message-----
> From:kiran chitturi <ch...@gmail.com>
> Sent: Sun 18-Nov-2012 18:38
> To: user@nutch.apache.org
> Subject: Best practices for running Nutch
>
> Hi!
>
> I have been running crawls using Nutch for 13000 documents (protocol http)
> on a single machine and it goes on to take 2-3 days to get finished. I am
> using 2.x version of Nutch.
>
> I use a depth of 20 and topN of 1000 (2000) when i initiate the 'sh
> bin/nutch crawl -depth 20 -topN 1000'.
>
> I keep running in to Exceptions after one day. Sometimes its
>
>
> - Memory Exception : Heap Space (after the parsing of the documents)
After parsing the documents? That should be during updatedb but are you sure? That job hardly ever runs out of memory.
> - Mysql Connection Error (because the crawler went on to fetch 10,000
> documents after the command 'sh bin/nutch crawl -continue -depth 10 -topN
> 700' as the crawl failed because
>
> I increased the heap space and increased the timeout.
>
> I am wondering what are the best practices to run Nutch crawls. Is a full
> crawl a good thing to do or should i do it in steps (generate, fetch,
> parse, updatedb) ?
Separate steps are good for debugging and give you more control.
>
> Also how do i choose the value of the parameters, even if i give topN as
> 700 the fetcher goes to fetch 3000 documents. What parameters have high
> impact on the running time of the crawl ?
Are you sure? The generator (at least in trunk) honors the topN parameter and will not generate more than specified. Keep in mind that using the crawl script and the depth parameter you're multiplying topN by depth.
>
> All these options might be system based and need not have general values
> which work for everyone.
>
> I am wondering what are things that Nutch Users and Developers follow here
> when running big crawls ?
What is a big crawl? 13.000 documents are very easy to manage on a very small machine running locally. If you're downloading from one or a few hosts it's expected to take a very long time due to crawler politeness, don't download faster than one page every > 5 seconds unless you're allowed to or own the host. If you own a host or are allowed to you can increase or increase the number of threads per queue (host, domain or IP).
>
> Some of the exceptions come after 1 or 2 days of running the crawler, so
> its getting hard to know how to fix them before hand.
I'm not sure this applies to you because i don't know what you mean by `running crawler`; never run the fetcher for longer than an hour orso.
> Are there any common exceptions that Nutch can run in to frequently ?
The usual exceptions are network errors.
>
> Is there any documentation for Nutch practices ? I have seen people crawls
> go for a long time because of the filtering sometimes.
I'm not sure but the best thing to do on this list is not talk about crawl (e.g. my crawl fails or takes too long) but to talk about the separate jobs. We don't know what's wrong if one tells us a crawl is taking long because it consists of the separate steps.
>
> Sorry for the long email.
>
> Thank you,
> --
> Kiran Chitturi
>
Re: Best practices for running Nutch
Posted by Muzaffer Tolga Özses <to...@ozses.net>.
I'd also be very much interested in knowing these!
On 11/18/2012 07:32 PM, kiran chitturi wrote:
> Hi!
>
> I have been running crawls using Nutch for 13000 documents (protocol http)
> on a single machine and it goes on to take 2-3 days to get finished. I am
> using 2.x version of Nutch.
>
> I use a depth of 20 and topN of 1000 (2000) when i initiate the 'sh
> bin/nutch crawl -depth 20 -topN 1000'.
>
> I keep running in to Exceptions after one day. Sometimes its
>
>
> - Memory Exception : Heap Space (after the parsing of the documents)
> - Mysql Connection Error (because the crawler went on to fetch 10,000
> documents after the command 'sh bin/nutch crawl -continue -depth 10 -topN
> 700' as the crawl failed because
>
> I increased the heap space and increased the timeout.
>
> I am wondering what are the best practices to run Nutch crawls. Is a full
> crawl a good thing to do or should i do it in steps (generate, fetch,
> parse, updatedb) ?
>
> Also how do i choose the value of the parameters, even if i give topN as
> 700 the fetcher goes to fetch 3000 documents. What parameters have high
> impact on the running time of the crawl ?
>
> All these options might be system based and need not have general values
> which work for everyone.
>
> I am wondering what are things that Nutch Users and Developers follow here
> when running big crawls ?
>
> Some of the exceptions come after 1 or 2 days of running the crawler, so
> its getting hard to know how to fix them before hand. Are there any common
> exceptions that Nutch can run in to frequently ?
>
> Is there any documentation for Nutch practices ? I have seen people crawls
> go for a long time because of the filtering sometimes.
>
> Sorry for the long email.
>
> Thank you,