You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Law-Firms-In.com" <we...@law-firms-in.com> on 2013/12/28 20:38:48 UTC
nutch 2.2.1/hbase performance
Hello all,
I wanted to inquiry about the general performance of nutch. I have seen
this page here
(http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
where it takes
78minutes
for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
I have myself the same setup as in the test but with currently only
around 70k urls in the database.
The steps fetch/parse go very quick but the steps generate/update take
both _forever_. I have for 1 run about 12 hours and by far the most time
is spent at update followed by generate.
Is there ANYTHING I can do to speedup the process? I have a strong
dedicated server with 52GB RAM. One thing I notice is that during
generate/update ALL available RAM is used (Mem: 52438M total,
52267M used, 170M free, 191M buffers).
I am thankful for any help/feedback!
Domi
RE: nutch retries
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - db.fetch.retry.max only sets status DB_GONE is retry value exceeds it. It really doens't too that much. You can use a custom scheduler or set db.gone.interval.max to true. This is only for Nutch 1.8 if i remember correctly.
-----Original message-----
> From:Martin Aesch <ma...@googlemail.com>
> Sent: Sunday 29th December 2013 17:57
> To: user@nutch.apache.org
> Subject: nutch retries
>
> Dear nutchers,
>
> below is an output of nutch-1.7 readdb -stats. Why is this retry count going so high?
>
> In nutch-default, there is db.fetch.retry.max set to 3. I did not overwrite this property. Anything I missed?
>
> Thanks,
> Martin
>
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: Statistics for CrawlDb: crawl/crawldb
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: TOTAL urls: 222298055
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 0: 221451536
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 1: 393954
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 10: 13831
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 11: 13833
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 12: 13615
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 13: 13981
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 14: 13649
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 15: 14691
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 16: 14549
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 17: 32747
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 18: 6356
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 2: 111174
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 3: 62275
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 4: 46550
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 5: 35149
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 6: 17968
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 7: 15727
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 8: 13339
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 9: 13131
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: min score: 0.0
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: avg score: 0.037087735
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: max score: 587.999
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 1 (db_unfetched): 158627810
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 2 (db_fetched): 58450261
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 3 (db_gone): 2755726
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 4 (db_redir_temp): 889557
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 5 (db_redir_perm): 1574578
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 6 (db_notmodified): 123
> 13/12/29 15:24:35 INFO crawl.CrawlDbReader: CrawlDb statistics: done
>
>
>
Re: nutch 2.2.1/hbase performance
Posted by "Law-Firms-In.com" <we...@law-firms-in.com>.
Actually the answer is for Talat. Sorry for the confusion.
On 12/29/2013 02:31 AM, Law-Firms-In.com wrote:
> Hi Yazdi,
>
> thank you for your reply.
>
> I am running on a single server currently (1 server).
>
> In my hbase (hbase-0.90.6/conf) folder I see the following files:
>
> hadoop-metrics.properties
> hbase-env.sh
> hbase-site.xml
> log4j.properties
> regionservers
>
> I dont see the files you ask for. Please let me know what info might
> help you pinpoint the problem.
>
> Greetings,
>
> Domi
>
>
> On 12/28/2013 10:42 PM, Talat Uyarer wrote:
>> Hi,
>>
>> May i ask you do you run on local mode with one server. If you will say no.
>> Can you share with me your mapred-site.xml or you can share job.xml options
>> with your cluster infrastructer.
>>
>> Thanks
>> Talat
>> 28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <we...@law-firms-in.com>
>> yazdı:
>>
>>> Hello all,
>>>
>>> I wanted to inquiry about the general performance of nutch. I have seen
>>> this page here
>>> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
>>> where it takes
>>>
>>> 78minutes
>>>
>>> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
>>>
>>> I have myself the same setup as in the test but with currently only
>>> around 70k urls in the database.
>>>
>>> The steps fetch/parse go very quick but the steps generate/update take
>>> both _forever_. I have for 1 run about 12 hours and by far the most time
>>> is spent at update followed by generate.
>>>
>>> Is there ANYTHING I can do to speedup the process? I have a strong
>>> dedicated server with 52GB RAM. One thing I notice is that during
>>> generate/update ALL available RAM is used (Mem: 52438M total,
>>> 52267M used, 170M free, 191M buffers).
>>>
>>> I am thankful for any help/feedback!
>>>
>>> Domi
>>>
>>>
>>>
nutch retries
Posted by Martin Aesch <ma...@googlemail.com>.
Dear nutchers,
below is an output of nutch-1.7 readdb -stats. Why is this retry count going so high?
In nutch-default, there is db.fetch.retry.max set to 3. I did not overwrite this property. Anything I missed?
Thanks,
Martin
13/12/29 15:24:35 INFO crawl.CrawlDbReader: Statistics for CrawlDb: crawl/crawldb
13/12/29 15:24:35 INFO crawl.CrawlDbReader: TOTAL urls: 222298055
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 0: 221451536
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 1: 393954
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 10: 13831
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 11: 13833
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 12: 13615
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 13: 13981
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 14: 13649
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 15: 14691
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 16: 14549
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 17: 32747
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 18: 6356
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 2: 111174
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 3: 62275
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 4: 46550
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 5: 35149
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 6: 17968
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 7: 15727
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 8: 13339
13/12/29 15:24:35 INFO crawl.CrawlDbReader: retry 9: 13131
13/12/29 15:24:35 INFO crawl.CrawlDbReader: min score: 0.0
13/12/29 15:24:35 INFO crawl.CrawlDbReader: avg score: 0.037087735
13/12/29 15:24:35 INFO crawl.CrawlDbReader: max score: 587.999
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 1 (db_unfetched): 158627810
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 2 (db_fetched): 58450261
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 3 (db_gone): 2755726
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 4 (db_redir_temp): 889557
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 5 (db_redir_perm): 1574578
13/12/29 15:24:35 INFO crawl.CrawlDbReader: status 6 (db_notmodified): 123
13/12/29 15:24:35 INFO crawl.CrawlDbReader: CrawlDb statistics: done
have a happy new year everyone
Posted by kaveh minooie <ka...@plutoz.com>.
I hope everyone would take a minute from reading all the jira updates
and join me to wish every body a very happy new year :)
Re: nutch 2.2.1/hbase performance
Posted by Talat Uyarer <ta...@uyarer.com>.
Hi Domi,
If you think your subject will not be related with user list yes we can.
Talat
30 Ara 2013 13:17 tarihinde "Law-Firms-In.com" <we...@law-firms-in.com>
yazdı:
> Hi Talat,
>
> is there a way I can contact you outside of this list?
>
> Thank you,
>
> Domi
>
>
> On 12/29/2013 04:32 PM, Talat Uyarer wrote:
> > Hi again,
> >
> > In document, They made configuration maximum of 2 Mappers and 2 Reducers
> > available. If you use local mode you can only run 1 Map and 1 Reduce
> slot.
> > You should install hadoop for mapreduce calculating or you can use apache
> > whirr for instalation. But your main problem is running style. You should
> > run deploy mod.
> >
> > Talat
> >
> >
> > 2013/12/29 Law-Firms-In.com <we...@law-firms-in.com>
> >
> >> Hi Yazdi,
> >>
> >> thank you for your reply.
> >>
> >> I am running on a single server currently (1 server).
> >>
> >> In my hbase (hbase-0.90.6/conf) folder I see the following files:
> >>
> >> hadoop-metrics.properties
> >> hbase-env.sh
> >> hbase-site.xml
> >> log4j.properties
> >> regionservers
> >>
> >> I dont see the files you ask for. Please let me know what info might
> >> help you pinpoint the problem.
> >>
> >> Greetings,
> >>
> >> Domi
> >>
> >>
> >> On 12/28/2013 10:42 PM, Talat Uyarer wrote:
> >>> Hi,
> >>>
> >>> May i ask you do you run on local mode with one server. If you will say
> >> no.
> >>> Can you share with me your mapred-site.xml or you can share job.xml
> >> options
> >>> with your cluster infrastructer.
> >>>
> >>> Thanks
> >>> Talat
> >>> 28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <
> >> webmaster@law-firms-in.com>
> >>> yazdı:
> >>>
> >>>> Hello all,
> >>>>
> >>>> I wanted to inquiry about the general performance of nutch. I have
> seen
> >>>> this page here
> >>>> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
> >>>> where it takes
> >>>>
> >>>> 78minutes
> >>>>
> >>>> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
> >>>>
> >>>> I have myself the same setup as in the test but with currently only
> >>>> around 70k urls in the database.
> >>>>
> >>>> The steps fetch/parse go very quick but the steps generate/update take
> >>>> both _forever_. I have for 1 run about 12 hours and by far the most
> time
> >>>> is spent at update followed by generate.
> >>>>
> >>>> Is there ANYTHING I can do to speedup the process? I have a strong
> >>>> dedicated server with 52GB RAM. One thing I notice is that during
> >>>> generate/update ALL available RAM is used (Mem: 52438M total,
> >>>> 52267M used, 170M free, 191M buffers).
> >>>>
> >>>> I am thankful for any help/feedback!
> >>>>
> >>>> Domi
> >>>>
> >>>>
> >>>>
> >>
> >
>
>
Re: nutch 2.2.1/hbase performance
Posted by "Law-Firms-In.com" <we...@law-firms-in.com>.
Hi Talat,
is there a way I can contact you outside of this list?
Thank you,
Domi
On 12/29/2013 04:32 PM, Talat Uyarer wrote:
> Hi again,
>
> In document, They made configuration maximum of 2 Mappers and 2 Reducers
> available. If you use local mode you can only run 1 Map and 1 Reduce slot.
> You should install hadoop for mapreduce calculating or you can use apache
> whirr for instalation. But your main problem is running style. You should
> run deploy mod.
>
> Talat
>
>
> 2013/12/29 Law-Firms-In.com <we...@law-firms-in.com>
>
>> Hi Yazdi,
>>
>> thank you for your reply.
>>
>> I am running on a single server currently (1 server).
>>
>> In my hbase (hbase-0.90.6/conf) folder I see the following files:
>>
>> hadoop-metrics.properties
>> hbase-env.sh
>> hbase-site.xml
>> log4j.properties
>> regionservers
>>
>> I dont see the files you ask for. Please let me know what info might
>> help you pinpoint the problem.
>>
>> Greetings,
>>
>> Domi
>>
>>
>> On 12/28/2013 10:42 PM, Talat Uyarer wrote:
>>> Hi,
>>>
>>> May i ask you do you run on local mode with one server. If you will say
>> no.
>>> Can you share with me your mapred-site.xml or you can share job.xml
>> options
>>> with your cluster infrastructer.
>>>
>>> Thanks
>>> Talat
>>> 28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <
>> webmaster@law-firms-in.com>
>>> yazdı:
>>>
>>>> Hello all,
>>>>
>>>> I wanted to inquiry about the general performance of nutch. I have seen
>>>> this page here
>>>> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
>>>> where it takes
>>>>
>>>> 78minutes
>>>>
>>>> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
>>>>
>>>> I have myself the same setup as in the test but with currently only
>>>> around 70k urls in the database.
>>>>
>>>> The steps fetch/parse go very quick but the steps generate/update take
>>>> both _forever_. I have for 1 run about 12 hours and by far the most time
>>>> is spent at update followed by generate.
>>>>
>>>> Is there ANYTHING I can do to speedup the process? I have a strong
>>>> dedicated server with 52GB RAM. One thing I notice is that during
>>>> generate/update ALL available RAM is used (Mem: 52438M total,
>>>> 52267M used, 170M free, 191M buffers).
>>>>
>>>> I am thankful for any help/feedback!
>>>>
>>>> Domi
>>>>
>>>>
>>>>
>>
>
Re: nutch 2.2.1/hbase performance
Posted by Talat Uyarer <ta...@uyarer.com>.
Hi again,
In document, They made configuration maximum of 2 Mappers and 2 Reducers
available. If you use local mode you can only run 1 Map and 1 Reduce slot.
You should install hadoop for mapreduce calculating or you can use apache
whirr for instalation. But your main problem is running style. You should
run deploy mod.
Talat
2013/12/29 Law-Firms-In.com <we...@law-firms-in.com>
> Hi Yazdi,
>
> thank you for your reply.
>
> I am running on a single server currently (1 server).
>
> In my hbase (hbase-0.90.6/conf) folder I see the following files:
>
> hadoop-metrics.properties
> hbase-env.sh
> hbase-site.xml
> log4j.properties
> regionservers
>
> I dont see the files you ask for. Please let me know what info might
> help you pinpoint the problem.
>
> Greetings,
>
> Domi
>
>
> On 12/28/2013 10:42 PM, Talat Uyarer wrote:
> > Hi,
> >
> > May i ask you do you run on local mode with one server. If you will say
> no.
> > Can you share with me your mapred-site.xml or you can share job.xml
> options
> > with your cluster infrastructer.
> >
> > Thanks
> > Talat
> > 28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <
> webmaster@law-firms-in.com>
> > yazdı:
> >
> >> Hello all,
> >>
> >> I wanted to inquiry about the general performance of nutch. I have seen
> >> this page here
> >> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
> >> where it takes
> >>
> >> 78minutes
> >>
> >> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
> >>
> >> I have myself the same setup as in the test but with currently only
> >> around 70k urls in the database.
> >>
> >> The steps fetch/parse go very quick but the steps generate/update take
> >> both _forever_. I have for 1 run about 12 hours and by far the most time
> >> is spent at update followed by generate.
> >>
> >> Is there ANYTHING I can do to speedup the process? I have a strong
> >> dedicated server with 52GB RAM. One thing I notice is that during
> >> generate/update ALL available RAM is used (Mem: 52438M total,
> >> 52267M used, 170M free, 191M buffers).
> >>
> >> I am thankful for any help/feedback!
> >>
> >> Domi
> >>
> >>
> >>
>
>
--
Talat UYARER
Websitesi: http://talat.uyarer.com
Twitter: http://twitter.com/talatuyarer
Linkedin: http://tr.linkedin.com/pub/talat-uyarer/10/142/304
Re: nutch 2.2.1/hbase performance
Posted by "Law-Firms-In.com" <we...@law-firms-in.com>.
Hi Yazdi,
thank you for your reply.
I am running on a single server currently (1 server).
In my hbase (hbase-0.90.6/conf) folder I see the following files:
hadoop-metrics.properties
hbase-env.sh
hbase-site.xml
log4j.properties
regionservers
I dont see the files you ask for. Please let me know what info might
help you pinpoint the problem.
Greetings,
Domi
On 12/28/2013 10:42 PM, Talat Uyarer wrote:
> Hi,
>
> May i ask you do you run on local mode with one server. If you will say no.
> Can you share with me your mapred-site.xml or you can share job.xml options
> with your cluster infrastructer.
>
> Thanks
> Talat
> 28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <we...@law-firms-in.com>
> yazdı:
>
>> Hello all,
>>
>> I wanted to inquiry about the general performance of nutch. I have seen
>> this page here
>> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
>> where it takes
>>
>> 78minutes
>>
>> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
>>
>> I have myself the same setup as in the test but with currently only
>> around 70k urls in the database.
>>
>> The steps fetch/parse go very quick but the steps generate/update take
>> both _forever_. I have for 1 run about 12 hours and by far the most time
>> is spent at update followed by generate.
>>
>> Is there ANYTHING I can do to speedup the process? I have a strong
>> dedicated server with 52GB RAM. One thing I notice is that during
>> generate/update ALL available RAM is used (Mem: 52438M total,
>> 52267M used, 170M free, 191M buffers).
>>
>> I am thankful for any help/feedback!
>>
>> Domi
>>
>>
>>
Re: nutch 2.2.1/hbase performance
Posted by Talat Uyarer <ta...@uyarer.com>.
Hi,
May i ask you do you run on local mode with one server. If you will say no.
Can you share with me your mapred-site.xml or you can share job.xml options
with your cluster infrastructer.
Thanks
Talat
28 Ara 2013 21:39 tarihinde "Law-Firms-In.com" <we...@law-firms-in.com>
yazdı:
> Hello all,
>
> I wanted to inquiry about the general performance of nutch. I have seen
> this page here
> (http://digitalpebble.blogspot.cz/2013/09/nutch-fight-17-vs-221.html)
> where it takes
>
> 78minutes
>
> for 1 iteration with 3M urls/ 5K per iteration with 100 urls/host.
>
> I have myself the same setup as in the test but with currently only
> around 70k urls in the database.
>
> The steps fetch/parse go very quick but the steps generate/update take
> both _forever_. I have for 1 run about 12 hours and by far the most time
> is spent at update followed by generate.
>
> Is there ANYTHING I can do to speedup the process? I have a strong
> dedicated server with 52GB RAM. One thing I notice is that during
> generate/update ALL available RAM is used (Mem: 52438M total,
> 52267M used, 170M free, 191M buffers).
>
> I am thankful for any help/feedback!
>
> Domi
>
>
>