nutch/hadoop performance and optimal configuration

[DS jha <ae...@gmail.com>]

I am using Nutch to crawl about 10M pages and looks like the machines
on my cluster are heavily under-utilized – I wanted to know what would
be a good configuration I can use to improve the performance.

I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
instances of type high CPU/Medium  (dual processors with about 1.7G
RAM each) running on Cent OS.  I have configured mapred-default.xml
file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
I have set generate.max.per.host to 50 and number of fetcher threads
to about 300. However, I noticed that whenever I run any nutch jobs,
it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
2 map tasks, one each on two slaves and a reduce task on the third
machine.  And so, CPU & memory utilization on each of the boxes is
very low – is that normal? For fetch jobs of 300K records, it is
taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
only one of the boxes and fail to finish because of outofMemory
exception.

What would be a good configuration of map/reduce tasks I can use for
these boxes? Or does nutch override configuration defined in
mapred-default file and set its own number of tasks for each job? Is
above set of machines enough for me to be able to crawl up to 10M
pages?


Cheers,

Re: nutch/hadoop performance and optimal configuration

[DS jha <ae...@gmail.com>]

replication is set to 2. and Network speed seems pretty decent. My
main concern is number of map/reduce tasks that gets generated for
Nutch jobs. In my mapred-default file, I have set 30 map tasks and 7
reduce tasks and tried with several other values as well (high and
low) - but it seems that nutch jobs overwrites those values - For most
nutch jobs, including parse, fetch, updatedb, etc - it always creates
only 1 reduce task. On jobtracker, job configuration file also has
mapred.reduce.tasks value set to 1. For fetch jobs, even though there
are 3 slaves, it creates only 2 map tasks and 1 redue tasks (one on
each of the slave).  Am I missing any config step/param entry?



Cheers,


On Thu, Apr 2, 2009 at 9:45 PM, Jack Yu <ja...@gmail.com> wrote:
> maybe more map,replication?
> what is the network speed between 3?
> what is the internet speed they have?
>
> On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:
>
>> I am using Nutch to crawl about 10M pages and looks like the machines
>> on my cluster are heavily under-utilized – I wanted to know what would
>> be a good configuration I can use to improve the performance.
>>
>> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
>> instances of type high CPU/Medium  (dual processors with about 1.7G
>> RmaAM each) running on Cent OS.  I have configured mapred-default.xml
>> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
>> I have set generate.max.per.host to 50 and number of fetcher threads
>> to about 300. However, I noticed that whenever I run any nutch jobs,
>> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
>> 2 map tasks, one each on two slaves and a reduce task on the third
>> machine.  And so, CPU & memory utilization on each of the boxes is
>> very low – is that normal? For fetch jobs of 300K records, it is
>> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
>> only one of the boxes and fail to finish because of outofMemory
>> exception.
>>
>> What would be a good configuration of map/reduce tasks I can use for
>> these boxes? Or does nutch override configuration defined in
>> mapred-default file and set its own number of tasks for each job? Is
>> above set of machines enough for me to be able to crawl up to 10M
>> pages?
>>
>>
>> Cheers,
>>
>

Re: nutch/hadoop performance and optimal configuration

[Jack Yu <ja...@gmail.com>]

maybe more map,replication?
what is the network speed between 3?
what is the internet speed they have?

On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:

> I am using Nutch to crawl about 10M pages and looks like the machines
> on my cluster are heavily under-utilized – I wanted to know what would
> be a good configuration I can use to improve the performance.
>
> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
> instances of type high CPU/Medium  (dual processors with about 1.7G
> RmaAM each) running on Cent OS.  I have configured mapred-default.xml
> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
> I have set generate.max.per.host to 50 and number of fetcher threads
> to about 300. However, I noticed that whenever I run any nutch jobs,
> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
> 2 map tasks, one each on two slaves and a reduce task on the third
> machine.  And so, CPU & memory utilization on each of the boxes is
> very low – is that normal? For fetch jobs of 300K records, it is
> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
> only one of the boxes and fail to finish because of outofMemory
> exception.
>
> What would be a good configuration of map/reduce tasks I can use for
> these boxes? Or does nutch override configuration defined in
> mapred-default file and set its own number of tasks for each job? Is
> above set of machines enough for me to be able to crawl up to 10M
> pages?
>
>
> Cheers,
>

Re: nutch/hadoop performance and optimal configuration

[perezcebreros <pe...@hotmail.com>]

About that....

please, I have a question.  Why do i need to set a prime number in
hadoo-site.xml,, So is it necessary 
may be for performance ?? do you know please, any link

Thank you


DS jha wrote:
> 
> Where do you set your task properties? (mapred.map.tasks and
> mapred.reduce.tasks)? I had these set in mapred-default.xml - but it
> didn't seem to have an impact on nutch jobs. When I set these
> properties in hadoop-site.xml or nutch-site.xml, nutch jobs uses these
> task properties.  Where should these config entries be defined?
> 
> Thank you,
> 
> 
> 
> 
> On Thu, Apr 2, 2009 at 9:54 PM, Jack Yu <ja...@gmail.com> wrote:
>> on my 3 local machine
>> 36 map(should be more),16 reduce setting
>> there are 2 map jobs and 2 reduce jobs each machine runnng
>>
>>
>> On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:
>>
>>> I am using Nutch to crawl about 10M pages and looks like the machines
>>> on my cluster are heavily under-utilized – I wanted to know what would
>>> be a good configuration I can use to improve the performance.
>>>
>>> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
>>> instances of type high CPU/Medium  (dual processors with about 1.7G
>>> RAM each) running on Cent OS.  I have configured mapred-default.xml
>>> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
>>> I have set generate.max.per.host to 50 and number of fetcher threads
>>> to about 300. However, I noticed that whenever I run any nutch jobs,
>>> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
>>> 2 map tasks, one each on two slaves and a reduce task on the third
>>> machine.  And so, CPU & memory utilization on each of the boxes is
>>> very low – is that normal? For fetch jobs of 300K records, it is
>>> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
>>> only one of the boxes and fail to finish because of outofMemory
>>> exception.
>>>
>>> What would be a good configuration of map/reduce tasks I can use for
>>> these boxes? Or does nutch override configuration defined in
>>> mapred-default file and set its own number of tasks for each job? Is
>>> above set of machines enough for me to be able to crawl up to 10M
>>> pages?
>>>
>>>
>>> Cheers,
>>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/nutch-hadoop-performance-and-optimal-configuration-tp22858445p23604528.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: nutch/hadoop performance and optimal configuration

[DS jha <ae...@gmail.com>]

Where do you set your task properties? (mapred.map.tasks and
mapred.reduce.tasks)? I had these set in mapred-default.xml - but it
didn't seem to have an impact on nutch jobs. When I set these
properties in hadoop-site.xml or nutch-site.xml, nutch jobs uses these
task properties.  Where should these config entries be defined?

Thank you,




On Thu, Apr 2, 2009 at 9:54 PM, Jack Yu <ja...@gmail.com> wrote:
> on my 3 local machine
> 36 map(should be more),16 reduce setting
> there are 2 map jobs and 2 reduce jobs each machine runnng
>
>
> On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:
>
>> I am using Nutch to crawl about 10M pages and looks like the machines
>> on my cluster are heavily under-utilized – I wanted to know what would
>> be a good configuration I can use to improve the performance.
>>
>> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
>> instances of type high CPU/Medium  (dual processors with about 1.7G
>> RAM each) running on Cent OS.  I have configured mapred-default.xml
>> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
>> I have set generate.max.per.host to 50 and number of fetcher threads
>> to about 300. However, I noticed that whenever I run any nutch jobs,
>> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
>> 2 map tasks, one each on two slaves and a reduce task on the third
>> machine.  And so, CPU & memory utilization on each of the boxes is
>> very low – is that normal? For fetch jobs of 300K records, it is
>> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
>> only one of the boxes and fail to finish because of outofMemory
>> exception.
>>
>> What would be a good configuration of map/reduce tasks I can use for
>> these boxes? Or does nutch override configuration defined in
>> mapred-default file and set its own number of tasks for each job? Is
>> above set of machines enough for me to be able to crawl up to 10M
>> pages?
>>
>>
>> Cheers,
>>
>

Re: nutch/hadoop performance and optimal configuration

[al...@aim.com]

 


 Are not EC2 virtual hosts. I had problem with speed in my virtual hosts in local linux box. 




What is preferable, a dedicated server or an EC2?

 

-----Original Message-----
From: Jack Yu <ja...@gmail.com>
To: nutch-user@lucene.apache.org
Sent: Thu, 2 Apr 2009 6:54 pm
Subject: Re: nutch/hadoop performance and optimal configuration










on my 3 local machine
36 map(should be more),16 reduce setting
there are 2 map jobs and 2 reduce jobs each machine runnng


On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:

> I am using Nutch to crawl about 10M pages and looks like the machines
> on my cluster are heavily under-utilized – I wanted to know what would
> be a good configuration I can use to improve the performance.
>
> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
> instances of type high CPU/Medium  (dual processors with about 1.7G
> RAM each) running on Cent OS.  I have configured mapred-default.xml
> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
> I have set generate.max.per.host to 50 and number of fetcher threads
> to about 300. However, I noticed that whenever I run any nutch jobs,
> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
> 2 map tasks, one each on two slaves and a reduce task on the third
> machine.  And so, CPU & memory utilization on each of the boxes is
> very low – is that normal? For fetch jobs of 300K record
s, it is
> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
> only one of the boxes and fail to finish because of outofMemory
> exception.
>
> What would be a good configuration of map/reduce tasks I can use for
> these boxes? Or does nutch override configuration defined in
> mapred-default file and set its own number of tasks for each job? Is
> above set of machines enough for me to be able to crawl up to 10M
> pages?
>
>
> Cheers,
>



 

Re: nutch/hadoop performance and optimal configuration

[Jack Yu <ja...@gmail.com>]

on my 3 local machine
36 map(should be more),16 reduce setting
there are 2 map jobs and 2 reduce jobs each machine runnng


On Fri, Apr 3, 2009 at 6:39 AM, DS jha <ae...@gmail.com> wrote:

> I am using Nutch to crawl about 10M pages and looks like the machines
> on my cluster are heavily under-utilized – I wanted to know what would
> be a good configuration I can use to improve the performance.
>
> I have setup my cluster on Amazon/EC2 and currently I have 3 EC2
> instances of type high CPU/Medium  (dual processors with about 1.7G
> RAM each) running on Cent OS.  I have configured mapred-default.xml
> file to define 30 map tasks and 7 reduce tasks. For nutch fetch jobs,
> I have set generate.max.per.host to 50 and number of fetcher threads
> to about 300. However, I noticed that whenever I run any nutch jobs,
> it runs only 1 reduce jobs. Also, for tasks like fetch, it starts only
> 2 map tasks, one each on two slaves and a reduce task on the third
> machine.  And so, CPU & memory utilization on each of the boxes is
> very low – is that normal? For fetch jobs of 300K records, it is
> taking over 10 hrs to finish and sometimes jobs like mergesegs runs on
> only one of the boxes and fail to finish because of outofMemory
> exception.
>
> What would be a good configuration of map/reduce tasks I can use for
> these boxes? Or does nutch override configuration defined in
> mapred-default file and set its own number of tasks for each job? Is
> above set of machines enough for me to be able to crawl up to 10M
> pages?
>
>
> Cheers,
>