You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Derek Gottfrid <dg...@gmail.com> on 2007/11/07 05:03:07 UTC

configuration suggestions for 1k nodes

Are there configuration suggestions for 1k nodes ? I was seeing tons
of timeouts trying to run 1k nodes. Are there network settings that I
need to make? Out of the box stuff seemed to work up to a couple
hundred but I want to go bigger.  Pointers/Suggestions?

derek


ps: i wrote up my ec2/hadoop at nytimes.com - check it out
http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/

Re: configuration suggestions for 1k nodes

Posted by Derek Gottfrid <de...@codecubed.com>.
excellent. I will keep the group posted as I progress.

On 11/7/07, Raghu Angadi <ra...@yahoo-inc.com> wrote:
>
> Not a complete list by far, but just a start :
>
> For HDFS:
>
> - Make sure you run Java 6 (jdk1.6).
> - Set namenode handler count to 40 or more (dfs.namenode.handler.count,
> and may be  mapred.job.tracker.handler.count etc).
>
> - more config guides are in the works :
> https://issues.apache.org/jira/browse/HADOOP-1917
>
> Raghu.
>
> Derek Gottfrid wrote:
> > Are there configuration suggestions for 1k nodes ? I was seeing tons
> > of timeouts trying to run 1k nodes. Are there network settings that I
> > need to make? Out of the box stuff seemed to work up to a couple
> > hundred but I want to go bigger.  Pointers/Suggestions?
> >
> > derek
> >
> >
> > ps: i wrote up my ec2/hadoop at nytimes.com - check it out
> > http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
>
>

Re: configuration suggestions for 1k nodes

Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Not a complete list by far, but just a start :

For HDFS:

- Make sure you run Java 6 (jdk1.6).
- Set namenode handler count to 40 or more (dfs.namenode.handler.count, 
and may be  mapred.job.tracker.handler.count etc).

- more config guides are in the works : 
https://issues.apache.org/jira/browse/HADOOP-1917

Raghu.

Derek Gottfrid wrote:
> Are there configuration suggestions for 1k nodes ? I was seeing tons
> of timeouts trying to run 1k nodes. Are there network settings that I
> need to make? Out of the box stuff seemed to work up to a couple
> hundred but I want to go bigger.  Pointers/Suggestions?
> 
> derek
> 
> 
> ps: i wrote up my ec2/hadoop at nytimes.com - check it out
> http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/


Re: configuration suggestions for 1k nodes

Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Doug Cutting wrote:
> Derek Gottfrid wrote:
>> Are there configuration suggestions for 1k nodes ? 
> 
> http://wiki.apache.org/lucene-hadoop/FAQ#3

Updated this entry with Java 1.6 recommendation.

Raghu.


Re: configuration suggestions for 1k nodes

Posted by Doug Cutting <cu...@apache.org>.
Derek Gottfrid wrote:
> Are there configuration suggestions for 1k nodes ? 

http://wiki.apache.org/lucene-hadoop/FAQ#3

Doug