You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Derek Gottfrid <dg...@gmail.com> on 2007/11/07 05:03:07 UTC
configuration suggestions for 1k nodes
Are there configuration suggestions for 1k nodes ? I was seeing tons
of timeouts trying to run 1k nodes. Are there network settings that I
need to make? Out of the box stuff seemed to work up to a couple
hundred but I want to go bigger. Pointers/Suggestions?
derek
ps: i wrote up my ec2/hadoop at nytimes.com - check it out
http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
Re: configuration suggestions for 1k nodes
Posted by Derek Gottfrid <de...@codecubed.com>.
excellent. I will keep the group posted as I progress.
On 11/7/07, Raghu Angadi <ra...@yahoo-inc.com> wrote:
>
> Not a complete list by far, but just a start :
>
> For HDFS:
>
> - Make sure you run Java 6 (jdk1.6).
> - Set namenode handler count to 40 or more (dfs.namenode.handler.count,
> and may be mapred.job.tracker.handler.count etc).
>
> - more config guides are in the works :
> https://issues.apache.org/jira/browse/HADOOP-1917
>
> Raghu.
>
> Derek Gottfrid wrote:
> > Are there configuration suggestions for 1k nodes ? I was seeing tons
> > of timeouts trying to run 1k nodes. Are there network settings that I
> > need to make? Out of the box stuff seemed to work up to a couple
> > hundred but I want to go bigger. Pointers/Suggestions?
> >
> > derek
> >
> >
> > ps: i wrote up my ec2/hadoop at nytimes.com - check it out
> > http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
>
>
Re: configuration suggestions for 1k nodes
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Not a complete list by far, but just a start :
For HDFS:
- Make sure you run Java 6 (jdk1.6).
- Set namenode handler count to 40 or more (dfs.namenode.handler.count,
and may be mapred.job.tracker.handler.count etc).
- more config guides are in the works :
https://issues.apache.org/jira/browse/HADOOP-1917
Raghu.
Derek Gottfrid wrote:
> Are there configuration suggestions for 1k nodes ? I was seeing tons
> of timeouts trying to run 1k nodes. Are there network settings that I
> need to make? Out of the box stuff seemed to work up to a couple
> hundred but I want to go bigger. Pointers/Suggestions?
>
> derek
>
>
> ps: i wrote up my ec2/hadoop at nytimes.com - check it out
> http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
Re: configuration suggestions for 1k nodes
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Doug Cutting wrote:
> Derek Gottfrid wrote:
>> Are there configuration suggestions for 1k nodes ?
>
> http://wiki.apache.org/lucene-hadoop/FAQ#3
Updated this entry with Java 1.6 recommendation.
Raghu.
Re: configuration suggestions for 1k nodes
Posted by Doug Cutting <cu...@apache.org>.
Derek Gottfrid wrote:
> Are there configuration suggestions for 1k nodes ?
http://wiki.apache.org/lucene-hadoop/FAQ#3
Doug