You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Devaraj Das <dd...@yahoo-inc.com> on 2007/09/05 11:29:24 UTC
Sort benchmark on 2000 nodes
This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
version) on a cluster of 2000 nodes. The programs we ran were RandomWriter
and Sort. Sort performance was pretty good - we could sort 20TB of data in
2.5 hours! Not many task failures - most of those that failed encountered
file checksum errors during merge and map output serving, some got killed
due to lack of progress reporting. Overall, a pretty successful run.
RE: Sort benchmark on 2000 nodes
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Will do so as soon as I get hold of a 1000 node cluster.
> -----Original Message-----
> From: Enis Soztutar [mailto:enis.soz.nutch@gmail.com]
> Sent: Wednesday, September 05, 2007 6:32 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Sort benchmark on 2000 nodes
>
> I am wondering how hadoop scores on sorting 1TB with say 1000
> nodes. Is it possible for you to try the Terasort benchmark?
>
>
> Devaraj Das wrote:
> > This is FYI. We at Yahoo! could successfully run hadoop
> (upto date trunk
> > version) on a cluster of 2000 nodes. The programs we ran
> were RandomWriter
> > and Sort. Sort performance was pretty good - we could sort
> 20TB of data in
> > 2.5 hours! Not many task failures - most of those that
> failed encountered
> > file checksum errors during merge and map output serving,
> some got killed
> > due to lack of progress reporting. Overall, a pretty successful run.
> >
> >
> >
>
Re: Sort benchmark on 2000 nodes
Posted by Enis Soztutar <en...@gmail.com>.
I am wondering how hadoop scores on sorting 1TB with say 1000 nodes. Is
it possible for you to try the Terasort benchmark?
Devaraj Das wrote:
> This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
> version) on a cluster of 2000 nodes. The programs we ran were RandomWriter
> and Sort. Sort performance was pretty good - we could sort 20TB of data in
> 2.5 hours! Not many task failures - most of those that failed encountered
> file checksum errors during merge and map output serving, some got killed
> due to lack of progress reporting. Overall, a pretty successful run.
>
>
>
RE: Sort benchmark on 2000 nodes
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Thanks Eric for pointing out the hardware spec. I have updated the
hadoop-config on the hadoop FAQ - http://wiki.apache.org/lucene-hadoop/FAQ
> -----Original Message-----
> From: Eric Baldeschwieler [mailto:eric14@yahoo-inc.com]
> Sent: Thursday, September 06, 2007 1:37 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Sort benchmark on 2000 nodes
>
> hardware is similar top that discussed here:
>
> http://wiki.apache.org/lucene-hadoop-data/attachments/
> HadoopPresentations/attachments/oscon-part-2.pdf
>
> - 10:1 oversubscribed network (so 100mBit bandwidth all nodes to all
> nodes)
> - 40 nodes / leaf switch
> - Machines are beefy
> - 4SATA drives, 500 or 750 GB each, 7200 RPM
> - 4+ cores (modern Intels or AMDs)
> - 4+ GB RAM
>
> On Sep 5, 2007, at 10:19 AM, Joydeep Sen Sarma wrote:
>
> > It will be very useful to see the hadoop/job config
> settings and get
> > some sense of the underlying hardware config.
> >
> > -----Original Message-----
> > From: Devaraj Das [mailto:ddas@yahoo-inc.com]
> > Sent: Wednesday, September 05, 2007 2:29 AM
> > To: hadoop-user@lucene.apache.org
> > Subject: Sort benchmark on 2000 nodes
> >
> > This is FYI. We at Yahoo! could successfully run hadoop (upto date
> > trunk
> > version) on a cluster of 2000 nodes. The programs we ran were
> > RandomWriter and Sort. Sort performance was pretty good - we could
> > sort 20TB of data in
> > 2.5 hours! Not many task failures - most of those that failed
> > encountered file checksum errors during merge and map
> output serving,
> > some got killed due to lack of progress reporting. Overall,
> a pretty
> > successful run.
> >
> >
>
>
Re: Sort benchmark on 2000 nodes
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
hardware is similar top that discussed here:
http://wiki.apache.org/lucene-hadoop-data/attachments/
HadoopPresentations/attachments/oscon-part-2.pdf
- 10:1 oversubscribed network (so 100mBit bandwidth all nodes to all
nodes)
- 40 nodes / leaf switch
- Machines are beefy
- 4SATA drives, 500 or 750 GB each, 7200 RPM
- 4+ cores (modern Intels or AMDs)
- 4+ GB RAM
On Sep 5, 2007, at 10:19 AM, Joydeep Sen Sarma wrote:
> It will be very useful to see the hadoop/job config settings and get
> some sense of the underlying hardware config.
>
> -----Original Message-----
> From: Devaraj Das [mailto:ddas@yahoo-inc.com]
> Sent: Wednesday, September 05, 2007 2:29 AM
> To: hadoop-user@lucene.apache.org
> Subject: Sort benchmark on 2000 nodes
>
> This is FYI. We at Yahoo! could successfully run hadoop (upto date
> trunk
> version) on a cluster of 2000 nodes. The programs we ran were
> RandomWriter
> and Sort. Sort performance was pretty good - we could sort 20TB of
> data
> in
> 2.5 hours! Not many task failures - most of those that failed
> encountered
> file checksum errors during merge and map output serving, some got
> killed
> due to lack of progress reporting. Overall, a pretty successful run.
>
>
RE: Sort benchmark on 2000 nodes
Posted by Joydeep Sen Sarma <js...@facebook.com>.
It will be very useful to see the hadoop/job config settings and get
some sense of the underlying hardware config.
-----Original Message-----
From: Devaraj Das [mailto:ddas@yahoo-inc.com]
Sent: Wednesday, September 05, 2007 2:29 AM
To: hadoop-user@lucene.apache.org
Subject: Sort benchmark on 2000 nodes
This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
version) on a cluster of 2000 nodes. The programs we ran were
RandomWriter
and Sort. Sort performance was pretty good - we could sort 20TB of data
in
2.5 hours! Not many task failures - most of those that failed
encountered
file checksum errors during merge and map output serving, some got
killed
due to lack of progress reporting. Overall, a pretty successful run.