You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Matt Cheah <mc...@palantir.com> on 2013/12/04 00:50:25 UTC
Benchmark numbers for terabytes of data
Hi everyone,
I notice the benchmark page for AMPLab provides some numbers on Gbs of data: https://amplab.cs.berkeley.edu/benchmark/ I was wondering if similar benchmark numbers existed for even larger data sets, in the terabytes if possible.
Also, are there any for just raw spark, i.e. No shark?
Thanks,
-Matt Chetah
Re: Benchmark numbers for terabytes of data
Posted by Christopher Nguyen <ct...@adatao.com>.
Matt, we've done 1TB linear models in 2-3 minutes on 40 node clusters
(30GB/node, just enough to hold all partitions simultaneously in memory).
You can do with fewer nodes if you're willing to slow things down.
Some of our TB benchmark numbers are available in my Spark Summit slides.
Sorry I'm on a plane now but you should be able to find the slides fairly
easily.
Re your other comment: monolithic 100-node analytic clusters are not
unusual, but not yet common outside of large companies. I'd eduguesstimate
it to be at the top 5%ile among companies with less than $500MM revenues,
with selection bias among Silicon Valley companies.
Sent while mobile. Pls excuse typos etc.
On Dec 4, 2013 11:06 AM, "Matt Cheah" <mc...@palantir.com> wrote:
> I'm reading the paper now, thanks. It states 100-node clusters were
> used. Is this typical in the field to have 100 node clusters for the 1TB
> scale? We were expecting to be using ~10 nodes.
>
> I'm still pretty new to cluster computing, so just not sure how people
> have set these up.
>
> -Matt Cheah
>
> From: Matei Zaharia <ma...@gmail.com>
> Reply-To: "user@spark.incubator.apache.org" <
> user@spark.incubator.apache.org>
> Date: Wednesday, December 4, 2013 10:53 AM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Cc: Mingyu Kim <mk...@palantir.com>
> Subject: Re: Benchmark numbers for terabytes of data
>
> Yes, check out the Shark paper for example:
> https://amplab.cs.berkeley.edu/publication/shark-sql-and-rich-analytics-at-scale/
>
> The numbers on that benchmark are for Shark.
>
> Matei
>
> On Dec 3, 2013, at 3:50 PM, Matt Cheah <mc...@palantir.com> wrote:
>
> Hi everyone,
>
> I notice the benchmark page for AMPLab provides some numbers on Gbs of
> data: https://amplab.cs.berkeley.edu/benchmark/ I was wondering if
> similar benchmark numbers existed for even larger data sets, in the
> terabytes if possible.
>
> Also, are there any for just raw spark, i.e. No shark?
>
> Thanks,
>
> -Matt Chetah
>
>
>
Re: Benchmark numbers for terabytes of data
Posted by Matei Zaharia <ma...@gmail.com>.
These were EC2 clusters, so the machines were smaller than modern machines. You can definitely have 1 TB datasets on 10 nodes too. Actually if you’re curious about hardware configuration, take a look at http://spark.incubator.apache.org/docs/latest/hardware-provisioning.html.
Also, regarding Spark vs Shark — raw Spark code is usually faster than Shark, but we don’t have as many recent benchmarks on large datasets. Some of the code running in the Shark paper is Spark-based though (specifically the machine learning algorithms).
Matei
On Dec 4, 2013, at 11:06 AM, Matt Cheah <mc...@palantir.com> wrote:
> I'm reading the paper now, thanks. It states 100-node clusters were used. Is this typical in the field to have 100 node clusters for the 1TB scale? We were expecting to be using ~10 nodes.
>
> I'm still pretty new to cluster computing, so just not sure how people have set these up.
>
> -Matt Cheah
>
> From: Matei Zaharia <ma...@gmail.com>
> Reply-To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Date: Wednesday, December 4, 2013 10:53 AM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Cc: Mingyu Kim <mk...@palantir.com>
> Subject: Re: Benchmark numbers for terabytes of data
>
> Yes, check out the Shark paper for example: https://amplab.cs.berkeley.edu/publication/shark-sql-and-rich-analytics-at-scale/
>
> The numbers on that benchmark are for Shark.
>
> Matei
>
> On Dec 3, 2013, at 3:50 PM, Matt Cheah <mc...@palantir.com> wrote:
>
>> Hi everyone,
>>
>> I notice the benchmark page for AMPLab provides some numbers on Gbs of data: https://amplab.cs.berkeley.edu/benchmark/ I was wondering if similar benchmark numbers existed for even larger data sets, in the terabytes if possible.
>>
>> Also, are there any for just raw spark, i.e. No shark?
>>
>> Thanks,
>>
>> -Matt Chetah
>
Re: Benchmark numbers for terabytes of data
Posted by Matt Cheah <mc...@palantir.com>.
I'm reading the paper now, thanks. It states 100-node clusters were used. Is this typical in the field to have 100 node clusters for the 1TB scale? We were expecting to be using ~10 nodes.
I'm still pretty new to cluster computing, so just not sure how people have set these up.
-Matt Cheah
From: Matei Zaharia <ma...@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Date: Wednesday, December 4, 2013 10:53 AM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Cc: Mingyu Kim <mk...@palantir.com>>
Subject: Re: Benchmark numbers for terabytes of data
Yes, check out the Shark paper for example: https://amplab.cs.berkeley.edu/publication/shark-sql-and-rich-analytics-at-scale/
The numbers on that benchmark are for Shark.
Matei
On Dec 3, 2013, at 3:50 PM, Matt Cheah <mc...@palantir.com>> wrote:
Hi everyone,
I notice the benchmark page for AMPLab provides some numbers on Gbs of data: https://amplab.cs.berkeley.edu/benchmark/ I was wondering if similar benchmark numbers existed for even larger data sets, in the terabytes if possible.
Also, are there any for just raw spark, i.e. No shark?
Thanks,
-Matt Chetah
Re: Benchmark numbers for terabytes of data
Posted by Matei Zaharia <ma...@gmail.com>.
Yes, check out the Shark paper for example: https://amplab.cs.berkeley.edu/publication/shark-sql-and-rich-analytics-at-scale/
The numbers on that benchmark are for Shark.
Matei
On Dec 3, 2013, at 3:50 PM, Matt Cheah <mc...@palantir.com> wrote:
> Hi everyone,
>
> I notice the benchmark page for AMPLab provides some numbers on Gbs of data: https://amplab.cs.berkeley.edu/benchmark/ I was wondering if similar benchmark numbers existed for even larger data sets, in the terabytes if possible.
>
> Also, are there any for just raw spark, i.e. No shark?
>
> Thanks,
>
> -Matt Chetah