You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rohit Pujari <rp...@hortonworks.com> on 2014/07/16 05:47:08 UTC
Can Spark stack scale to petabyte scale without performance degradation?
Hello Folks:
There is lot of buzz in the hadoop community around Spark's inability to
scale beyond the 1 TB datasets ( or 10-20 nodes). It is being regarded as
great tech for cpu intensive workloads on smaller data( less that TB) but
fails to scale and perform effectively on larger datasets. How true it is?
Are there any customers in who are running petabyte scale workloads on
spark in production? Are there any benchmarks performed by databricks or
other companies to clear this perception?
I'm a big fan of spark. Knowing spark is in its early stages, I'd like to
better understand boundaries of the tech and recommend right solution for
right problem.
Thanks,
Rohit Pujari
Solutions Engineer, Hortonworks
rpujari@hortonworks.com
716-430-6899
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Spark stack scale to petabyte scale without performance degradation?
Posted by Rohit Pujari <rp...@hortonworks.com>.
Thanks Matei.
On Tue, Jul 15, 2014 at 11:47 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> Yup, as mentioned in the FAQ, we are aware of multiple deployments running
> jobs on over 1000 nodes. Some of our proof of concepts involved people
> running a 2000-node job on EC2.
>
> I wouldn't confuse buzz with FUD :).
>
> Matei
>
> On Jul 15, 2014, at 9:17 PM, Sonal Goyal <so...@gmail.com> wrote:
>
> Hi Rohit,
>
> I think the 3rd question on the FAQ may help you.
>
> https://spark.apache.org/faq.html
>
> Some other links that talk about building bigger clusters and processing
> more data:
>
>
> http://spark-summit.org/wp-content/uploads/2014/07/Building-1000-node-Spark-Cluster-on-EMR.pdf
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Largest-Spark-Cluster-td3782.html
>
>
>
> Best Regards,
> Sonal
> Nube Technologies <http://www.nubetech.co/>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
> On Wed, Jul 16, 2014 at 9:17 AM, Rohit Pujari <rp...@hortonworks.com>
> wrote:
>
>> Hello Folks:
>>
>> There is lot of buzz in the hadoop community around Spark's inability to
>> scale beyond the 1 TB datasets ( or 10-20 nodes). It is being regarded as
>> great tech for cpu intensive workloads on smaller data( less that TB) but
>> fails to scale and perform effectively on larger datasets. How true it is?
>>
>> Are there any customers in who are running petabyte scale workloads on
>> spark in production? Are there any benchmarks performed by databricks or
>> other companies to clear this perception?
>>
>> I'm a big fan of spark. Knowing spark is in its early stages, I'd like
>> to better understand boundaries of the tech and recommend right solution
>> for right problem.
>>
>> Thanks,
>> Rohit Pujari
>> Solutions Engineer, Hortonworks
>> rpujari@hortonworks.com
>> 716-430-6899
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
--
Rohit Pujari
Solutions Engineer, Hortonworks
rpujari@hortonworks.com
716-430-6899
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Spark stack scale to petabyte scale without performance degradation?
Posted by Matei Zaharia <ma...@gmail.com>.
Yup, as mentioned in the FAQ, we are aware of multiple deployments running jobs on over 1000 nodes. Some of our proof of concepts involved people running a 2000-node job on EC2.
I wouldn't confuse buzz with FUD :).
Matei
On Jul 15, 2014, at 9:17 PM, Sonal Goyal <so...@gmail.com> wrote:
> Hi Rohit,
>
> I think the 3rd question on the FAQ may help you.
>
> https://spark.apache.org/faq.html
>
> Some other links that talk about building bigger clusters and processing more data:
>
> http://spark-summit.org/wp-content/uploads/2014/07/Building-1000-node-Spark-Cluster-on-EMR.pdf
> http://apache-spark-user-list.1001560.n3.nabble.com/Largest-Spark-Cluster-td3782.html
>
>
>
> Best Regards,
> Sonal
> Nube Technologies
>
>
>
>
>
>
> On Wed, Jul 16, 2014 at 9:17 AM, Rohit Pujari <rp...@hortonworks.com> wrote:
> Hello Folks:
>
> There is lot of buzz in the hadoop community around Spark's inability to scale beyond the 1 TB datasets ( or 10-20 nodes). It is being regarded as great tech for cpu intensive workloads on smaller data( less that TB) but fails to scale and perform effectively on larger datasets. How true it is?
>
> Are there any customers in who are running petabyte scale workloads on spark in production? Are there any benchmarks performed by databricks or other companies to clear this perception?
>
> I'm a big fan of spark. Knowing spark is in its early stages, I'd like to better understand boundaries of the tech and recommend right solution for right problem.
>
> Thanks,
> Rohit Pujari
> Solutions Engineer, Hortonworks
> rpujari@hortonworks.com
> 716-430-6899
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
Re: Can Spark stack scale to petabyte scale without performance degradation?
Posted by Sonal Goyal <so...@gmail.com>.
Hi Rohit,
I think the 3rd question on the FAQ may help you.
https://spark.apache.org/faq.html
Some other links that talk about building bigger clusters and processing
more data:
http://spark-summit.org/wp-content/uploads/2014/07/Building-1000-node-Spark-Cluster-on-EMR.pdf
http://apache-spark-user-list.1001560.n3.nabble.com/Largest-Spark-Cluster-td3782.html
Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Wed, Jul 16, 2014 at 9:17 AM, Rohit Pujari <rp...@hortonworks.com>
wrote:
> Hello Folks:
>
> There is lot of buzz in the hadoop community around Spark's inability to
> scale beyond the 1 TB datasets ( or 10-20 nodes). It is being regarded as
> great tech for cpu intensive workloads on smaller data( less that TB) but
> fails to scale and perform effectively on larger datasets. How true it is?
>
> Are there any customers in who are running petabyte scale workloads on
> spark in production? Are there any benchmarks performed by databricks or
> other companies to clear this perception?
>
> I'm a big fan of spark. Knowing spark is in its early stages, I'd like
> to better understand boundaries of the tech and recommend right solution
> for right problem.
>
> Thanks,
> Rohit Pujari
> Solutions Engineer, Hortonworks
> rpujari@hortonworks.com
> 716-430-6899
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.