You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gary Malouf <ma...@gmail.com> on 2013/11/14 04:40:05 UTC
Continued performance issues on a small EC2 Spark cluster
Hi,
We have an HDFS set up of a namenode and three datanodes all on EC2
larges. One of our data partitions basically has files that are fed from a
few Flume instances rolling *hourly*. This equates to around 3 4-8mb files
per hour right now
Our Mesos cluster consists of a Master and the three slave nodes colocated
on these EC2 larges as well (slaves -> datanodes, mesos master ->
namenode). Spark scheduled jobs are launched from spark shell ad-hoc today.
The data is serialized protobuf messages in sequence files. Our operations
typically consist of deserializing the data, grabbing a few primitive
fields out of the message and doing some maps/reduces.
For grabbing on the order of 2 days of data this size, what would the
expected Spark performance be? We are seeing simple maps and 'takes' on
this data taking on the order of 15 minutes.
Thanks,
Gary
Re: Continued performance issues on a small EC2 Spark cluster
Posted by "Michael (Bach) Bui" <fr...@adatao.com>.
Hi Gary,
What are other frameworks running on your Mesos cluster?
If they are all Spark frameworks. Another option you may want to consider (in order to improve your cluster utilization) is to let all of them share a single SparkContext.
We also experienced degraded performance while running multiple Spark frameworks on Mesos, mainly due to there are a lot of non-preferred-location tasks.
Best,
~ mbbui
On Nov 15, 2013, at 11:30 AM, Gary Malouf <ma...@gmail.com> wrote:
> Hi everyone,
>
> I want to share what helped us resolve the issue short term and also our concerns longer term.
>
> Some Background:
>
> Many of our jobs that look at a few weeks of data have task counts around 3500+. We went with a small cluster of 4 EC2 larges for Mesos+Spark in production for now because our data sizes are small (3 files hourly on the order of 10-20mb). The idea is to get this steady and scale horizontally as the data flow continues to go up.
>
> Our Problem:
>
> In short, the time to execute simple tasks with this data of this size for the past 2 weeks or so was not reasonable for us. The jobs were taking well over an hour to complete. We were very concerned that as the hourly data grows in size this will become completely unacceptable.
>
> The (Short Term) Solution:
>
> Thanks to Paco and Evan's input, we decided to switch Spark to run in coarse grained mode for the time-being. We saw an immediate 6-8x improvement on our job runtime and we were able to complete our short term tasks.
>
> The downside we see to this is that it takes away many of the benefits of running on Mesos - namely, the fine-grained resource management.
>
> The Longer Term Solution:
>
> We are looking at ways of tuning to reduce the number of Spark tasks being generated overall. We think if we can get this down enough, it justifies taking the performance hit in fine-grained mode since the cluster resources will now be allocated dynamically for multiple jobs.
>
>
> Hope this is helpful to people,
>
> Gary
>
>
> On Thu, Nov 14, 2013 at 11:47 AM, Gary Malouf <ma...@gmail.com> wrote:
> I bring this up because the performance we are seeing is dreadful. From cpu usage, it appears the issue is the spark shell cpu power. We have increased this node from a EC2 medium to an xl, we are seeing slightly better performance but still not great.
>
> My understanding of Spark was that most of the work should be done on the slaves with just the results going back to the shell at the end if we do a take. It appears from what we see that the client is doing much more work than expected.
>
>
>
>
> On Wed, Nov 13, 2013 at 10:40 PM, Gary Malouf <ma...@gmail.com> wrote:
> Hi,
>
> We have an HDFS set up of a namenode and three datanodes all on EC2 larges. One of our data partitions basically has files that are fed from a few Flume instances rolling hourly. This equates to around 3 4-8mb files per hour right now
>
> Our Mesos cluster consists of a Master and the three slave nodes colocated on these EC2 larges as well (slaves -> datanodes, mesos master -> namenode). Spark scheduled jobs are launched from spark shell ad-hoc today.
>
> The data is serialized protobuf messages in sequence files. Our operations typically consist of deserializing the data, grabbing a few primitive fields out of the message and doing some maps/reduces.
>
> For grabbing on the order of 2 days of data this size, what would the expected Spark performance be? We are seeing simple maps and 'takes' on this data taking on the order of 15 minutes.
>
> Thanks,
>
> Gary
>
>
Re: Continued performance issues on a small EC2 Spark cluster
Posted by Gary Malouf <ma...@gmail.com>.
Hi everyone,
I want to share what helped us resolve the issue short term and also our
concerns longer term.
*Some Background*:
Many of our jobs that look at a few weeks of data have task counts around
3500+. We went with a small cluster of 4 EC2 larges for Mesos+Spark in
production for now because our data sizes are small (3 files hourly on the
order of 10-20mb). The idea is to get this steady and scale horizontally
as the data flow continues to go up.
*Our Problem*:
In short, the time to execute simple tasks with this data of this size for
the past 2 weeks or so was not reasonable for us. The jobs were taking
well over an hour to complete. We were very concerned that as the hourly
data grows in size this will become completely unacceptable.
*The (Short Term) Solution*:
Thanks to Paco and Evan's input, we decided to switch Spark to run in
coarse grained mode for the time-being. We saw an immediate 6-8x
improvement on our job runtime and we were able to complete our short term
tasks.
The downside we see to this is that it takes away many of the benefits of
running on Mesos - namely, the fine-grained resource management.
*The Longer Term Solution*:
We are looking at ways of tuning to reduce the number of Spark tasks being
generated overall. We think if we can get this down enough, it justifies
taking the performance hit in fine-grained mode since the cluster resources
will now be allocated dynamically for multiple jobs.
Hope this is helpful to people,
Gary
On Thu, Nov 14, 2013 at 11:47 AM, Gary Malouf <ma...@gmail.com> wrote:
> I bring this up because the performance we are seeing is dreadful. From
> cpu usage, it appears the issue is the spark shell cpu power. We have
> increased this node from a EC2 medium to an xl, we are seeing slightly
> better performance but still not great.
>
> My understanding of Spark was that most of the work should be done on the
> slaves with just the results going back to the shell at the end if we do a
> take. It appears from what we see that the client is doing much more work
> than expected.
>
>
>
>
> On Wed, Nov 13, 2013 at 10:40 PM, Gary Malouf <ma...@gmail.com>wrote:
>
>> Hi,
>>
>> We have an HDFS set up of a namenode and three datanodes all on EC2
>> larges. One of our data partitions basically has files that are fed from a
>> few Flume instances rolling *hourly*. This equates to around 3 4-8mb
>> files per hour right now
>>
>> Our Mesos cluster consists of a Master and the three slave nodes
>> colocated on these EC2 larges as well (slaves -> datanodes, mesos master ->
>> namenode). Spark scheduled jobs are launched from spark shell ad-hoc today.
>>
>> The data is serialized protobuf messages in sequence files. Our
>> operations typically consist of deserializing the data, grabbing a few
>> primitive fields out of the message and doing some maps/reduces.
>>
>> For grabbing on the order of 2 days of data this size, what would the
>> expected Spark performance be? We are seeing simple maps and 'takes' on
>> this data taking on the order of 15 minutes.
>>
>> Thanks,
>>
>> Gary
>>
>
>
Re: Continued performance issues on a small EC2 Spark cluster
Posted by Gary Malouf <ma...@gmail.com>.
I bring this up because the performance we are seeing is dreadful. From
cpu usage, it appears the issue is the spark shell cpu power. We have
increased this node from a EC2 medium to an xl, we are seeing slightly
better performance but still not great.
My understanding of Spark was that most of the work should be done on the
slaves with just the results going back to the shell at the end if we do a
take. It appears from what we see that the client is doing much more work
than expected.
On Wed, Nov 13, 2013 at 10:40 PM, Gary Malouf <ma...@gmail.com> wrote:
> Hi,
>
> We have an HDFS set up of a namenode and three datanodes all on EC2
> larges. One of our data partitions basically has files that are fed from a
> few Flume instances rolling *hourly*. This equates to around 3 4-8mb
> files per hour right now
>
> Our Mesos cluster consists of a Master and the three slave nodes colocated
> on these EC2 larges as well (slaves -> datanodes, mesos master ->
> namenode). Spark scheduled jobs are launched from spark shell ad-hoc today.
>
> The data is serialized protobuf messages in sequence files. Our
> operations typically consist of deserializing the data, grabbing a few
> primitive fields out of the message and doing some maps/reduces.
>
> For grabbing on the order of 2 days of data this size, what would the
> expected Spark performance be? We are seeing simple maps and 'takes' on
> this data taking on the order of 15 minutes.
>
> Thanks,
>
> Gary
>