You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Matthias Niehoff <ma...@codecentric.de> on 2015/10/26 21:00:33 UTC
Dynamic Resource Allocation with Spark Streaming (Standalone Cluster,
Spark 1.5.1)
Hello everybody,
I have a few (~15) Spark Streaming jobs which have load peaks as well as
long times with a low load. So I thought the new Dynamic Resource
Allocation for Standalone Clusters might be helpful (SPARK-4751).
I have a test "cluster" with 1 worker consisting of 4 executors with 2
cores each, so 8 cores in total.
I started a simple streaming application without limiting the max cores for
this app. As expected the app occupied every core of the cluster. Then I
started a second app, also without limiting the maximum cores. As the first
app did not get any input through the stream, my naive expectation was that
the second app would get at least 2 cores (1 receiver, 1 processing), but
that's not what happened. The cores are still assigned to the first app.
When I look at the application UI of the first app every executor is still
running. That explains why no executor is used for the second app.
I end up with two questions:
- When does an executor getting idle in a Spark Streaming application? (and
so could be reassigned to another app)
- Is there another way to compete with uncertain load when using Spark
Streaming Applications? I already combined multiple jobs to a Spark
Application using different threads, but this approach comes to a limit for
me, because Spark Applications get to big to manage.
Thank You!
Re: Dynamic Resource Allocation with Spark Streaming (Standalone
Cluster, Spark 1.5.1)
Posted by Deenar Toraskar <de...@gmail.com>.
Till Spark Streaming supports dynamic allocation, you could use
StreamingListener to monitor batch execution times and based on it
sparkContext.requestExecutors() and sparkContext.killExecutors() to add and
remove executors explicitly and .
On 26 October 2015 at 21:37, Ted Yu <yu...@gmail.com> wrote:
> This is related:
> SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs
>
> which went into 1.6.0 as well.
>
> FYI
>
> On Mon, Oct 26, 2015 at 2:26 PM, Silvio Fiorito <
> silvio.fiorito@granturing.com> wrote:
>
>> Hi Matthias,
>>
>> Unless there was a change in 1.5, I'm afraid dynamic resource allocation
>> is not yet supported in streaming apps.
>>
>> Thanks,
>> Silvio
>>
>> Sent from my Lumia 930
>> ------------------------------
>> From: Matthias Niehoff <ma...@codecentric.de>
>> Sent: 10/26/2015 4:00 PM
>> To: user@spark.apache.org
>> Subject: Dynamic Resource Allocation with Spark Streaming (Standalone
>> Cluster, Spark 1.5.1)
>>
>> Hello everybody,
>>
>> I have a few (~15) Spark Streaming jobs which have load peaks as well as
>> long times with a low load. So I thought the new Dynamic Resource
>> Allocation for Standalone Clusters might be helpful (SPARK-4751).
>>
>> I have a test "cluster" with 1 worker consisting of 4 executors with 2
>> cores each, so 8 cores in total.
>>
>> I started a simple streaming application without limiting the max cores
>> for this app. As expected the app occupied every core of the cluster. Then
>> I started a second app, also without limiting the maximum cores. As the
>> first app did not get any input through the stream, my naive expectation
>> was that the second app would get at least 2 cores (1 receiver, 1
>> processing), but that's not what happened. The cores are still assigned to
>> the first app.
>> When I look at the application UI of the first app every executor is
>> still running. That explains why no executor is used for the second app.
>>
>> I end up with two questions:
>> - When does an executor getting idle in a Spark Streaming application?
>> (and so could be reassigned to another app)
>> - Is there another way to compete with uncertain load when using Spark
>> Streaming Applications? I already combined multiple jobs to a Spark
>> Application using different threads, but this approach comes to a limit for
>> me, because Spark Applications get to big to manage.
>>
>> Thank You!
>>
>>
>>
>
Re: Dynamic Resource Allocation with Spark Streaming (Standalone
Cluster, Spark 1.5.1)
Posted by Ted Yu <yu...@gmail.com>.
This is related:
SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs
which went into 1.6.0 as well.
FYI
On Mon, Oct 26, 2015 at 2:26 PM, Silvio Fiorito <
silvio.fiorito@granturing.com> wrote:
> Hi Matthias,
>
> Unless there was a change in 1.5, I'm afraid dynamic resource allocation
> is not yet supported in streaming apps.
>
> Thanks,
> Silvio
>
> Sent from my Lumia 930
> ------------------------------
> From: Matthias Niehoff <ma...@codecentric.de>
> Sent: 10/26/2015 4:00 PM
> To: user@spark.apache.org
> Subject: Dynamic Resource Allocation with Spark Streaming (Standalone
> Cluster, Spark 1.5.1)
>
> Hello everybody,
>
> I have a few (~15) Spark Streaming jobs which have load peaks as well as
> long times with a low load. So I thought the new Dynamic Resource
> Allocation for Standalone Clusters might be helpful (SPARK-4751).
>
> I have a test "cluster" with 1 worker consisting of 4 executors with 2
> cores each, so 8 cores in total.
>
> I started a simple streaming application without limiting the max cores
> for this app. As expected the app occupied every core of the cluster. Then
> I started a second app, also without limiting the maximum cores. As the
> first app did not get any input through the stream, my naive expectation
> was that the second app would get at least 2 cores (1 receiver, 1
> processing), but that's not what happened. The cores are still assigned to
> the first app.
> When I look at the application UI of the first app every executor is still
> running. That explains why no executor is used for the second app.
>
> I end up with two questions:
> - When does an executor getting idle in a Spark Streaming application?
> (and so could be reassigned to another app)
> - Is there another way to compete with uncertain load when using Spark
> Streaming Applications? I already combined multiple jobs to a Spark
> Application using different threads, but this approach comes to a limit for
> me, because Spark Applications get to big to manage.
>
> Thank You!
>
>
>
RE: Dynamic Resource Allocation with Spark Streaming (Standalone
Cluster, Spark 1.5.1)
Posted by Silvio Fiorito <si...@granturing.com>.
Hi Matthias,
Unless there was a change in 1.5, I'm afraid dynamic resource allocation is not yet supported in streaming apps.
Thanks,
Silvio
Sent from my Lumia 930
________________________________
From: Matthias Niehoff<ma...@codecentric.de>
Sent: 10/26/2015 4:00 PM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Dynamic Resource Allocation with Spark Streaming (Standalone Cluster, Spark 1.5.1)
Hello everybody,
I have a few (~15) Spark Streaming jobs which have load peaks as well as long times with a low load. So I thought the new Dynamic Resource Allocation for Standalone Clusters might be helpful (SPARK-4751).
I have a test "cluster" with 1 worker consisting of 4 executors with 2 cores each, so 8 cores in total.
I started a simple streaming application without limiting the max cores for this app. As expected the app occupied every core of the cluster. Then I started a second app, also without limiting the maximum cores. As the first app did not get any input through the stream, my naive expectation was that the second app would get at least 2 cores (1 receiver, 1 processing), but that's not what happened. The cores are still assigned to the first app.
When I look at the application UI of the first app every executor is still running. That explains why no executor is used for the second app.
I end up with two questions:
- When does an executor getting idle in a Spark Streaming application? (and so could be reassigned to another app)
- Is there another way to compete with uncertain load when using Spark Streaming Applications? I already combined multiple jobs to a Spark Application using different threads, but this approach comes to a limit for me, because Spark Applications get to big to manage.
Thank You!