You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by thomas lavocat <th...@univ-grenoble-alpes.fr> on 2018/06/05 08:20:27 UTC
[Spark Streaming] is spark.streaming.concurrentJobs a per node or a
cluster global value ?
Hi everyone,
I'm wondering if the property spark.streaming.concurrentJobs should
reflects the total number of possible concurrent task on the cluster, or
the a local number of concurrent tasks on one compute node.
Thanks for your help.
Thomas
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by thomas lavocat <th...@univ-grenoble-alpes.fr>.
Thank you very much for your answer.
Since I don't have dependent jobs I will continue to use this functionality.
On 05/06/2018 13:52, Saisai Shao wrote:
> "dependent" I mean this batch's job relies on the previous batch's
> result. So this batch should wait for the finish of previous batch, if
> you set "spark.streaming.concurrentJobs" larger than 1, then the
> current batch could start without waiting for the previous batch (if
> it is delayed), which will lead to unexpected results.
>
>
> thomas lavocat <thomas.lavocat@univ-grenoble-alpes.fr
> <ma...@univ-grenoble-alpes.fr>> 于2018年6月5日周二
> 下午7:48写道：
>
>
> On 05/06/2018 13:44, Saisai Shao wrote:
>> You need to read the code, this is an undocumented configuration.
> I'm on it right now, but, Spark is a big piece of software.
>> Basically this will break the ordering of Streaming jobs, AFAIK
>> it may get unexpected results if you streaming jobs are not
>> independent.
> What do you mean exactly by not independent ?
> Are several source joined together dependent ?
>
> Thanks,
> Thomas
>>
>> thomas lavocat <thomas.lavocat@univ-grenoble-alpes.fr
>> <ma...@univ-grenoble-alpes.fr>> 于2018年6月5日周二
>> 下午7:17写道：
>>
>> Hello,
>>
>> Thank's for your answer.
>>
>>
>> On 05/06/2018 11:24, Saisai Shao wrote:
>>> spark.streaming.concurrentJobs is a driver side internal
>>> configuration, this means that how many streaming jobs can
>>> be submitted concurrently in one batch. Usually this should
>>> not be configured by user, unless you're familiar with Spark
>>> Streaming internals, and know the implication of this
>>> configuration.
>>
>> How can I find some documentation about those implications ?
>>
>> I've experimented some configuration of this parameters and
>> found out that my overall throughput is increased in
>> correlation with this property.
>> But I'm experiencing scalability issues. With more than 16
>> receivers spread over 8 executors, my executors no longer
>> receive work from the driver and fall idle.
>> Is there an explanation ?
>>
>> Thanks,
>> Thomas
>>
>
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by Saisai Shao <sa...@gmail.com>.
"dependent" I mean this batch's job relies on the previous batch's result.
So this batch should wait for the finish of previous batch, if you set "
spark.streaming.concurrentJobs" larger than 1, then the current batch could
start without waiting for the previous batch (if it is delayed), which will
lead to unexpected results.
thomas lavocat <th...@univ-grenoble-alpes.fr> 于2018年6月5日周二
下午7:48写道：
>
> On 05/06/2018 13:44, Saisai Shao wrote:
>
> You need to read the code, this is an undocumented configuration.
>
> I'm on it right now, but, Spark is a big piece of software.
>
> Basically this will break the ordering of Streaming jobs, AFAIK it may get
> unexpected results if you streaming jobs are not independent.
>
> What do you mean exactly by not independent ?
> Are several source joined together dependent ?
>
> Thanks,
> Thomas
>
>
> thomas lavocat <th...@univ-grenoble-alpes.fr> 于2018年6月5日周二
> 下午7:17写道：
>
>> Hello,
>>
>> Thank's for your answer.
>>
>> On 05/06/2018 11:24, Saisai Shao wrote:
>>
>> spark.streaming.concurrentJobs is a driver side internal configuration,
>> this means that how many streaming jobs can be submitted concurrently in
>> one batch. Usually this should not be configured by user, unless you're
>> familiar with Spark Streaming internals, and know the implication of this
>> configuration.
>>
>>
>> How can I find some documentation about those implications ?
>>
>> I've experimented some configuration of this parameters and found out
>> that my overall throughput is increased in correlation with this property.
>> But I'm experiencing scalability issues. With more than 16 receivers
>> spread over 8 executors, my executors no longer receive work from the
>> driver and fall idle.
>> Is there an explanation ?
>>
>> Thanks,
>> Thomas
>>
>>
>
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by thomas lavocat <th...@univ-grenoble-alpes.fr>.
On 05/06/2018 13:44, Saisai Shao wrote:
> You need to read the code, this is an undocumented configuration.
I'm on it right now, but, Spark is a big piece of software.
> Basically this will break the ordering of Streaming jobs, AFAIK it may
> get unexpected results if you streaming jobs are not independent.
What do you mean exactly by not independent ?
Are several source joined together dependent ?
Thanks,
Thomas
>
> thomas lavocat <thomas.lavocat@univ-grenoble-alpes.fr
> <ma...@univ-grenoble-alpes.fr>> 于2018年6月5日周二
> 下午7:17写道：
>
> Hello,
>
> Thank's for your answer.
>
>
> On 05/06/2018 11:24, Saisai Shao wrote:
>> spark.streaming.concurrentJobs is a driver side internal
>> configuration, this means that how many streaming jobs can be
>> submitted concurrently in one batch. Usually this should not be
>> configured by user, unless you're familiar with Spark Streaming
>> internals, and know the implication of this configuration.
>
> How can I find some documentation about those implications ?
>
> I've experimented some configuration of this parameters and found
> out that my overall throughput is increased in correlation with
> this property.
> But I'm experiencing scalability issues. With more than 16
> receivers spread over 8 executors, my executors no longer receive
> work from the driver and fall idle.
> Is there an explanation ?
>
> Thanks,
> Thomas
>
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by Saisai Shao <sa...@gmail.com>.
You need to read the code, this is an undocumented configuration.
Basically this will break the ordering of Streaming jobs, AFAIK it may get
unexpected results if you streaming jobs are not independent.
thomas lavocat <th...@univ-grenoble-alpes.fr> 于2018年6月5日周二
下午7:17写道：
> Hello,
>
> Thank's for your answer.
>
> On 05/06/2018 11:24, Saisai Shao wrote:
>
> spark.streaming.concurrentJobs is a driver side internal configuration,
> this means that how many streaming jobs can be submitted concurrently in
> one batch. Usually this should not be configured by user, unless you're
> familiar with Spark Streaming internals, and know the implication of this
> configuration.
>
>
> How can I find some documentation about those implications ?
>
> I've experimented some configuration of this parameters and found out that
> my overall throughput is increased in correlation with this property.
> But I'm experiencing scalability issues. With more than 16 receivers
> spread over 8 executors, my executors no longer receive work from the
> driver and fall idle.
> Is there an explanation ?
>
> Thanks,
> Thomas
>
>
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by thomas lavocat <th...@univ-grenoble-alpes.fr>.
Hello,
Thank's for your answer.
On 05/06/2018 11:24, Saisai Shao wrote:
> spark.streaming.concurrentJobs is a driver side internal
> configuration, this means that how many streaming jobs can be
> submitted concurrently in one batch. Usually this should not be
> configured by user, unless you're familiar with Spark Streaming
> internals, and know the implication of this configuration.
How can I find some documentation about those implications ?
I've experimented some configuration of this parameters and found out
that my overall throughput is increased in correlation with this property.
But I'm experiencing scalability issues. With more than 16 receivers
spread over 8 executors, my executors no longer receive work from the
driver and fall idle.
Is there an explanation ?
Thanks,
Thomas
Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or
a cluster global value ?
Posted by Saisai Shao <sa...@gmail.com>.
spark.streaming.concurrentJobs is a driver side internal configuration,
this means that how many streaming jobs can be submitted concurrently in
one batch. Usually this should not be configured by user, unless you're
familiar with Spark Streaming internals, and know the implication of this
configuration.
thomas lavocat <th...@univ-grenoble-alpes.fr> 于2018年6月5日周二
下午4:20写道：
> Hi everyone,
>
> I'm wondering if the property spark.streaming.concurrentJobs should
> reflects the total number of possible concurrent task on the cluster, or
> the a local number of concurrent tasks on one compute node.
>
> Thanks for your help.
>
> Thomas
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>