You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by chandan prakash <ch...@gmail.com> on 2016/05/01 04:29:38 UTC
spark.streaming.concurrentJobs parameter in Spark Streaming
I have a doubt regarding this spark.streaming.concurrentJobs parameter.
Setting
spark.streaming.concurrentJobs =2
i can see 2 parallel jobs getting processed.
But setting
spark.streaming.concurrentJobs =10,
*i can still see only 4-5 concurrent jobs getting processed and other jobs
getting queued while many cores in my machine remain ideal*.
I have 2 machines of 24 cores each for spark processing.
*Any reason why jobs getting queued when concurrent jobs parameter allows
more jobs and there are available cores as well ?*
Thanks,
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
i figured it out. :)
My observation was wrong.
i was looking at concurrent batches being executed while i should have
looked ta concurrent jobs being executed.
Each batch can have multiple jobs within itself depending upon the code
logic e.g. for my case, every batch has 4 jobs.
So at any given time, total 10 jobs indeed were running ( as per
spark.streaming.concurrentJobs
=10 parameter ) which can span over 3 or more batches.
On Sun, May 1, 2016 at 7:59 AM, chandan prakash <ch...@gmail.com>
wrote:
> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>
> Setting
> spark.streaming.concurrentJobs =2
> i can see 2 parallel jobs getting processed.
>
> But setting
> spark.streaming.concurrentJobs =10,
> *i can still see only 4-5 concurrent jobs getting processed and other
> jobs getting queued while many cores in my machine remain ideal*.
> I have 2 machines of 24 cores each for spark processing.
> *Any reason why jobs getting queued when concurrent jobs parameter allows
> more jobs and there are available cores as well ?*
>
>
>
>
>
> Thanks,
>
> --
> Chandan Prakash
>
>
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
executors tab show info about tasks and executor id . there are 2 executors
in my case as there are 2 machine node, each with 24 cores and 64gb ram.
if you meant Environment tab, then yes show number of concurrent jobs same
as what i assigned e.g. 10
Thanks,
Chandan
On Mon, May 2, 2016 at 2:45 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Can you check in the executor tab and see the resources allocated?
>
> Thanks
> Best Regards
>
> On Mon, May 2, 2016 at 1:18 PM, chandan prakash <chandanbaranwal@gmail.com
> > wrote:
>
>> Any suggestions?
>> i want to increase number of concurrent jobs actually getting processed.
>>
>> Regards,
>> Chandan
>>
>> On Sun, May 1, 2016 at 5:29 PM, chandan prakash <
>> chandanbaranwal@gmail.com> wrote:
>>
>>> few more details....
>>> my spark set up includes 2 machines...running under Mesos as resource
>>> manager....running 1 executor on each spark machine with 24 cores/64gb
>>> ram.....
>>>
>>> On Sun, May 1, 2016 at 5:21 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> ok I will test this
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 1 May 2016 at 11:25, chandan prakash <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> sparkConf.set("spark.driver.allowMultipleContexts", "true")
>>>>>
>>>>> sparkConf.set("spark.streaming.blockInterval", "400ms")
>>>>>
>>>>> *sparkConf.set("spark.streaming.concurrentJobs","10")*
>>>>>
>>>>> On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> How do you define this parameter in below?
>>>>>>
>>>>>> val sparkConf = new SparkConf().
>>>>>> setAppName("CEP_streaming").
>>>>>> setMaster("local[12]").
>>>>>> set("spark.cores.max", "2").
>>>>>> set("spark.streaming.concurrentJobs", "2").
>>>>>> set("spark.driver.allowMultipleContexts", "true").
>>>>>> set("spark.hadoop.validateOutputSpecs", "false")
>>>>>>
>>>>>> Dr Mich Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://talebzadehmich.wordpress.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> attaching screenshot :
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>>>>>>> chandanbaranwal@gmail.com> wrote:
>>>>>>>
>>>>>>>> I have a doubt regarding this spark.streaming.concurrentJobs
>>>>>>>> parameter.
>>>>>>>>
>>>>>>>> Setting
>>>>>>>> spark.streaming.concurrentJobs =2
>>>>>>>> i can see 2 parallel jobs getting processed.
>>>>>>>>
>>>>>>>> But setting
>>>>>>>> spark.streaming.concurrentJobs =10,
>>>>>>>> *i can still see only 4-5 concurrent jobs getting processed and
>>>>>>>> other jobs getting queued while many cores in my machine remain ideal*
>>>>>>>> .
>>>>>>>> I have 2 machines of 24 cores each for spark processing.
>>>>>>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>>>>>>> allows more jobs and there are available cores as well ?*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chandan Prakash
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chandan Prakash
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chandan Prakash
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Chandan Prakash
>>>
>>>
>>
>>
>> --
>> Chandan Prakash
>>
>>
>
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you check in the executor tab and see the resources allocated?
Thanks
Best Regards
On Mon, May 2, 2016 at 1:18 PM, chandan prakash <ch...@gmail.com>
wrote:
> Any suggestions?
> i want to increase number of concurrent jobs actually getting processed.
>
> Regards,
> Chandan
>
> On Sun, May 1, 2016 at 5:29 PM, chandan prakash <chandanbaranwal@gmail.com
> > wrote:
>
>> few more details....
>> my spark set up includes 2 machines...running under Mesos as resource
>> manager....running 1 executor on each spark machine with 24 cores/64gb
>> ram.....
>>
>> On Sun, May 1, 2016 at 5:21 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> ok I will test this
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 1 May 2016 at 11:25, chandan prakash <ch...@gmail.com>
>>> wrote:
>>>
>>>> sparkConf.set("spark.driver.allowMultipleContexts", "true")
>>>>
>>>> sparkConf.set("spark.streaming.blockInterval", "400ms")
>>>>
>>>> *sparkConf.set("spark.streaming.concurrentJobs","10")*
>>>>
>>>> On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> How do you define this parameter in below?
>>>>>
>>>>> val sparkConf = new SparkConf().
>>>>> setAppName("CEP_streaming").
>>>>> setMaster("local[12]").
>>>>> set("spark.cores.max", "2").
>>>>> set("spark.streaming.concurrentJobs", "2").
>>>>> set("spark.driver.allowMultipleContexts", "true").
>>>>> set("spark.hadoop.validateOutputSpecs", "false")
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>>
>>>>> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> attaching screenshot :
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>>>>>> chandanbaranwal@gmail.com> wrote:
>>>>>>
>>>>>>> I have a doubt regarding this spark.streaming.concurrentJobs
>>>>>>> parameter.
>>>>>>>
>>>>>>> Setting
>>>>>>> spark.streaming.concurrentJobs =2
>>>>>>> i can see 2 parallel jobs getting processed.
>>>>>>>
>>>>>>> But setting
>>>>>>> spark.streaming.concurrentJobs =10,
>>>>>>> *i can still see only 4-5 concurrent jobs getting processed and
>>>>>>> other jobs getting queued while many cores in my machine remain ideal*
>>>>>>> .
>>>>>>> I have 2 machines of 24 cores each for spark processing.
>>>>>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>>>>>> allows more jobs and there are available cores as well ?*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> Chandan Prakash
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chandan Prakash
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Chandan Prakash
>>>>
>>>>
>>>
>>
>>
>> --
>> Chandan Prakash
>>
>>
>
>
> --
> Chandan Prakash
>
>
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
Any suggestions?
i want to increase number of concurrent jobs actually getting processed.
Regards,
Chandan
On Sun, May 1, 2016 at 5:29 PM, chandan prakash <ch...@gmail.com>
wrote:
> few more details....
> my spark set up includes 2 machines...running under Mesos as resource
> manager....running 1 executor on each spark machine with 24 cores/64gb
> ram.....
>
> On Sun, May 1, 2016 at 5:21 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> ok I will test this
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 1 May 2016 at 11:25, chandan prakash <ch...@gmail.com>
>> wrote:
>>
>>> sparkConf.set("spark.driver.allowMultipleContexts", "true")
>>>
>>> sparkConf.set("spark.streaming.blockInterval", "400ms")
>>>
>>> *sparkConf.set("spark.streaming.concurrentJobs","10")*
>>>
>>> On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> How do you define this parameter in below?
>>>>
>>>> val sparkConf = new SparkConf().
>>>> setAppName("CEP_streaming").
>>>> setMaster("local[12]").
>>>> set("spark.cores.max", "2").
>>>> set("spark.streaming.concurrentJobs", "2").
>>>> set("spark.driver.allowMultipleContexts", "true").
>>>> set("spark.hadoop.validateOutputSpecs", "false")
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> attaching screenshot :
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>>>>> chandanbaranwal@gmail.com> wrote:
>>>>>
>>>>>> I have a doubt regarding this spark.streaming.concurrentJobs
>>>>>> parameter.
>>>>>>
>>>>>> Setting
>>>>>> spark.streaming.concurrentJobs =2
>>>>>> i can see 2 parallel jobs getting processed.
>>>>>>
>>>>>> But setting
>>>>>> spark.streaming.concurrentJobs =10,
>>>>>> *i can still see only 4-5 concurrent jobs getting processed and
>>>>>> other jobs getting queued while many cores in my machine remain ideal*
>>>>>> .
>>>>>> I have 2 machines of 24 cores each for spark processing.
>>>>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>>>>> allows more jobs and there are available cores as well ?*
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --
>>>>>> Chandan Prakash
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chandan Prakash
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Chandan Prakash
>>>
>>>
>>
>
>
> --
> Chandan Prakash
>
>
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
few more details....
my spark set up includes 2 machines...running under Mesos as resource
manager....running 1 executor on each spark machine with 24 cores/64gb
ram.....
On Sun, May 1, 2016 at 5:21 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> ok I will test this
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 1 May 2016 at 11:25, chandan prakash <ch...@gmail.com> wrote:
>
>> sparkConf.set("spark.driver.allowMultipleContexts", "true")
>>
>> sparkConf.set("spark.streaming.blockInterval", "400ms")
>>
>> *sparkConf.set("spark.streaming.concurrentJobs","10")*
>>
>> On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> How do you define this parameter in below?
>>>
>>> val sparkConf = new SparkConf().
>>> setAppName("CEP_streaming").
>>> setMaster("local[12]").
>>> set("spark.cores.max", "2").
>>> set("spark.streaming.concurrentJobs", "2").
>>> set("spark.driver.allowMultipleContexts", "true").
>>> set("spark.hadoop.validateOutputSpecs", "false")
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com>
>>> wrote:
>>>
>>>> attaching screenshot :
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>>>> chandanbaranwal@gmail.com> wrote:
>>>>
>>>>> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>>>>>
>>>>> Setting
>>>>> spark.streaming.concurrentJobs =2
>>>>> i can see 2 parallel jobs getting processed.
>>>>>
>>>>> But setting
>>>>> spark.streaming.concurrentJobs =10,
>>>>> *i can still see only 4-5 concurrent jobs getting processed and
>>>>> other jobs getting queued while many cores in my machine remain ideal*
>>>>> .
>>>>> I have 2 machines of 24 cores each for spark processing.
>>>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>>>> allows more jobs and there are available cores as well ?*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> Chandan Prakash
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Chandan Prakash
>>>>
>>>>
>>>
>>
>>
>> --
>> Chandan Prakash
>>
>>
>
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
ok I will test this
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 1 May 2016 at 11:25, chandan prakash <ch...@gmail.com> wrote:
> sparkConf.set("spark.driver.allowMultipleContexts", "true")
>
> sparkConf.set("spark.streaming.blockInterval", "400ms")
>
> *sparkConf.set("spark.streaming.concurrentJobs","10")*
>
> On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> How do you define this parameter in below?
>>
>> val sparkConf = new SparkConf().
>> setAppName("CEP_streaming").
>> setMaster("local[12]").
>> set("spark.cores.max", "2").
>> set("spark.streaming.concurrentJobs", "2").
>> set("spark.driver.allowMultipleContexts", "true").
>> set("spark.hadoop.validateOutputSpecs", "false")
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com>
>> wrote:
>>
>>> attaching screenshot :
>>>
>>>
>>>
>>>
>>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>>> chandanbaranwal@gmail.com> wrote:
>>>
>>>> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>>>>
>>>> Setting
>>>> spark.streaming.concurrentJobs =2
>>>> i can see 2 parallel jobs getting processed.
>>>>
>>>> But setting
>>>> spark.streaming.concurrentJobs =10,
>>>> *i can still see only 4-5 concurrent jobs getting processed and other
>>>> jobs getting queued while many cores in my machine remain ideal*.
>>>> I have 2 machines of 24 cores each for spark processing.
>>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>>> allows more jobs and there are available cores as well ?*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Chandan Prakash
>>>>
>>>>
>>>
>>>
>>> --
>>> Chandan Prakash
>>>
>>>
>>
>
>
> --
> Chandan Prakash
>
>
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
sparkConf.set("spark.driver.allowMultipleContexts", "true")
sparkConf.set("spark.streaming.blockInterval", "400ms")
*sparkConf.set("spark.streaming.concurrentJobs","10")*
On Sun, May 1, 2016 at 1:48 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> How do you define this parameter in below?
>
> val sparkConf = new SparkConf().
> setAppName("CEP_streaming").
> setMaster("local[12]").
> set("spark.cores.max", "2").
> set("spark.streaming.concurrentJobs", "2").
> set("spark.driver.allowMultipleContexts", "true").
> set("spark.hadoop.validateOutputSpecs", "false")
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com> wrote:
>
>> attaching screenshot :
>>
>>
>>
>>
>> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <
>> chandanbaranwal@gmail.com> wrote:
>>
>>> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>>>
>>> Setting
>>> spark.streaming.concurrentJobs =2
>>> i can see 2 parallel jobs getting processed.
>>>
>>> But setting
>>> spark.streaming.concurrentJobs =10,
>>> *i can still see only 4-5 concurrent jobs getting processed and other
>>> jobs getting queued while many cores in my machine remain ideal*.
>>> I have 2 machines of 24 cores each for spark processing.
>>> *Any reason why jobs getting queued when concurrent jobs parameter
>>> allows more jobs and there are available cores as well ?*
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> --
>>> Chandan Prakash
>>>
>>>
>>
>>
>> --
>> Chandan Prakash
>>
>>
>
--
Chandan Prakash
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by Mich Talebzadeh <mi...@gmail.com>.
How do you define this parameter in below?
val sparkConf = new SparkConf().
setAppName("CEP_streaming").
setMaster("local[12]").
set("spark.cores.max", "2").
set("spark.streaming.concurrentJobs", "2").
set("spark.driver.allowMultipleContexts", "true").
set("spark.hadoop.validateOutputSpecs", "false")
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 1 May 2016 at 03:43, chandan prakash <ch...@gmail.com> wrote:
> attaching screenshot :
>
>
>
>
> On Sun, May 1, 2016 at 7:59 AM, chandan prakash <chandanbaranwal@gmail.com
> > wrote:
>
>> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>>
>> Setting
>> spark.streaming.concurrentJobs =2
>> i can see 2 parallel jobs getting processed.
>>
>> But setting
>> spark.streaming.concurrentJobs =10,
>> *i can still see only 4-5 concurrent jobs getting processed and other
>> jobs getting queued while many cores in my machine remain ideal*.
>> I have 2 machines of 24 cores each for spark processing.
>> *Any reason why jobs getting queued when concurrent jobs parameter allows
>> more jobs and there are available cores as well ?*
>>
>>
>>
>>
>>
>> Thanks,
>>
>> --
>> Chandan Prakash
>>
>>
>
>
> --
> Chandan Prakash
>
>
Re: spark.streaming.concurrentJobs parameter in Spark Streaming
Posted by chandan prakash <ch...@gmail.com>.
attaching screenshot :
On Sun, May 1, 2016 at 7:59 AM, chandan prakash <ch...@gmail.com>
wrote:
> I have a doubt regarding this spark.streaming.concurrentJobs parameter.
>
> Setting
> spark.streaming.concurrentJobs =2
> i can see 2 parallel jobs getting processed.
>
> But setting
> spark.streaming.concurrentJobs =10,
> *i can still see only 4-5 concurrent jobs getting processed and other
> jobs getting queued while many cores in my machine remain ideal*.
> I have 2 machines of 24 cores each for spark processing.
> *Any reason why jobs getting queued when concurrent jobs parameter allows
> more jobs and there are available cores as well ?*
>
>
>
>
>
> Thanks,
>
> --
> Chandan Prakash
>
>
--
Chandan Prakash