You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nishant Kumar <ni...@applift.com> on 2016/12/28 04:19:39 UTC

Spark streaming with Yarn: executors not fully utilized

 Hi,

I am running spark streaming with Yarn with -

*spark-submit --master yarn --deploy-mode cluster --num-executors 2
--executor-memory 8g --driver-memory 2g --executor-cores 8 ..*

I am consuming Kafka through DireactStream approach (No receiver). I have 2
topics (each with 3 partitions).

I reparation RDD (i have one DStream) into 16 parts (assuming no of
executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
foreachPartition and writes each partition to local file and then send it
to other server (not spark) through http (Using apache http client post
with multi-part).

*When i checked details of this step (or JOB is it correct naming?) through
Spark UI, it showed that total 16 task executed on single executor with 8
task at a time.*

This is Spark UI details -

*Details for Stage 717 (Attempt 0)*

Index  ID  Attempt Status  Locality Level  Executor ID / Host  Launch
Time Duration  GC Time Shuffle Read Size / Records Errors
0  5080  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
1  5081  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
2  5082  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
3  5083  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
4  5084  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
5  5085  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
6  5086  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
7  5087  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
8  5088  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   317.7 KB / 6245
9  5089  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   320.4 KB / 6280
10  5090  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   323.0 KB / 6334
11  5091  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   323.7 KB / 6371
12  5092  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   316.7 KB / 6218
13  5093  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   321.0 KB / 6301
14  5094  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s   321.4 KB / 6304

I was expecting it to execute 16 parallel task (2 executor * 8 core) on
either one or more executor. I think i am missing something. Please help.


-- 
*Nishant Kumar*
Senior Software Engineer

Phone: +91 80088 42030
Skype: nishant.kumar_applift


*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>

-- 


Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3

Click here <http://www.applift.com> to see all the events we will be 
attending. 

Re: Spark streaming with Yarn: executors not fully utilized

Posted by Nishant Kumar <ni...@applift.com>.
Any update on this guys ?

On Wed, Dec 28, 2016 at 10:19 AM, Nishant Kumar <ni...@applift.com>
wrote:

> I have updated my question:
>
> http://stackoverflow.com/questions/41345552/spark-
> streaming-with-yarn-executors-not-fully-utilized
>
> On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <ni...@applift.com>
> wrote:
>
>> Hi,
>>
>> I am running spark streaming with Yarn with -
>>
>> *spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
>>
>> I am consuming Kafka through DireactStream approach (No receiver). I
>> have 2 topics (each with 3 partitions).
>>
>> I reparation RDD (i have one DStream) into 16 parts (assuming no of
>> executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
>> foreachPartition and writes each partition to local file and then send it
>> to other server (not spark) through http (Using apache http client post
>> with multi-part).
>>
>> *When i checked details of this step (or JOB is it correct naming?)
>> through Spark UI, it showed that total 16 task executed on single executor
>> with 8 task at a time.*
>>
>> This is Spark UI details -
>>
>> *Details for Stage 717 (Attempt 0)*
>>
>> Index  ID  Attempt Status  Locality Level  Executor ID / Host  Launch Time Duration  GC Time Shuffle Read Size / Records Errors
>> 0  5080  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
>> 1  5081  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
>> 2  5082  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
>> 3  5083  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
>> 4  5084  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
>> 5  5085  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
>> 6  5086  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
>> 7  5087  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
>> 8  5088  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   317.7 KB / 6245
>> 9  5089  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   320.4 KB / 6280
>> 10  5090  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   323.0 KB / 6334
>> 11  5091  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   323.7 KB / 6371
>> 12  5092  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   316.7 KB / 6218
>> 13  5093  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   321.0 KB / 6301
>> 14  5094  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   321.4 KB / 6304
>>
>> I was expecting it to execute 16 parallel task (2 executor * 8 core) on
>> either one or more executor. I think i am missing something. Please help.
>>
>>
>> --
>> *Nishant Kumar*
>> Senior Software Engineer
>>
>> Phone: +91 80088 42030
>> Skype: nishant.kumar_applift
>>
>>
>> *AppLift India*
>> 107/3, 80 Feet Main Road,
>> Koramangala 4th Block,
>> Bangalore - 560034
>> www.AppLift.com <http://www.applift.com/>
>>
>
>
>
> --
> *Nishant Kumar*
> Senior Software Engineer
>
> Phone: +91 80088 42030
> Skype: nishant.kumar_applift
>
>
> *AppLift India*
> 107/3, 80 Feet Main Road,
> Koramangala 4th Block,
> Bangalore - 560034
> www.AppLift.com <http://www.applift.com/>
>



-- 
*Nishant Kumar*
Senior Software Engineer

Phone: +91 80088 42030
Skype: nishant.kumar_applift


*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>

-- 


Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3

Click here <http://www.applift.com> to see all the events we will be 
attending. 

Re: Spark streaming with Yarn: executors not fully utilized

Posted by Nishant Kumar <ni...@applift.com>.
I have updated my question:

http://stackoverflow.com/questions/41345552/spark-streaming-with-yarn-executors-not-fully-utilized

On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <ni...@applift.com>
wrote:

> Hi,
>
> I am running spark streaming with Yarn with -
>
> *spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
>
> I am consuming Kafka through DireactStream approach (No receiver). I have
> 2 topics (each with 3 partitions).
>
> I reparation RDD (i have one DStream) into 16 parts (assuming no of
> executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
> foreachPartition and writes each partition to local file and then send it
> to other server (not spark) through http (Using apache http client post
> with multi-part).
>
> *When i checked details of this step (or JOB is it correct naming?)
> through Spark UI, it showed that total 16 task executed on single executor
> with 8 task at a time.*
>
> This is Spark UI details -
>
> *Details for Stage 717 (Attempt 0)*
>
> Index  ID  Attempt Status  Locality Level  Executor ID / Host  Launch Time Duration  GC Time Shuffle Read Size / Records Errors
> 0  5080  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
> 1  5081  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
> 2  5082  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
> 3  5083  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
> 4  5084  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
> 5  5085  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
> 6  5086  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
> 7  5087  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
> 8  5088  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   317.7 KB / 6245
> 9  5089  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   320.4 KB / 6280
> 10  5090  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   323.0 KB / 6334
> 11  5091  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   323.7 KB / 6371
> 12  5092  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   316.7 KB / 6218
> 13  5093  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   321.0 KB / 6301
> 14  5094  0 SUCCESS NODE_LOCAL  1 / executor1_machine_host_name  2016/12/27 12:11:48 2 s   321.4 KB / 6304
>
> I was expecting it to execute 16 parallel task (2 executor * 8 core) on
> either one or more executor. I think i am missing something. Please help.
>
>
> --
> *Nishant Kumar*
> Senior Software Engineer
>
> Phone: +91 80088 42030
> Skype: nishant.kumar_applift
>
>
> *AppLift India*
> 107/3, 80 Feet Main Road,
> Koramangala 4th Block,
> Bangalore - 560034
> www.AppLift.com <http://www.applift.com/>
>



-- 
*Nishant Kumar*
Senior Software Engineer

Phone: +91 80088 42030
Skype: nishant.kumar_applift


*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>

-- 


Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3

Click here <http://www.applift.com> to see all the events we will be 
attending.