You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nishant Kumar <ni...@applift.com> on 2016/12/28 04:19:39 UTC
Spark streaming with Yarn: executors not fully utilized
Hi,
I am running spark streaming with Yarn with -
*spark-submit --master yarn --deploy-mode cluster --num-executors 2
--executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
I am consuming Kafka through DireactStream approach (No receiver). I have 2
topics (each with 3 partitions).
I reparation RDD (i have one DStream) into 16 parts (assuming no of
executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
foreachPartition and writes each partition to local file and then send it
to other server (not spark) through http (Using apache http client post
with multi-part).
*When i checked details of this step (or JOB is it correct naming?) through
Spark UI, it showed that total 16 task executed on single executor with 8
task at a time.*
This is Spark UI details -
*Details for Stage 717 (Attempt 0)*
Index ID Attempt Status Locality Level Executor ID / Host Launch
Time Duration GC Time Shuffle Read Size / Records Errors
0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 317.7 KB / 6245
9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 320.4 KB / 6280
10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 323.0 KB / 6334
11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 323.7 KB / 6371
12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 316.7 KB / 6218
13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 321.0 KB / 6301
14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name
2016/12/27 12:11:48 2 s 321.4 KB / 6304
I was expecting it to execute 16 parallel task (2 executor * 8 core) on
either one or more executor. I think i am missing something. Please help.
--
*Nishant Kumar*
Senior Software Engineer
Phone: +91 80088 42030
Skype: nishant.kumar_applift
*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>
--
Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3
Click here <http://www.applift.com> to see all the events we will be
attending.
Re: Spark streaming with Yarn: executors not fully utilized
Posted by Nishant Kumar <ni...@applift.com>.
Any update on this guys ?
On Wed, Dec 28, 2016 at 10:19 AM, Nishant Kumar <ni...@applift.com>
wrote:
> I have updated my question:
>
> http://stackoverflow.com/questions/41345552/spark-
> streaming-with-yarn-executors-not-fully-utilized
>
> On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <ni...@applift.com>
> wrote:
>
>> Hi,
>>
>> I am running spark streaming with Yarn with -
>>
>> *spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
>>
>> I am consuming Kafka through DireactStream approach (No receiver). I
>> have 2 topics (each with 3 partitions).
>>
>> I reparation RDD (i have one DStream) into 16 parts (assuming no of
>> executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
>> foreachPartition and writes each partition to local file and then send it
>> to other server (not spark) through http (Using apache http client post
>> with multi-part).
>>
>> *When i checked details of this step (or JOB is it correct naming?)
>> through Spark UI, it showed that total 16 task executed on single executor
>> with 8 task at a time.*
>>
>> This is Spark UI details -
>>
>> *Details for Stage 717 (Attempt 0)*
>>
>> Index ID Attempt Status Locality Level Executor ID / Host Launch Time Duration GC Time Shuffle Read Size / Records Errors
>> 0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
>> 1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
>> 2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
>> 3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
>> 4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
>> 5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
>> 6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
>> 7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
>> 8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 317.7 KB / 6245
>> 9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 320.4 KB / 6280
>> 10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.0 KB / 6334
>> 11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.7 KB / 6371
>> 12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 316.7 KB / 6218
>> 13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.0 KB / 6301
>> 14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.4 KB / 6304
>>
>> I was expecting it to execute 16 parallel task (2 executor * 8 core) on
>> either one or more executor. I think i am missing something. Please help.
>>
>>
>> --
>> *Nishant Kumar*
>> Senior Software Engineer
>>
>> Phone: +91 80088 42030
>> Skype: nishant.kumar_applift
>>
>>
>> *AppLift India*
>> 107/3, 80 Feet Main Road,
>> Koramangala 4th Block,
>> Bangalore - 560034
>> www.AppLift.com <http://www.applift.com/>
>>
>
>
>
> --
> *Nishant Kumar*
> Senior Software Engineer
>
> Phone: +91 80088 42030
> Skype: nishant.kumar_applift
>
>
> *AppLift India*
> 107/3, 80 Feet Main Road,
> Koramangala 4th Block,
> Bangalore - 560034
> www.AppLift.com <http://www.applift.com/>
>
--
*Nishant Kumar*
Senior Software Engineer
Phone: +91 80088 42030
Skype: nishant.kumar_applift
*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>
--
Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3
Click here <http://www.applift.com> to see all the events we will be
attending.
Re: Spark streaming with Yarn: executors not fully utilized
Posted by Nishant Kumar <ni...@applift.com>.
I have updated my question:
http://stackoverflow.com/questions/41345552/spark-streaming-with-yarn-executors-not-fully-utilized
On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <ni...@applift.com>
wrote:
> Hi,
>
> I am running spark streaming with Yarn with -
>
> *spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
>
> I am consuming Kafka through DireactStream approach (No receiver). I have
> 2 topics (each with 3 partitions).
>
> I reparation RDD (i have one DStream) into 16 parts (assuming no of
> executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do
> foreachPartition and writes each partition to local file and then send it
> to other server (not spark) through http (Using apache http client post
> with multi-part).
>
> *When i checked details of this step (or JOB is it correct naming?)
> through Spark UI, it showed that total 16 task executed on single executor
> with 8 task at a time.*
>
> This is Spark UI details -
>
> *Details for Stage 717 (Attempt 0)*
>
> Index ID Attempt Status Locality Level Executor ID / Host Launch Time Duration GC Time Shuffle Read Size / Records Errors
> 0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 313.3 KB / 6137
> 1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 328.5 KB / 6452
> 2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.3 KB / 6364
> 3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 321.5 KB / 6306
> 4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 324.8 KB / 6364
> 5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 320.8 KB / 6307
> 6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 2 s 11 ms 323.4 KB / 6356
> 7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:46 3 s 11 ms 316.8 KB / 6207
> 8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 317.7 KB / 6245
> 9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 320.4 KB / 6280
> 10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.0 KB / 6334
> 11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 323.7 KB / 6371
> 12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 316.7 KB / 6218
> 13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.0 KB / 6301
> 14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 12:11:48 2 s 321.4 KB / 6304
>
> I was expecting it to execute 16 parallel task (2 executor * 8 core) on
> either one or more executor. I think i am missing something. Please help.
>
>
> --
> *Nishant Kumar*
> Senior Software Engineer
>
> Phone: +91 80088 42030
> Skype: nishant.kumar_applift
>
>
> *AppLift India*
> 107/3, 80 Feet Main Road,
> Koramangala 4th Block,
> Bangalore - 560034
> www.AppLift.com <http://www.applift.com/>
>
--
*Nishant Kumar*
Senior Software Engineer
Phone: +91 80088 42030
Skype: nishant.kumar_applift
*AppLift India*
107/3, 80 Feet Main Road,
Koramangala 4th Block,
Bangalore - 560034
www.AppLift.com <http://www.applift.com/>
--
Meet us at:
@ *PG Connects*, London, Jan 16-17
@ *Mobile Games Forum*, London, Jan 17-18
@ *GMASA*, Jakarta, Jan 26
@* Casual Connect*, Berlin, Feb 7-9
@* Mobile World Congress*, Barcelona, Feb 27-Mar 2
@* GDC*, San Francisco, Feb 27-Mar 3
Click here <http://www.applift.com> to see all the events we will be
attending.