You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by شجاع الرحمن بیگ <sh...@gmail.com> on 2016/09/14 19:11:41 UTC

CPU Consumption of spark process

Hi All,

I have a system with 6 physical cores and each core has 8 hardware threads
resulting in 48 virtual cores. Following are the setting in configuration
files.

*spark-env.sh*
export SPARK_WORKER_CORES=1

*spark-defaults.conf*
spark.driver.cores 1
spark.executor.cores 1
spark.cores.max 1

So it means it should only use 1 virtual core but if we see the output from
the TOP command, some time, it has very huge spikes e.g the CPU consumption
is above 4000 e.g.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.064t  37312 S  4728  6.4   7:11.30 java
....
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.065t  37312 S  1502  6.5   8:22.75 java
...
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.065t  37312 S  4035  6.6   9:51.64 java
...
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.080t  37312 S  3445  8.1  15:06.26 java
...
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.082t  37312 S  4178  8.2  17:37.59 java
...

It means, instead of using 1 virtual core, spark is using all available
cores in the system so my question is why it is behaving like this? why it
is not using only 1 core during execution of job which we set in
SPARK_WORKER_CORES property.

I am using spark 1.6.1 with standalone mode.

Any help will be highly appreciated.
Thanks
Shuja

-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>