[jira] [Updated] (SPARK-10572) Investigate the contentions bewteen tasks in the same executor

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-10572:
------------------------------
    Component/s: Spark Core
                 Scheduler

> Investigate the contentions bewteen tasks in the same executor
> --------------------------------------------------------------
>
>                 Key: SPARK-10572
>                 URL: https://issues.apache.org/jira/browse/SPARK-10572
>             Project: Spark
>          Issue Type: Task
>          Components: Scheduler, Spark Core
>            Reporter: Davies Liu
>
> According to the benchmark results Jesse F Chen, It's surprised to see there are so much difference (4X) in term of number of executors, we should investigate the reason.
> ```
> > Just be curious how the difference would be if you use 20 executors
> > and 20G memory for each executor..
> So I tried the following combinations:
> (GB X # executors)  (query response time in secs)
> 20X20	415
> 10X40	230
> 5X80	141
> 4X100	128
> 2X200	104
> CPU utilization is high so spreading more JVMs onto more vCores helps in this case.
> For other workloads where memory utilization outweighs CPU, i can see larger JVM
> sizes maybe more beneficial. It's for sure case-by-case.
> Seems overhead for codegen and scheduler overhead are negligible.
> ```
> https://www.mail-archive.com/user@spark.apache.org/msg36486.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org