[jira] [Created] (YARN-9737) Performance degradation, distributed opportunistic scheduling

["Babble Shack (JIRA)" <ji...@apache.org>]

Babble Shack created YARN-9737:
----------------------------------

             Summary: Performance degradation, distributed opportunistic scheduling
                 Key: YARN-9737
                 URL: https://issues.apache.org/jira/browse/YARN-9737
             Project: Hadoop YARN
          Issue Type: Bug
          Components: distributed-scheduling, yarn
    Affects Versions: 3.1.2
         Environment: OS: Ubuntu 18.04
 JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB ssd.
37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB ssd. 
2 * 3.5 Gb slots per Node Manager.


yarn-site: [^yarn-site.xml]
yarn-client-yarn-site: [^yarn-client.yarn-site.xml]

 
            Reporter: Babble Shack
         Attachments: jct_cdf_100j_100t_1500.svg, jct_cdf_100j_50t_1500_with_outliers.svg, jet_boxplot_j100_50t_1500.svg, jet_boxplot_j100_50t_1500_with_outliers.svg, task_throughput_boxplot_100j_50t_1500.svg, yarn-client.yarn-site.xml, yarn-site.xml

Opportunistic scheduling is supposed to provide lower scheduling time, and thus higher task throughput and lower job completion times for short jobs/tasks.

Through my experiments I have found distributed scheduling can degrade performance.

I ran a gridmix trace of 100 short jobs, each with 50 tasks, with an average run time of 1523ms. 

Findings:
 * Job completion time, the time take from submitting a job to job completion, may degrade by over 200%
[^jct_cdf_100j_100t_1500.svg]
[^jct_cdf_100j_50t_1500_with_outliers.svg]
 * Job execution time may increase by up to 300%
[^jet_boxplot_j100_50t_1500.svg]
[^jet_boxplot_j100_50t_1500_with_outliers.svg]
 * Task throughput decreased by 50%
^[^task_throughput_boxplot_100j_50t_1500.svg]^



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org