You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "Huang, Jie" <ji...@intel.com> on 2015/06/26 13:24:38 UTC
[SparkScore]Performance portal for Apache Spark - WW26
Performance Portal for Apache Spark
Description
________________________________
Each data point represents each workload runtime percent compared with the previous week. Different lines represents different workloads running on spark yarn-client mode.
Hardware
________________________________
CPU type: Intel(r) Xeon(r) CPU E5-2697 v2 @ 2.70GHz
Memory: 128GB
NIC: 10GbE
Disk(s): 8 x 1TB SATA HDD
Software
________________________________
JAVA version: 1.8.0_25
Hadoop version: 2.5.0-CDH5.3.2
HiBench version: 4.0
Spark on yarn-client mode
Cluster
________________________________
1 node for Master
10 nodes for Slave
Regular
Summary
The lower percent the better performance.
________________________________
Group
ww19
ww20
ww22
ww23
ww24
ww25
ww26
HiBench
9.1%
6.6%
6.0%
7.9%
-6.5%
-3.1%
-2.1%
spark-perf
4.1%
4.4%
-1.8%
4.1%
-4.7%
-4.6%
-5.4%
[http://01org.github.io/sparkscore/image/plaf1.time/overall.png]
Y-Axis: normalized completion time; X-Axis: Work Week.
The commit number can be found in the result table.
The performance score for each workload is normalized based on the elapsed time for 1.2 release. The lower the better.
Detail
________________________________
HiBench
________________________________
JOB
ww19
ww20
ww22
ww23
ww24
ww25
ww26
commit
489700c8
8e3822a0
530efe3e
90c60692
db81b9d8
4eb48ed1
32e3cdaa
sleep
%
%
-2.1%
-2.9%
-4.1%
12.8%
-5.1%
wordcount
17.6%
11.4%
8.0%
8.3%
-18.6%
-10.9%
6.9%
kmeans
92.1%
61.5%
72.1%
92.9%
86.9%
95.8%
123.3%
scan
-4.9%
-7.2%
%
-1.1%
-25.5%
-21.0%
-12.4%
bayes
-24.3%
-20.1%
-18.3%
-11.1%
-29.7%
-31.3%
-30.9%
aggregation
5.6%
10.5%
%
9.2%
-15.3%
-15.0%
-37.6%
join
4.5%
1.2%
%
1.0%
-12.7%
-13.9%
-16.4%
sort
-3.3%
-0.5%
-11.9%
-12.5%
-17.5%
-17.3%
-20.7%
pagerank
2.2%
3.2%
4.0%
2.9%
-11.4%
-13.0%
-11.4%
terasort
-7.1%
-0.2%
-9.5%
-7.3%
-16.7%
-17.0%
-16.3%
Comments: null means no such workload running or workload failed in this time.
[http://01org.github.io/sparkscore/image/plaf1.time/HiBench_workloads.png]
Y-Axis: normalized completion time; X-Axis: Work Week.
The commit number can be found in the result table.
The performance score for each workload is normalized based on the elapsed time for 1.2 release.The lower the better.
spark-perf
________________________________
JOB
ww19
ww20
ww22
ww23
ww24
ww25
ww26
commit
489700c8
8e3822a0
530efe3e
90c60692
db81b9d8
4eb48ed1
32e3cdaa
agg
13.2%
7.0%
%
18.3%
5.2%
2.5%
1.1%
agg-int
16.4%
21.2%
%
9.6%
4.0%
8.2%
7.0%
agg-naive
4.3%
-2.4%
%
-0.8%
-6.7%
-6.8%
-8.5%
scheduling
-6.1%
-8.9%
-14.5%
-2.1%
-6.4%
-6.5%
-5.7%
count-filter
4.1%
1.0%
6.6%
6.8%
-10.2%
-10.4%
-9.8%
count
4.8%
4.6%
6.7%
8.0%
-7.3%
-7.0%
-8.0%
sort
-8.1%
-2.5%
-6.2%
-7.0%
-14.6%
-14.4%
-13.9%
sort-int
4.5%
15.3%
-1.6%
-0.1%
-1.5%
-2.2%
-5.3%
Comments: null means no such workload running or workload failed in this time.
[http://01org.github.io/sparkscore/image/plaf1.time/spark-perf_workloads.png]
Y-Axis: normalized completion time; X-Axis: Work Week.
The commit number can be found in the result table.
The performance score for each workload is normalized based on the elapsed time for 1.2 release. The lower the better.
Release
Summary
The lower percent the better performance.
________________________________
Group
1.2.1
1.3.0
1.3.1
1.4.0
HiBench
-1.0%
10.5%
8.4%
8.6%
spark-perf
3.2%
0.9%
1.9%
1.3%
[http://01org.github.io/sparkscore/image/plaf1.release/overall.png]
Y-Axis: normalized completion time; X-Axis: Release.
The performance score for each workload is normalized based on the elapsed time for 1.2 release. The lower the better.
Detail
________________________________
HiBench
________________________________
JOB
1.2.1
1.3.0
1.3.1
1.4.0
sleep
%
%
%
-0.5%
wordcount
3.5%
5.4%
5.1%
8.7%
kmeans
6.0%
72.6%
82.7%
100.7%
scan
-0.7%
-3.2%
-1.9%
-4.4%
bayes
-19.7%
7.7%
-24.5%
-14.4%
aggregation
4.6%
7.1%
9.9%
9.3%
join
0.7%
4.0%
8.6%
1.3%
sort
-1.0%
2.1%
-1.8%
-10.4%
pagerank
1.5%
2.2%
1.3%
5.4%
terasort
-3.7%
-3.3%
-3.7%
-9.5%
Comments: null means no such workload running or workload failed in this time.
[http://01org.github.io/sparkscore/image/plaf1.release/HiBench_workloads.png]
Y-Axis: normalized completion time; X-Axis: Release.
The commit number can be found in the result table.
The performance score for each workload is normalized based on the elapsed time for 1.2 release. The lower the better.
spark-perf
________________________________
JOB
1.2.1
1.3.0
1.3.1
1.4.0
agg
1.9%
3.1%
6.2%
5.0%
agg-int
6.4%
17.1%
18.0%
24.2%
agg-naive
-2.6%
-3.2%
-1.8%
-5.2%
scheduling
8.2%
-16.8%
-14.4%
-19.1%
count-filter
-0.4%
0.3%
-0.5%
0.4%
count
0.6%
-0.3%
0.4%
0.9%
sort
1.2%
-3.3%
-5.3%
-1.9%
sort-int
10.1%
10.0%
12.3%
6.0%
Comments: null means no such workload running or workload failed in this time.
[http://01org.github.io/sparkscore/image/plaf1.release/spark-perf_workloads.png]
Y-Axis: normalized completion time; X-Axis: Release.
The commit number can be found in the result table.
The performance score for each workload is normalized based on the elapsed time for 1.2 release. The lower the better.
________________________________
Copyright (c) 2015 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.
Project Email: sparkscore@lists.01.org<ma...@lists.01.org> Please subscribe to the list at: https://lists.01.org/mailman/listinfo/sparkscore
RE: [SparkScore]Performance portal for Apache Spark - WW26
Posted by "Huang, Jie" <ji...@intel.com>.
Thanks. In general, we can see a stable trend in Spark master branch and latest release.
And we are also considering to add more benchmarks/workloads into this automation perf tool. Any comment and feedback is warmly welcomed.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 8:21 PM
To: Huang, Jie
Cc: user@spark.apache.org; dev@spark.apache.org
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Thank you, Jie! Very nice work!
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 8:17 AM, Huang, Jie wrote:
Correct. Your calculation is right!
We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 7:59 PM
To: Huang, Jie
Cc: user@spark.apache.org<ma...@spark.apache.org>; dev@spark.apache.org<ma...@spark.apache.org>
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
Intel® Xeon® CPU E5-2697
RE: [SparkScore]Performance portal for Apache Spark - WW26
Posted by "Huang, Jie" <ji...@intel.com>.
Thanks. In general, we can see a stable trend in Spark master branch and latest release.
And we are also considering to add more benchmarks/workloads into this automation perf tool. Any comment and feedback is warmly welcomed.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 8:21 PM
To: Huang, Jie
Cc: user@spark.apache.org; dev@spark.apache.org
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Thank you, Jie! Very nice work!
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 8:17 AM, Huang, Jie wrote:
Correct. Your calculation is right!
We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 7:59 PM
To: Huang, Jie
Cc: user@spark.apache.org<ma...@spark.apache.org>; dev@spark.apache.org<ma...@spark.apache.org>
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
Intel® Xeon® CPU E5-2697
Re: [SparkScore]Performance portal for Apache Spark - WW26
Posted by Nan Zhu <zh...@gmail.com>.
Thank you, Jie! Very nice work!
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 8:17 AM, Huang, Jie wrote:
> Correct. Your calculation is right!
>
> We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
>
> And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
>
> Thank you && Best Regards,
> Grace （Huang Jie)
>
> From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
> Sent: Friday, June 26, 2015 7:59 PM
> To: Huang, Jie
> Cc: user@spark.apache.org (mailto:user@spark.apache.org); dev@spark.apache.org (mailto:dev@spark.apache.org)
> Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
>
> Hi, Jie,
>
>
>
> Thank you very much for this work! Very helpful!
>
>
>
> I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
>
>
>
> 9.1% - means the running time is 109.1 s?
>
>
>
> -4% - means it comes 96s?
>
>
>
> If that’s the true meaning of the numbers, what happened to k-means in HiBench?
>
>
>
> Best,
>
>
>
> --
>
> Nan Zhu
>
> http://codingcat.me
>
>
>
>
> On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
> > Intel® Xeon® CPU E5-2697
> >
>
>
>
>
>
>
Re: [SparkScore]Performance portal for Apache Spark - WW26
Posted by Nan Zhu <zh...@gmail.com>.
Thank you, Jie! Very nice work!
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 8:17 AM, Huang, Jie wrote:
> Correct. Your calculation is right!
>
> We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
>
> And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
>
> Thank you && Best Regards,
> Grace （Huang Jie)
>
> From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
> Sent: Friday, June 26, 2015 7:59 PM
> To: Huang, Jie
> Cc: user@spark.apache.org (mailto:user@spark.apache.org); dev@spark.apache.org (mailto:dev@spark.apache.org)
> Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
>
> Hi, Jie,
>
>
>
> Thank you very much for this work! Very helpful!
>
>
>
> I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
>
>
>
> 9.1% - means the running time is 109.1 s?
>
>
>
> -4% - means it comes 96s?
>
>
>
> If that’s the true meaning of the numbers, what happened to k-means in HiBench?
>
>
>
> Best,
>
>
>
> --
>
> Nan Zhu
>
> http://codingcat.me
>
>
>
>
> On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
> > Intel® Xeon® CPU E5-2697
> >
>
>
>
>
>
>
RE: [SparkScore]Performance portal for Apache Spark - WW26
Posted by "Huang, Jie" <ji...@intel.com>.
Correct. Your calculation is right!
We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 7:59 PM
To: Huang, Jie
Cc: user@spark.apache.org; dev@spark.apache.org
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
Intel® Xeon® CPU E5-2697
RE: [SparkScore]Performance portal for Apache Spark - WW26
Posted by "Huang, Jie" <ji...@intel.com>.
Correct. Your calculation is right!
We have been aware of that kmeans performance drop also. According to our observation, it is caused by some unbalanced executions among different tasks. Even we used the same test data between different versions (i.e., not caused by the data skew).
And the corresponding run time information has been shared with Xiangrui. Now he is also helping to identify the root cause altogether.
Thank you && Best Regards,
Grace （Huang Jie)
From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Friday, June 26, 2015 7:59 PM
To: Huang, Jie
Cc: user@spark.apache.org; dev@spark.apache.org
Subject: Re: [SparkScore]Performance portal for Apache Spark - WW26
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
Intel® Xeon® CPU E5-2697
Re: [SparkScore]Performance portal for Apache Spark - WW26
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
> Intel® Xeon® CPU E5-2697
Re: [SparkScore]Performance portal for Apache Spark - WW26
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Jie,
Thank you very much for this work! Very helpful!
I just would like to confirm that I understand the numbers correctly: if we take the running time of 1.2 release as 100s
9.1% - means the running time is 109.1 s?
-4% - means it comes 96s?
If that’s the true meaning of the numbers, what happened to k-means in HiBench?
Best,
--
Nan Zhu
http://codingcat.me
On Friday, June 26, 2015 at 7:24 AM, Huang, Jie wrote:
> Intel® Xeon® CPU E5-2697