You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2020/12/24 15:54:58 UTC
[spark] branch master updated: [SPARK-33900][WEBUI] Show shuffle
read size / records correctly when only remotebytesread is available
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 700f5ab [SPARK-33900][WEBUI] Show shuffle read size / records correctly when only remotebytesread is available
700f5ab is described below
commit 700f5ab65c1c84522302ce92d176adf229c34daa
Author: sychen <sy...@ctrip.com>
AuthorDate: Fri Dec 25 00:54:26 2020 +0900
[SPARK-33900][WEBUI] Show shuffle read size / records correctly when only remotebytesread is available
### What changes were proposed in this pull request?
Shuffle Read Size / Records can also be displayed in remoteBytesRead>0 localBytesRead=0.
current:
![image](https://user-images.githubusercontent.com/3898450/103079421-c4ca2280-460e-11eb-9e2f-49d35b5d324d.png)
fix:
![image](https://user-images.githubusercontent.com/3898450/103079439-cc89c700-460e-11eb-9a41-6b2882980d11.png)
### Why are the changes needed?
At present, the page only displays the data of Shuffle Read Size / Records when localBytesRead>0.
When there is only remote reading, metrics cannot be seen on the stage page.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manual test
Closes #30916 from cxzl25/SPARK-33900.
Authored-by: sychen <sy...@ctrip.com>
Signed-off-by: Kousuke Saruta <sa...@oss.nttdata.com>
---
core/src/main/resources/org/apache/spark/ui/static/stagepage.js | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
index 336edff..ebb79f5 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
@@ -946,7 +946,8 @@ $(document).ready(function () {
},
{
data : function (row, type) {
- if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics && row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics &&
+ (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0 || row.taskMetrics.shuffleReadMetrics.remoteBytesRead > 0)) {
var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
if (type === 'display') {
return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org