You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2020/12/24 15:55:57 UTC
[spark] branch branch-3.1 updated: [SPARK-33900][WEBUI] Show
shuffle read size / records correctly when only remotebytesread is
available
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 9a9a40f [SPARK-33900][WEBUI] Show shuffle read size / records correctly when only remotebytesread is available
9a9a40f is described below
commit 9a9a40fdd7f0d87fa25f19c5284e7892cb6392cd
Author: sychen <sy...@ctrip.com>
AuthorDate: Fri Dec 25 00:54:26 2020 +0900
[SPARK-33900][WEBUI] Show shuffle read size / records correctly when only remotebytesread is available
### What changes were proposed in this pull request?
Shuffle Read Size / Records can also be displayed in remoteBytesRead>0 localBytesRead=0.
current:
fix:
### Why are the changes needed?
At present, the page only displays the data of Shuffle Read Size / Records when localBytesRead>0.
When there is only remote reading, metrics cannot be seen on the stage page.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manual test
Closes #30916 from cxzl25/SPARK-33900.
Authored-by: sychen <sy...@ctrip.com>
Signed-off-by: Kousuke Saruta <sa...@oss.nttdata.com>
(cherry picked from commit 700f5ab65c1c84522302ce92d176adf229c34daa)
Signed-off-by: Kousuke Saruta <sa...@oss.nttdata.com>
---
core/src/main/resources/org/apache/spark/ui/static/stagepage.js | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
index 336edff..ebb79f5 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
@@ -946,7 +946,8 @@ $(document).ready(function () {
},
{
data : function (row, type) {
- if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics && row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics &&
+ (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0 || row.taskMetrics.shuffleReadMetrics.remoteBytesRead > 0)) {
var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
if (type === 'display') {
return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org