You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2021/04/26 03:15:09 UTC
[spark] branch branch-3.1 updated: [SPARK-35087][UI] Some columns
in table Aggregated Metrics by Executor of stage-detail page shows
incorrectly.
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 6595db2 [SPARK-35087][UI] Some columns in table Aggregated Metrics by Executor of stage-detail page shows incorrectly.
6595db2 is described below
commit 6595db222190c2b7617b8e282a10bb4d606966f8
Author: kyoty <ec...@gmail.com>
AuthorDate: Mon Apr 26 12:13:22 2021 +0900
[SPARK-35087][UI] Some columns in table Aggregated Metrics by Executor of stage-detail page shows incorrectly.
### What changes were proposed in this pull request?
columns like 'Shuffle Read Size / Records', 'Output Size/ Records' etc in table ` Aggregated Metrics by Executor` of stage-detail page should be sorted as numerical-order instead of lexicographical-order.
### Why are the changes needed?
buf fix,the sorting style should be consistent between different columns.
The correspondence between the table and the index is shown below(it is defined in stagespage-template.html)：
| index | column name |
| ----- | -------------------------------------- |
| 0 | Executor ID |
| 1 | Logs |
| 2 | Address |
| 3 | Task Time |
| 4 | Total Tasks |
| 5 | Failed Tasks |
| 6 | Killed Tasks |
| 7 | Succeeded Tasks |
| 8 | Excluded |
| 9 | Input Size / Records |
| 10 | Output Size / Records |
| 11 | Shuffle Read Size / Records |
| 12 | Shuffle Write Size / Records |
| 13 | Spill (Memory) |
| 14 | Spill (Disk) |
| 15 | Peak JVM Memory OnHeap / OffHeap |
| 16 | Peak Execution Memory OnHeap / OffHeap |
| 17 | Peak Storage Memory OnHeap / OffHeap |
| 18 | Peak Pool Memory Direct / Mapped |
I constructed some data to simulate the sorting results of the index columns from 9 to 18.
As shown below,it can be seen that the sorting results of columns 9-12 are wrong:
The reason is that the real data corresponding to columns 9-12 (note that it is not the data displayed on the page) are **all strings similar to`94685/131`(bytes/records),while the real data corresponding to columns 13-18 are all numbers,**
so the sorting corresponding to columns 13-18 loos well, but the results of columns 9-12 are incorrect because the strings are sorted according to lexicographical order.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Only JS was modified, and the manual test result works well.
**before modified:**
**after modified:**
Closes #32190 from kyoty/aggregated-metrics-by-executor-sorted-incorrectly.
Authored-by: kyoty <ec...@gmail.com>
Signed-off-by: Kousuke Saruta <sa...@oss.nttdata.com>
(cherry picked from commit 2d6467d6d1633a06b6416eacd4f8167973e88136)
Signed-off-by: Kousuke Saruta <sa...@oss.nttdata.com>
---
.../org/apache/spark/ui/static/stagepage.js | 42 +++++++++++++++++++---
1 file changed, 38 insertions(+), 4 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
index 06e28c7..8b32fe7d 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js
@@ -43,6 +43,23 @@ $.extend( $.fn.dataTable.ext.type.order, {
a = ConvertDurationString( a );
b = ConvertDurationString( b );
return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ },
+
+ "size-pre": function (data) {
+ var floatValue = parseFloat(data)
+ return isNaN(floatValue) ? 0 : floatValue;
+ },
+
+ "size-asc": function (a, b) {
+ a = parseFloat(a);
+ b = parseFloat(b);
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "size-desc": function (a, b) {
+ a = parseFloat(a);
+ b = parseFloat(b);
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
}
} );
@@ -562,10 +579,27 @@ $(document).ready(function () {
}
],
"columnDefs": [
- { "visible": false, "targets": 15 },
- { "visible": false, "targets": 16 },
- { "visible": false, "targets": 17 },
- { "visible": false, "targets": 18 }
+ // SPARK-35087 [type:size] means String with structures like : 'size / records',
+ // they should be sorted as numerical-order instead of lexicographical-order by default.
+ // The targets: $id represents column id which comes from stagespage-template.html
+ // #summary-executor-table.If the relative position of the columns in the table
+ // #summary-executor-table has changed,please be careful to adjust the column index here
+ // Input Size / Records
+ {"type": "size", "targets": 9},
+ // Output Size / Records
+ {"type": "size", "targets": 10},
+ // Shuffle Read Size / Records
+ {"type": "size", "targets": 11},
+ // Shuffle Write Size / Records
+ {"type": "size", "targets": 12},
+ // Peak JVM Memory OnHeap / OffHeap
+ {"visible": false, "targets": 15},
+ // Peak Execution Memory OnHeap / OffHeap
+ {"visible": false, "targets": 16},
+ // Peak Storage Memory OnHeap / OffHeap
+ {"visible": false, "targets": 17},
+ // Peak Pool Memory Direct / Mapped
+ {"visible": false, "targets": 18}
],
"deferRender": true,
"order": [[0, "asc"]],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org