You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pgandhi999 <gi...@git.apache.org> on 2018/07/01 19:28:59 UTC
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
GitHub user pgandhi999 opened a pull request:
https://github.com/apache/spark/pull/21688
[SPARK-21809] : Change Stage Page to use datatables to support sorting columns and searching
Support column sort, pagination and search for Stage Page using jQuery DataTable and REST API. Before this commit, the Stage page generated a hard-coded HTML table that could not support search. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve the user experience.
Created the stagespage-template.html for displaying application information in datables. Added REST api endpoint and javascript code to fetch data from the endpoint and display it on the data table.
Because of the above change, certain functionalities in the page had to be modified to support the addition of datatables. For example, the toggle checkbox 'Select All' previously would add the checked fields as columns in the Task table and as rows in the Summary Metrics table, but after the change, only columns are added in the Task Table as it got tricky to add rows dynamically in the datatables.
## How was this patch tested?
I have attached the screenshots of the Stage Page UI before and after the fix.
**Before:**
<img width="1419" alt="30564304-35991e1c-9c8a-11e7-850f-2ac7a347f600" src="https://user-images.githubusercontent.com/22228190/42137915-52054558-7d3a-11e8-8c85-433b2c94161d.png">
<img width="1435" alt="31360592-cbaa2bae-ad14-11e7-941d-95b4c7d14970" src="https://user-images.githubusercontent.com/22228190/42137928-79df500a-7d3a-11e8-9068-5630afe46ff3.png">
**After:**
<img width="1432" alt="31360591-c5650ee4-ad14-11e7-9665-5a08d8f21830" src="https://user-images.githubusercontent.com/22228190/42137936-a3fb9f42-7d3a-11e8-8502-22b3897cbf64.png">
<img width="1388" alt="31360604-d266b6b0-ad14-11e7-94b5-dcc4bb5443f4" src="https://user-images.githubusercontent.com/22228190/42137970-0fabc58c-7d3b-11e8-95ad-383b1bd1f106.png">
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pgandhi999/spark SPARK-21809-2.3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21688.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21688
----
commit 5450258d7aef0772b9406c0ea5abb51096642ebb
Author: pgandhi <pg...@...>
Date: 2018-06-04T21:00:06Z
[SPARK-21809] : Changing file executorspage-template.html and executorspage.js
commit dadc55f624e515b5a1771c48f300bfa466fb1754
Author: pgandhi <pg...@...>
Date: 2018-06-04T21:26:42Z
[SPARK-21809] : Making changes in ui resources files
commit fc88decce9ac37ec3c639cda722464f26727e1be
Author: pgandhi <pg...@...>
Date: 2018-06-15T19:30:57Z
[SPARK-21809] : Getting rquired data in api endpoints
commit 495caafc800f33ae09b2d4ff0eb8ce33e31d7cc9
Author: pgandhi <pg...@...>
Date: 2018-06-20T16:32:31Z
[SPARK-21809] : Rendering the ui compatible with current version using datatables
commit c1d41d836f4d577e1188ca42e20abf9940f818ab
Author: pgandhi <pg...@...>
Date: 2018-06-22T22:10:15Z
Working on Adding Server Side Sort Functionality
commit 567d22b07a7c434141b2d8e8bcc183f87b0d8d52
Author: pgandhi <pg...@...>
Date: 2018-06-25T21:39:47Z
[SPARK-21809] : Implement Server Side Sorting Functionality
commit 649e95b86c7dd9c09173e11bcadfe9b6855b44a4
Author: pgandhi <pg...@...>
Date: 2018-06-27T18:14:18Z
[SPARK-21809] : Implementing server side pagination and searching functionality
commit 590e93f814c93f4ccc30af50d9f28ae92bb708e3
Author: pgandhi <pg...@...>
Date: 2018-07-01T05:03:14Z
[SPARK-21809] : Fixing Tests and task metric table
commit d79eb800bd8169cd063e1083d0444f755ffdb036
Author: pgandhi <pg...@...>
Date: 2018-07-01T19:18:59Z
[SPARK-21809] : Refactoring Code and adding shufflereadmetrics data
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220264138
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ }
+ if (_tasksToShow.nonEmpty) {
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ val ret = new util.HashMap[String, Object]()
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = ui.store.filterTaskList(_tasksToShow, searchValue)
+ if (filteredTaskList.length > 0) {
+ ret.put("aaData", filteredTaskList)
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
--- End diff --
Added sorting and pagination capability on search results as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220264300
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ }
+ if (_tasksToShow.nonEmpty) {
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ val ret = new util.HashMap[String, Object]()
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = ui.store.filterTaskList(_tasksToShow, searchValue)
+ if (filteredTaskList.length > 0) {
+ ret.put("aaData", filteredTaskList)
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97494/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96081/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558309
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224251226
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -278,7 +198,7 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
}
}
val currentTime = System.currentTimeMillis()
- val (taskTable, taskTableHTML) = try {
+ val taskTable = try {
val _taskTable = new TaskPagedTable(
--- End diff --
ok might be something we should look at closer to see if we want to clean up and use the rest api for
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #4442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4442/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220264258
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96721/testReport)** for PR 21688 at commit [`2bd0fcf`](https://github.com/apache/spark/commit/2bd0fcfa5aa89cfd569def82e525bca7cfb469c2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96146/testReport)** for PR 21688 at commit [`07c1f19`](https://github.com/apache/spark/commit/07c1f198e96a00add575e45bf9a0927e7d75b8ce).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220263590
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ for (r <- ret) {
+ val executorIdArray = r.executorSummary.get.keys.toArray
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220253587
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
+ var taskMetricIndices = Object.keys(response1);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": response1[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[0]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[1]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[2]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[3]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[4]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": response[0].numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Write Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.shuffleWriteMetrics.bytesWritten, type) + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.shuffleWriteMetrics.bytesWritten + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Write Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
+ if (type === 'display') {
+ return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ } else {
+ return totalBytesRead + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.memoryBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.memoryBytesSpilled, type) : row.taskMetrics.memoryBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Memory)"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.diskBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.diskBytesSpilled, type) : row.taskMetrics.diskBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Disk)"
+ },
+ {
+ data : function (row, type) {
+ var msg = row.errorMessage;
+ if (typeof msg === 'undefined'){
+ return "";
+ } else {
+ var form_head = msg.substring(0, msg.indexOf("at"));
+ var form = "<span onclick=\"this.parentNode.querySelector('.stacktrace-details').classList.toggle('collapsed')\" class=\"expand-details\">+details</span>";
+ var form_msg = "<div class=\"stacktrace-details collapsed\"><pre>" + row.errorMessage + "</pre></div>";
+ return form_head + form + form_msg;
+ }
+ },
+ name: "Errors"
+ }
+ ],
+ "columnDefs": [
+ { "visible": false, "targets": 11 },
+ { "visible": false, "targets": 12 },
+ { "visible": false, "targets": 13 },
+ { "visible": false, "targets": 14 },
+ { "visible": false, "targets": 15 },
+ { "visible": false, "targets": 16 },
+ { "visible": false, "targets": 17 },
+ { "visible": false, "targets": 18 }
+ ],
+ };
+ var taskTableSelector = $(taskTable).DataTable(task_conf);
+ $('#active-tasks-table_filter input').unbind();
+ $('#active-tasks-table_filter input').bind('keyup', function(e) {
+ if(e.keyCode == 13) {
+ taskTableSelector.search( this.value ).draw();
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220263860
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/21688
Hi @pgandhi999
Thanks for the work. One minor comment here:
Currently the table header looks like this
The arrow png file is on the most right side, which can be confusing.
In other tables, we are using U+25BE and U+25B4, right next to the text, which is more straightforward:
Can you adjust the position of the png files, or use the unicode instead to keep consistency.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96911/testReport)** for PR 21688 at commit [`1ba30ce`](https://github.com/apache/spark/commit/1ba30ceaa0b5f0f6ed38db4fae750030fc4c58ba).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216732491
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
+ var taskMetricIndices = Object.keys(response1);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": response1[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
--- End diff --
for this chained if, and the ones below, change to a switch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220262119
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219275020
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -71,7 +71,9 @@ class ExecutorStageSummary private[spark](
val shuffleWriteRecords : Long,
val memoryBytesSpilled : Long,
val diskBytesSpilled : Long,
- val isBlacklistedForStage: Boolean)
+ val isBlacklistedForStage: Boolean,
+ var executorLogs: Map[String, String],
--- End diff --
I don't like these being var and modified after the fact. I think we should either include them from the original call or make another call, rather then add them in like this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96912/testReport)** for PR 21688 at commit [`8cf0e34`](https://github.com/apache/spark/commit/8cf0e3467b4ad89d5757f08521f368d5da5c97af).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96557/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219275132
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -196,8 +198,8 @@ class StageData private[spark](
val rddIds: Seq[Int],
val accumulatorUpdates: Seq[AccumulableInfo],
- val tasks: Option[Map[Long, TaskData]],
- val executorSummary: Option[Map[String, ExecutorStageSummary]],
+ var tasks: Option[Map[Long, TaskData]],
--- End diff --
similar here, I'd rather not modify these to be var
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431384
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
--- End diff --
@pgandhi999 I think we mentioned switching on `ix` in the code below, instead of `columnName`.
Also could we change `ix` to be `columnKey`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #92516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92516/testReport)** for PR 21688 at commit [`6a0622e`](https://github.com/apache/spark/commit/6a0622e94c3a866ec2c3c9e1177e49a11dd7d9c3).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97121/testReport)** for PR 21688 at commit [`a4ed19d`](https://github.com/apache/spark/commit/a4ed19da4ebd0e3689f9fcaf3e847fb50111f884).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by michal-databricks <gi...@git.apache.org>.
Github user michal-databricks commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223588245
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
--- End diff --
I never used online regex testers. Do they actually generate the regex? Can you post a link to the one you used?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217797298
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -185,89 +185,9 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
</ul>
</div>
- val showAdditionalMetrics =
- <div>
- <span class="expand-additional-metrics">
- <span class="expand-additional-metrics-arrow arrow-closed"></span>
- <a>Show Additional Metrics</a>
- </span>
- <div class="additional-metrics collapsed">
- <ul>
- <li>
- <input type="checkbox" id="select-all-metrics"/>
- <span class="additional-metric-title"><em>(De)select All</em></span>
- </li>
- <li>
- <span data-toggle="tooltip"
--- End diff --
Sorry about that, added those tooltips again. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96492/testReport)** for PR 21688 at commit [`cbfbd07`](https://github.com/apache/spark/commit/cbfbd07a31960f6d49eb38c66d73ff776ecd0ffb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229115449
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,944 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for(k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98236/testReport)** for PR 21688 at commit [`0ed3309`](https://github.com/apache/spark/commit/0ed3309debeb73575e5fdb1b342ce70a8b05f9db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219223939
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
--- End diff --
rename taskMetricKeys
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219252150
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -16,15 +16,18 @@
*/
package org.apache.spark.status.api.v1
-import java.util.{List => JList}
+import java.util
--- End diff --
assume this isn't needed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21688
cc @gengliangwang
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219252850
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ for (r <- ret) {
--- End diff --
add comment in here about what we are doing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99084/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558750
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (columnKeyIndex) {
+ responseBody.executorSummary[columnKeyIndex].id = columnKeyIndex;
+ executor_summary_table.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219233070
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
--- End diff --
rename executor summary table
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/21688
Hi @ajbozarth did you get a chance to review it yet? Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216814520
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -185,89 +185,9 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
</ul>
</div>
- val showAdditionalMetrics =
- <div>
- <span class="expand-additional-metrics">
- <span class="expand-additional-metrics-arrow arrow-closed"></span>
- <a>Show Additional Metrics</a>
- </span>
- <div class="additional-metrics collapsed">
- <ul>
- <li>
- <input type="checkbox" id="select-all-metrics"/>
- <span class="additional-metric-title"><em>(De)select All</em></span>
- </li>
- <li>
- <span data-toggle="tooltip"
--- End diff --
did these tooltips go away?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229002442
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,944 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for(k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ // prepare data for executor summary table
+ indices = Object.keys(responseBody.executorSummary);
--- End diff --
rename indices to be something more descriptive like stageExecutorSummaryInfo
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216484047
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
--- End diff --
In the current UI, the "Show Additional Metrics" button will go ahead and toggle in "Summary Metrics":
Scheduler Delay
Task Deserialization Time
Result Serialization Time
Getting Result TIme
Peak Execution Memory
But in my local run of this change, the "Summary Metrics" table is static and not changed by the button.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #92517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92517/testReport)** for PR 21688 at commit [`132fce9`](https://github.com/apache/spark/commit/132fce983a3c23798e9141e36b3ab06a42814a24).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219275714
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
--- End diff --
going with my comment above, I don't want to be modifying the StageData by changing the parameters to var, we should either get all the data we need from stageData or we could perhaps create a separate call to get these. Will have to look in more depth
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220254689
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
+ <h4 class="title-table">Aggregated Metrics by Executor</h4>
+ </a>
+ <br>
+ <div class="container-fluid" id="toggle-aggregatedMetrics" hidden>
+ <div class="container-fluid">
+ <table id="summary-stages-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Executor ID</th>
+ <th>Logs</th>
+ <th>Address</th>
+ <th>Task Time</th>
+ <th>Total Tasks</th>
+ <th>Failed Tasks</th>
+ <th>Killed Tasks</th>
+ <th>Succeeded Tasks</th>
+ <th>
+ <span data-toggle="tooltip" data-placement="top"
+ title="Shows if this executor has been blacklisted by the scheduler due to task failures.">
+ Blacklisted</span>
+ </th>
+ <th>Input Size / Records</th>
+ <th>Output:</th>
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231605818
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98483/testReport)** for PR 21688 at commit [`c26a9ff`](https://github.com/apache/spark/commit/c26a9ffdcbbd02fc6b309e3fa85b70b926f53515).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220263505
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -16,15 +16,18 @@
*/
package org.apache.spark.status.api.v1
-import java.util.{List => JList}
+import java.util
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97007 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97007/testReport)** for PR 21688 at commit [`a4ed19d`](https://github.com/apache/spark/commit/a4ed19da4ebd0e3689f9fcaf3e847fb50111f884).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219303958
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ }
+ if (_tasksToShow.nonEmpty) {
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ val ret = new util.HashMap[String, Object]()
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = ui.store.filterTaskList(_tasksToShow, searchValue)
+ if (filteredTaskList.length > 0) {
+ ret.put("aaData", filteredTaskList)
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
--- End diff --
totalRecords needs to be adjusted to what was filtered right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97947/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220254869
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
+ <h4 class="title-table">Aggregated Metrics by Executor</h4>
+ </a>
+ <br>
+ <div class="container-fluid" id="toggle-aggregatedMetrics" hidden>
+ <div class="container-fluid">
+ <table id="summary-stages-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Executor ID</th>
+ <th>Logs</th>
+ <th>Address</th>
+ <th>Task Time</th>
+ <th>Total Tasks</th>
+ <th>Failed Tasks</th>
+ <th>Killed Tasks</th>
+ <th>Succeeded Tasks</th>
+ <th>
+ <span data-toggle="tooltip" data-placement="top"
+ title="Shows if this executor has been blacklisted by the scheduler due to task failures.">
+ Blacklisted</span>
+ </th>
+ <th>Input Size / Records</th>
+ <th>Output:</th>
+ <th>Shuffle Read: </th>
+ <th>Shuffle Write: </th>
+ <th>Shuffle Spill (Memory): </th>
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96643/testReport)** for PR 21688 at commit [`2e88030`](https://github.com/apache/spark/commit/2e880306185ed6f3a875905352050c13e4b4cba2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96716/testReport)** for PR 21688 at commit [`204c708`](https://github.com/apache/spark/commit/204c708f09889c5cbc24fa50fa8ef81d8b2bd731).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229439305
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorSummaryMap = new Map();
+ for (var i = 0; i < executorSummaryResponse.length; i++) {
+ executorSummaryMap.set(executorSummaryResponse[i].id, executorSummaryResponse[i]);
+ }
+ var executor_summary_table = [];
+ stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) {
+ responseBody.executorSummary[columnKeyIndex].id = columnKeyIndex;
+ if ("executorLogs" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs != null) {
+ responseBody.executorSummary[columnKeyIndex].executorLogs =
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].executorLogs = {};
+ }
+ if ("hostPort" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort != null) {
+ responseBody.executorSummary[columnKeyIndex].hostPort =
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].hostPort = "CANNOT FIND ADDRESS";
+ }
+ executor_summary_table.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
--- End diff --
Lets do camel case for all variables.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r230923570
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
--- End diff --
nit, I'd make this a boolean
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/21688
@tgravescs Have fixed the bug where the page used to hang on clicking on the "Summary Metrics for 2 Completed Tasks" link and then refreshing the page. Thank you for pointing that out.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222733733
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
--- End diff --
As discussed, width is set based on size of the header names, so we can do this as part of another PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232809274
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232711679
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
--- End diff --
not used remove
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219293257
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
--- End diff --
if is search is on why do we get the taskList here and then later get the filterTasklist? seems like we basically do the work twice
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on the issue:
https://github.com/apache/spark/pull/21688
The failing suite is `OrcQuerySuite `. Looks unrelated.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96939/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98745/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231932184
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,960 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = true;
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225963998
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
Have made the required changes and tested them.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220254012
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98745/testReport)** for PR 21688 at commit [`271de2d`](https://github.com/apache/spark/commit/271de2d186bcd776105a419a0c4f2b8e26498e35).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98606/testReport)** for PR 21688 at commit [`d6b41ca`](https://github.com/apache/spark/commit/d6b41cafd92a5ec06dbd4bde3d62ab96a8ae7af4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224238974
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
these are actually going to be written to the tracking store. we don't want that if the values aren't set.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97505/testReport)** for PR 21688 at commit [`a4661fe`](https://github.com/apache/spark/commit/a4661feb5044f6d89328a4805edf0d0fbf115e89).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219221688
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
--- End diff --
get rid of accumulator_table_all by combining loop. array filter method may work here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219229287
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
--- End diff --
is bytesRead already a Javascript array?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r227117575
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ---
@@ -547,6 +519,24 @@ private[spark] class AppStatusStore(
store.close()
}
+ def interceptAndModifyTaskData(taskDataWrapper: TaskDataWrapper) : v1.TaskData = {
--- End diff --
rename this function constructTaskData
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220262036
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
--- End diff --
Replaced with filter function. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558995
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,124 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217796714
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
+ var taskMetricIndices = Object.keys(response1);
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
Test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97494/testReport)** for PR 21688 at commit [`7e486f1`](https://github.com/apache/spark/commit/7e486f1929cb568f2e96f70c8c7832194a286e67).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220260604
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
--- End diff --
Yep we can discuss that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92515/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by dbkerkela <gi...@git.apache.org>.
Github user dbkerkela commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222563886
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
--- End diff --
reusing the ind variable here is confusing to me - possibly a structure like this would be more obvious:
if (urlArray.indexOf("proxy") > -1) {
...
appId = ...
} else if (urlArray.indexOf("history") > -1) {
...
} else {
base = location.origin
appId = ...
...
}
return base + .... + appId
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218880301
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
+ <h4 class="title-table">Aggregated Metrics by Executor</h4>
+ </a>
+ <br>
+ <div class="container-fluid" id="toggle-aggregatedMetrics" hidden>
+ <div class="container-fluid">
+ <table id="summary-stages-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Executor ID</th>
+ <th>Logs</th>
+ <th>Address</th>
+ <th>Task Time</th>
+ <th>Total Tasks</th>
+ <th>Failed Tasks</th>
+ <th>Killed Tasks</th>
+ <th>Succeeded Tasks</th>
+ <th>
+ <span data-toggle="tooltip" data-placement="top"
+ title="Shows if this executor has been blacklisted by the scheduler due to task failures.">
+ Blacklisted</span>
+ </th>
+ <th>Input Size / Records</th>
+ <th>Output:</th>
--- End diff --
remove : from this and the next few lines
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431857
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
--- End diff --
This should be easy for the summary table (top), but the other tables might be more of a challenge.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r227515143
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order, {
function stageEndPoint(appId) {
var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
var urlArray = urlRegex.exec(document.baseURI);
- var ind = urlArray.indexOf("proxy");
+ var indexOfProxy = urlArray.indexOf("proxy");
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220254791
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
+ <h4 class="title-table">Aggregated Metrics by Executor</h4>
+ </a>
+ <br>
+ <div class="container-fluid" id="toggle-aggregatedMetrics" hidden>
+ <div class="container-fluid">
+ <table id="summary-stages-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Executor ID</th>
+ <th>Logs</th>
+ <th>Address</th>
+ <th>Task Time</th>
+ <th>Total Tasks</th>
+ <th>Failed Tasks</th>
+ <th>Killed Tasks</th>
+ <th>Succeeded Tasks</th>
+ <th>
+ <span data-toggle="tooltip" data-placement="top"
+ title="Shows if this executor has been blacklisted by the scheduler due to task failures.">
+ Blacklisted</span>
+ </th>
+ <th>Input Size / Records</th>
+ <th>Output:</th>
+ <th>Shuffle Read: </th>
+ <th>Shuffle Write: </th>
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r230102562
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorSummaryMap = new Map();
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98236/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by dbkerkela <gi...@git.apache.org>.
Github user dbkerkela commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222561249
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
--- End diff --
I think break is redundant after return?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225964304
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (columnKeyIndex) {
+ responseBody.executorSummary[columnKeyIndex].id = columnKeyIndex;
+ executor_summary_table.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (columnKey) {
+ if (columnKey == "shuffleReadMetrics") {
--- End diff --
Have made the required changes. Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r235184400
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -259,7 +278,8 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
}
- def makeTimeline(tasks: Seq[TaskData], currentTime: Long): Seq[Node] = {
+ def makeTimeline(tasks: Seq[TaskData], currentTime: Long, page: Int, pageSize: Int,
+ totalPages: Int, stageId: Int, stageAttemptId: Int, totalTasks: Int): Seq[Node] = {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224106218
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,124 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
+ var columnToSort = 0
+ if (queryParams.contains("order[0][column]")) {
+ columnToSort = queryParameters.getFirst("order[0][column]").toInt
+ }
+ var columnNameToSort = queryParameters.getFirst("columns[" + columnToSort + "][name]")
+ if (columnNameToSort.equalsIgnoreCase("Logs")) {
+ columnNameToSort = "Index"
+ columnToSort = 0
+ }
+ val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+ var pageStartIndex = 0
+ var pageLength = totalRecords
+ if (!isSearch) {
+ pageStartIndex = queryParameters.getFirst("start").toInt
+ pageLength = queryParameters.getFirst("length").toInt
+ }
+ return withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
+ indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+ }
+
+ // Filters task list based on search parameter
+ def filterTaskList(
+ taskDataList: Seq[TaskData],
+ searchValue: String): Seq[TaskData] = {
+ val defaultOptionString: String = "d"
+ // The task metrics dummy object below has been added to avoid throwing exception in cases
+ // when task metrics for a particular task do not exist as of yet
+ val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ new InputMetrics(0, 0), new OutputMetrics(0, 0),
+ new ShuffleReadMetrics(0, 0, 0, 0, 0, 0, 0), new ShuffleWriteMetrics(0, 0, 0))
+ val searchValueLowerCase = searchValue.toLowerCase(Locale.ROOT)
+ val containsValue = (taskDataParams: Any) => taskDataParams.toString.toLowerCase(
+ Locale.ROOT).contains(searchValueLowerCase)
+ val filteredTaskDataSequence: Seq[TaskData] = taskDataList.filter(f =>
+ (containsValue(f.taskId) || containsValue(f.index) || containsValue(f.attempt)
--- End diff --
I think this should also search on "errorMessage"
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #4442 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4442/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98606/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217797349
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,25 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ for (i <- 0 to (ret.length - 1)) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216729669
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
--- End diff --
can we set quantiles somewhere above the code? seems sneaky in here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232838308
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -117,35 +117,38 @@ private[v1] class StagesResource extends BaseAppResource {
val uriQueryParameters = uriInfo.getQueryParameters(true)
val totalRecords = uriQueryParameters.getFirst("numTasks")
var isSearch = false
- var searchValue: String = null
+ var searchValue: Option[String] = None
var filteredRecords = totalRecords
- var _tasksToShow: Seq[TaskData] = null
// The datatables client API sends a list of query parameters to the server which contain
// information like the columns to be sorted, search value typed by the user in the search
// box, pagination index etc. For more information on these query parameters,
// refer https://datatables.net/manual/server-side.
if (uriQueryParameters.getFirst("search[value]") != null &&
uriQueryParameters.getFirst("search[value]").length > 0) {
- _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
- totalRecords.toInt)
isSearch = true
- searchValue = uriQueryParameters.getFirst("search[value]")
- } else {
- _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
- totalRecords.toInt)
+ searchValue = try {
--- End diff --
Yep, makes sense, have changed the code.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223126615
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,159 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
+ var columnToSort = 0
+ if (queryParams.contains("order[0][column]")) {
+ columnToSort = queryParameters.getFirst("order[0][column]").toInt
+ }
+ var columnNameToSort = queryParameters.getFirst("columns[" + columnToSort + "][name]")
+ if (columnNameToSort.equalsIgnoreCase("Logs")) {
+ columnNameToSort = "Index"
+ columnToSort = 0
+ }
+ val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+ var pageStartIndex = 0
+ var pageLength = totalRecords
+ if (!isSearch) {
+ pageStartIndex = queryParameters.getFirst("start").toInt
+ pageLength = queryParameters.getFirst("length").toInt
+ }
+ return withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
+ indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+ }
+
+ // Filters task list based on search parameter
+ def filterTaskList(
+ taskDataList: Seq[TaskData],
+ searchValue: String): Seq[TaskData] = {
+ val defaultOptionString: String = "d"
+ // The task metrics dummy object below has been added to avoid throwing exception in cases
+ // when task metrics for a particular task do not exist as of yet
+ val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
--- End diff --
I did not get your comment clearly here, do you mean that I should have all task metric filters in a separate function or should I wrap dummytaskmetrics in an Option? Can you please elaborate a little? Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92516/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96939/testReport)** for PR 21688 at commit [`8cf0e34`](https://github.com/apache/spark/commit/8cf0e3467b4ad89d5757f08521f368d5da5c97af).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431891
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 2
+ };
+ if (dataToShow.showOutputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle Write Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 4
+ };
+ if (dataToShow.showShuffleWriteData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (disk)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 5
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (memory)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 6
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 0
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ });
+ var task_summary_metrics_table_filtered_array =
+ task_summary_metrics_table_array.filter(row => row.checkboxId < 11);
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_filtered_array);
+ });
+
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ var executorSummaryTableSelector =
+ $(executorSummaryTable).DataTable(executor_summary_conf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var tasksToShow = JSON.parse(jsonStr);
+ return tasksToShow.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
--- End diff --
executorId wasn't sorting correctly for me. May need to convert to integer... can you double check sorting for all columns?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96821/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218585485
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
--- End diff --
seems a bit brittle if the parameters were to change order, could you do a regexp for like id=[0-9]+[^&]*
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98370/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r227116341
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order, {
function stageEndPoint(appId) {
var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
var urlArray = urlRegex.exec(document.baseURI);
- var ind = urlArray.indexOf("proxy");
+ var indexOfProxy = urlArray.indexOf("proxy");
var queryString = document.baseURI.split('?');
var words = document.baseURI.split('/');
var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
- if (ind > 0) {
+ if (indexOfProxy > 0) {
var appId = urlArray[2];
- var indexOfProxy = words.indexOf("proxy");
- var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
}
- ind = urlArray.indexOf("history");
- if (ind > 0) {
+ var indexOfHistory = urlArray.indexOf("history");
+ if (indexOfHistory > 0) {
var appId = urlArray[2];
- var appAttemptId = urlArray[ind + 2];
- var indexOfHistory = words.indexOf("history");
- var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ var appAttemptId = urlArray[indexOfHistory + 2];
--- End diff --
same here just use words
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231148125
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
--- End diff --
My concern is that if I go later and change this code, I don't want to change a label ("Shuffle Read Block Time") and have logic break on me.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219235188
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
--- End diff --
change name of marrr
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217796571
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
+ var taskMetricIndices = Object.keys(response1);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": response1[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[0]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[1]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[2]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[3]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[4]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": response[0].numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Write Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.shuffleWriteMetrics.bytesWritten, type) + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.shuffleWriteMetrics.bytesWritten + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Write Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
+ if (type === 'display') {
+ return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ } else {
+ return totalBytesRead + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.memoryBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.memoryBytesSpilled, type) : row.taskMetrics.memoryBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Memory)"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.diskBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.diskBytesSpilled, type) : row.taskMetrics.diskBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Disk)"
+ },
+ {
+ data : function (row, type) {
+ var msg = row.errorMessage;
+ if (typeof msg === 'undefined'){
+ return "";
+ } else {
+ var form_head = msg.substring(0, msg.indexOf("at"));
+ var form = "<span onclick=\"this.parentNode.querySelector('.stacktrace-details').classList.toggle('collapsed')\" class=\"expand-details\">+details</span>";
+ var form_msg = "<div class=\"stacktrace-details collapsed\"><pre>" + row.errorMessage + "</pre></div>";
+ return form_head + form + form_msg;
+ }
+ },
+ name: "Errors"
+ }
+ ],
+ "columnDefs": [
+ { "visible": false, "targets": 11 },
+ { "visible": false, "targets": 12 },
+ { "visible": false, "targets": 13 },
+ { "visible": false, "targets": 14 },
+ { "visible": false, "targets": 15 },
+ { "visible": false, "targets": 16 },
+ { "visible": false, "targets": 17 },
+ { "visible": false, "targets": 18 }
+ ],
+ };
+ var taskTableSelector = $(taskTable).DataTable(task_conf);
+ $('#active-tasks-table_filter input').unbind();
+ $('#active-tasks-table_filter input').bind('keyup', function(e) {
+ if(e.keyCode == 13) {
+ taskTableSelector.search( this.value ).draw();
--- End diff --
The tasks datatable has been implemented with server-side sorting, pagination and searching. So the data is pushed directly from the server, thus, implementing a front-end filtering functionality is not possible here. Regarding pressing Enter to get the results, I can change it to happen on keypress. Let me know what you think.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219252977
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ for (r <- ret) {
+ val executorIdArray = r.executorSummary.get.keys.toArray
--- End diff --
remove executorIdArray and just put r.executorSummary.get.keys.toArray inside for
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96912/testReport)** for PR 21688 at commit [`8cf0e34`](https://github.com/apache/spark/commit/8cf0e3467b4ad89d5757f08521f368d5da5c97af).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218581872
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
--- End diff --
might be nice to add example when used with proxy
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218884278
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
--- End diff --
summary metrics table shows a bit odd when no tasks have finished, might not be a big deal but could see if we can improve. Basically it doesn't show any rows.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96081/testReport)** for PR 21688 at commit [`af83b6e`](https://github.com/apache/spark/commit/af83b6e300a009cea48915fb8471e3edfc4db66b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223126675
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219221157
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
--- End diff --
can you use Object.values here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221263645
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +42,29 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ // Some of the data that we want to display for the tasks like executorLogs,
+ // schedulerDelay etc. comes from other sources, thus, we basically add them to the
+ // executor and task data object before passing them to the client.
+ for (r <- ret) {
+ for (execId <- r.executorSummary.get.keys.toArray) {
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ val hostPort = ui.store.executorSummary(execId).hostPort
+ val taskDataArray = r.tasks.get.keys.toArray
+ var execStageSummary = r.executorSummary.get.get(execId).get
+ execStageSummary.executorLogs = executorLogs
+ execStageSummary.hostPort = hostPort
+ for (taskData <- taskDataArray) {
+ var taskDataObject = r.tasks.get.get(taskData).get
+ taskDataObject.executorLogs = executorLogs
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220263551
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,27 @@ private[v1] class StagesResource extends BaseAppResource {
@PathParam("stageId") stageId: Int,
@QueryParam("details") @DefaultValue("true") details: Boolean): Seq[StageData] = {
withUI { ui =>
- val ret = ui.store.stageData(stageId, details = details)
+ var ret = ui.store.stageData(stageId, details = details)
if (ret.nonEmpty) {
+ for (r <- ret) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99084/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218931652
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
--- End diff --
wrap this line at 100 chars if we can
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224108070
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -290,17 +210,11 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
desc = taskSortDesc,
store = parent.store
)
- (_taskTable, _taskTable.table(page))
+ _taskTable.table(page)
--- End diff --
This call should be removed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98357/testReport)** for PR 21688 at commit [`66a14bc`](https://github.com/apache/spark/commit/66a14bcc89207834e39cb290cc422dbaa252acb0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224166110
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
--- End diff --
wrap this line if we can
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558239
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -290,17 +210,11 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
desc = taskSortDesc,
store = parent.store
)
- (_taskTable, _taskTable.table(page))
+ _taskTable.table(page)
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222733891
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 2
+ };
+ if (dataToShow.showOutputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle Write Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 4
+ };
+ if (dataToShow.showShuffleWriteData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (disk)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 5
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (memory)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 6
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 0
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ });
+ var task_summary_metrics_table_filtered_array =
+ task_summary_metrics_table_array.filter(row => row.checkboxId < 11);
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_filtered_array);
+ });
+
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ var executorSummaryTableSelector =
+ $(executorSummaryTable).DataTable(executor_summary_conf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var tasksToShow = JSON.parse(jsonStr);
+ return tasksToShow.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
--- End diff --
Double checked sorting, works for me, let me know if you still see issues.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97938/testReport)** for PR 21688 at commit [`258e81b`](https://github.com/apache/spark/commit/258e81b4a36644733ce9663ab49f60936e407100).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99077/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98395/testReport)** for PR 21688 at commit [`24e6b45`](https://github.com/apache/spark/commit/24e6b457ee9e88489f99a674e8f35f514e78ed16).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218886347
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
--- End diff --
what is searching for the aggregated metrics table based on? If you search for an integer like 32 (which would match an id) it finds other things but also shows rows with no 32 in it. I also assume the search is based on the unformatted numbers because I tried ot search for 1.2 GB and 1.2 which should have matched on the Input Size but it showed nothing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r230102675
--- Diff: core/src/main/resources/org/apache/spark/ui/static/webui.css ---
@@ -260,4 +260,45 @@ a.expandbutton {
.paginate_button.active > a {
color: #999999;
text-decoration: underline;
+}
+
+.title-table {
+ clear: left;
+ display: inline-block;
+}
+
+.table-dataTable {
+ width="100%";
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by michal-databricks <gi...@git.apache.org>.
Github user michal-databricks commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223014017
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
--- End diff --
This regex does not improve the function and while maybe there could be a better one, overall parsing URI should not be done with custom code or regexp. This code exists in other functions as well so for me the best outcome would be to abstract it out, but this may be out of scope for this PR. The second best would be to keep the functions consistent (so revert to the previous way).
As for the regexp itself:
- the `g` and `m` don't do anything here (`g` does not influence `exec` and `m` affects only `$` and `^` anchors.
- `/` does not have to be escaped in `[]` groups which would make it a bit more readable (also using `new Regex('...')` would also help for that).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99011/testReport)** for PR 21688 at commit [`a696dcc`](https://github.com/apache/spark/commit/a696dcc40482c6b47d20729dfab10097f1ea679b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99077/testReport)** for PR 21688 at commit [`672b12a`](https://github.com/apache/spark/commit/672b12aa9f0fffb84ec44506b85c35fc881f71b7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229115504
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,944 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for(k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ // prepare data for executor summary table
+ indices = Object.keys(responseBody.executorSummary);
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96517/testReport)** for PR 21688 at commit [`af11138`](https://github.com/apache/spark/commit/af111380f9fad8b109842e53dbd7737b7d5c19bb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220262544
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97505/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98395/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223127069
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
+ null,
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by michal-databricks <gi...@git.apache.org>.
Github user michal-databricks commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222992058
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
--- End diff --
This line is identical in all 3 branches of the function and does not depend on the branch. It should be moved above the branching point.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222734361
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,906 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 2
+ };
+ if (dataToShow.showOutputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle Write Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 4
+ };
+ if (dataToShow.showShuffleWriteData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (disk)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 5
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (memory)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 6
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 0
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ });
+ var task_summary_metrics_table_filtered_array =
+ task_summary_metrics_table_array.filter(row => row.checkboxId < 11);
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_filtered_array);
+ });
+
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ var executorSummaryTableSelector =
+ $(executorSummaryTable).DataTable(executor_summary_conf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "lengthMenu": [[20, 40, 60, 100, responseBody.numTasks], [20, 40, 60, 100, "All"]],
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var tasksToShow = JSON.parse(jsonStr);
+ return tasksToShow.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
--- End diff --
As discussed, I am using existing sorting API's so I have to explicitly write code to sort it the way described above. If it is nit for now, can do it as part of another PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229469673
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorSummaryMap = new Map();
+ for (var i = 0; i < executorSummaryResponse.length; i++) {
+ executorSummaryMap.set(executorSummaryResponse[i].id, executorSummaryResponse[i]);
+ }
+ var executor_summary_table = [];
+ stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) {
+ responseBody.executorSummary[columnKeyIndex].id = columnKeyIndex;
+ if ("executorLogs" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs != null) {
+ responseBody.executorSummary[columnKeyIndex].executorLogs =
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].executorLogs = {};
+ }
+ if ("hostPort" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort != null) {
+ responseBody.executorSummary[columnKeyIndex].hostPort =
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].hostPort = "CANNOT FIND ADDRESS";
+ }
+ executor_summary_table.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
--- End diff --
if you follow the naming above, I'd remove this line, and put `"#summary-executor-table"` directly in ` $(executorSummaryTable).DataTable(executor_summary_conf);` => ` $("#summary-executor-table").DataTable(executor_summary_conf);` below...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/21688
I'll see if I get a chance to look at this this week, but with the holiday it may have to wait a week.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96780/testReport)** for PR 21688 at commit [`a91b306`](https://github.com/apache/spark/commit/a91b306d2ddbe977e7086d4392f5a9d292264b7c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99011/testReport)** for PR 21688 at commit [`a696dcc`](https://github.com/apache/spark/commit/a696dcc40482c6b47d20729dfab10097f1ea679b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232736621
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
--- End diff --
we set isSearch and pass in true above, just set this above and pass into doPagination.
Really I think you could simplify this and do the if then set isSearch and only have 1 call to doPagination outside the if.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232809245
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
+ var columnNameToSort = queryParameters.getFirst("columnNameToSort")
+ if (columnNameToSort.equalsIgnoreCase("Logs")) {
+ columnNameToSort = "Index"
+ }
+ val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+ var pageStartIndex = 0
+ var pageLength = totalRecords
+ if (!isSearch) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r223124830
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,159 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
+ var columnToSort = 0
+ if (queryParams.contains("order[0][column]")) {
+ columnToSort = queryParameters.getFirst("order[0][column]").toInt
+ }
+ var columnNameToSort = queryParameters.getFirst("columns[" + columnToSort + "][name]")
+ if (columnNameToSort.equalsIgnoreCase("Logs")) {
+ columnNameToSort = "Index"
+ columnToSort = 0
+ }
+ val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+ var pageStartIndex = 0
+ var pageLength = totalRecords
+ if (!isSearch) {
+ pageStartIndex = queryParameters.getFirst("start").toInt
+ pageLength = queryParameters.getFirst("length").toInt
+ }
+ return withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
+ indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+ }
+
+ // Filters task list based on search parameter
+ def filterTaskList(
+ taskDataList: Seq[TaskData],
+ searchValue: String): Seq[TaskData] = {
+ val defaultOptionString: String = "d"
+ // The task metrics dummy object below has been added to avoid throwing exception in cases
+ // when task metrics for a particular task do not exist as of yet
+ val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ new InputMetrics(0, 0), new OutputMetrics(0, 0),
+ new ShuffleReadMetrics(0, 0, 0, 0, 0, 0, 0), new ShuffleWriteMetrics(0, 0, 0))
+ val searchValueLowerCase = searchValue.toLowerCase(Locale.ROOT)
+ val filteredTaskDataSequence: Seq[TaskData] = taskDataList.filter(f =>
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96909/testReport)** for PR 21688 at commit [`a1d4317`](https://github.com/apache/spark/commit/a1d4317a2a17809923c16338c9b2a36ffb12bf67).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558473
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var indexOfHistory = words.indexOf("history");
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96909/testReport)** for PR 21688 at commit [`a1d4317`](https://github.com/apache/spark/commit/a1d4317a2a17809923c16338c9b2a36ffb12bf67).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class YarnShuffleServiceMetrics implements MetricsSource `
* `case class SchemaOfJson(`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r230102603
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorSummaryMap = new Map();
+ for (var i = 0; i < executorSummaryResponse.length; i++) {
+ executorSummaryMap.set(executorSummaryResponse[i].id, executorSummaryResponse[i]);
+ }
+ var executor_summary_table = [];
+ stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) {
+ responseBody.executorSummary[columnKeyIndex].id = columnKeyIndex;
+ if ("executorLogs" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs != null) {
+ responseBody.executorSummary[columnKeyIndex].executorLogs =
+ executorSummaryMap.get(columnKeyIndex.toString()).executorLogs;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].executorLogs = {};
+ }
+ if ("hostPort" in executorSummaryMap.get(columnKeyIndex.toString()) &&
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort != null) {
+ responseBody.executorSummary[columnKeyIndex].hostPort =
+ executorSummaryMap.get(columnKeyIndex.toString()).hostPort;
+ } else {
+ responseBody.executorSummary[columnKeyIndex].hostPort = "CANNOT FIND ADDRESS";
+ }
+ executor_summary_table.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97505/testReport)** for PR 21688 at commit [`a4661fe`](https://github.com/apache/spark/commit/a4661feb5044f6d89328a4805edf0d0fbf115e89).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222733306
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 2
+ };
+ if (dataToShow.showOutputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle Write Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 4
+ };
+ if (dataToShow.showShuffleWriteData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (disk)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 5
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (memory)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 6
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 0
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ });
+ var task_summary_metrics_table_filtered_array =
+ task_summary_metrics_table_array.filter(row => row.checkboxId < 11);
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_filtered_array);
+ });
+
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ var executorSummaryTableSelector =
+ $(executorSummaryTable).DataTable(executor_summary_conf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var tasksToShow = JSON.parse(jsonStr);
+ return tasksToShow.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
--- End diff --
Done. It will be blank now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r227116251
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order, {
function stageEndPoint(appId) {
var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
var urlArray = urlRegex.exec(document.baseURI);
- var ind = urlArray.indexOf("proxy");
+ var indexOfProxy = urlArray.indexOf("proxy");
--- End diff --
we seems to be using urlArray and then words, we don't really need both. Just get everything form words if that is what you need the format in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r222732441
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
--- End diff --
Done, albeit in a different manner.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219189328
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
+ var taskMetricIndices = Object.keys(response1);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": response1[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": response1[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[0]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[1]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[2]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[3]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ if (row.metric == 'Input Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesRead).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Output Size / Records') {
+ var str1arr = JSON.stringify(row.data.bytesWritten).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.recordsWritten).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Size / Records') {
+ var str1arr = JSON.stringify(row.data.readBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.readRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else if (row.metric == 'Shuffle Read Blocked Time') {
+ var str1arr = JSON.stringify(row.data.fetchWaitTime).split("[")[1].split("]")[0].split(",");
+ var str = formatDuration(str1arr[4]);
+ return str;
+ } else if (row.metric == 'Shuffle Remote Reads') {
+ var str1arr = JSON.stringify(row.data.remoteBytesRead).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ } else if (row.metric == 'Shuffle Write Size / Records') {
+ var str1arr = JSON.stringify(row.data.writeBytes).split("[")[1].split("]")[0].split(",");
+ var str2arr = JSON.stringify(row.data.writeRecords).split("[")[1].split("]")[0].split(",");
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ } else {
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": response[0].numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Write Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.shuffleWriteMetrics.bytesWritten, type) + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.shuffleWriteMetrics.bytesWritten + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Write Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
+ if (type === 'display') {
+ return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ } else {
+ return totalBytesRead + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.memoryBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.memoryBytesSpilled, type) : row.taskMetrics.memoryBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Memory)"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.diskBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.diskBytesSpilled, type) : row.taskMetrics.diskBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Disk)"
+ },
+ {
+ data : function (row, type) {
+ var msg = row.errorMessage;
+ if (typeof msg === 'undefined'){
+ return "";
+ } else {
+ var form_head = msg.substring(0, msg.indexOf("at"));
+ var form = "<span onclick=\"this.parentNode.querySelector('.stacktrace-details').classList.toggle('collapsed')\" class=\"expand-details\">+details</span>";
+ var form_msg = "<div class=\"stacktrace-details collapsed\"><pre>" + row.errorMessage + "</pre></div>";
+ return form_head + form + form_msg;
+ }
+ },
+ name: "Errors"
+ }
+ ],
+ "columnDefs": [
+ { "visible": false, "targets": 11 },
+ { "visible": false, "targets": 12 },
+ { "visible": false, "targets": 13 },
+ { "visible": false, "targets": 14 },
+ { "visible": false, "targets": 15 },
+ { "visible": false, "targets": 16 },
+ { "visible": false, "targets": 17 },
+ { "visible": false, "targets": 18 }
+ ],
+ };
+ var taskTableSelector = $(taskTable).DataTable(task_conf);
+ $('#active-tasks-table_filter input').unbind();
+ $('#active-tasks-table_filter input').bind('keyup', function(e) {
+ if(e.keyCode == 13) {
+ taskTableSelector.search( this.value ).draw();
--- End diff --
@pgandhi999 I meant to do this with a timer, so when you finish typing a 5 second timer for example starts (and gets reset on keyup). Only when the timer is up then it should make the fetch.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218880234
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
--- End diff --
missing tooltip
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232736598
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
--- End diff --
don't pre initialize this, just set it as results of call to doPagination which should no longer be in if (see comment below)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99076/testReport)** for PR 21688 at commit [`b3e2370`](https://github.com/apache/spark/commit/b3e23707869d87794992c12c7e782fd582a25ab7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217796686
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #92516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92516/testReport)** for PR 21688 at commit [`6a0622e`](https://github.com/apache/spark/commit/6a0622e94c3a866ec2c3c9e1177e49a11dd7d9c3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
+1 , going to merge to master
There are a few followup jiras on this.
1) make the timeline visualization better: https://issues.apache.org/jira/browse/SPARK-26130
2) improve search functionalit: https://issues.apache.org/jira/browse/SPARK-25719
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99127/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99127/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232809213
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220264190
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ---
@@ -391,6 +391,41 @@ private[spark] class AppStatusStore(
ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq
}
+ // Filters task list based on search parameter
+ def filterTaskList(
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231147892
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
--- End diff --
I think `createRowMetadataForColumn` could add a column key (alongside `row.metric`). Then you can switch on that `columnKey`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21688
cc @dbkerkela @gengliangwang
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r235184320
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: Option[String] = None
+ var filteredRecords = totalRecords
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ isSearch = true
+ searchValue = Some(uriQueryParameters.getFirst("search[value]"))
+ }
+ val _tasksToShow: Seq[TaskData] = doPagination(uriQueryParameters, stageId, stageAttemptId,
+ isSearch, totalRecords.toInt)
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue.get)
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431470
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
--- End diff --
same thing here, it wold be preferable to switch on `columnKey`, not `columnName`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219237589
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Write Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.shuffleWriteMetrics.bytesWritten, type) + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.shuffleWriteMetrics.bytesWritten + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Write Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) {
+ var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead);
+ if (type === 'display') {
+ return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ } else {
+ return totalBytesRead + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.memoryBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.memoryBytesSpilled, type) : row.taskMetrics.memoryBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Memory)"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.diskBytesSpilled > 0) {
+ return type === 'display' ? formatBytes(row.taskMetrics.diskBytesSpilled, type) : row.taskMetrics.diskBytesSpilled;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Spill (Disk)"
+ },
+ {
+ data : function (row, type) {
+ var msg = row.errorMessage;
+ if (typeof msg === 'undefined'){
+ return "";
+ } else {
+ var form_head = msg.substring(0, msg.indexOf("at"));
+ var form = "<span onclick=\"this.parentNode.querySelector('.stacktrace-details').classList.toggle('collapsed')\" class=\"expand-details\">+details</span>";
+ var form_msg = "<div class=\"stacktrace-details collapsed\"><pre>" + row.errorMessage + "</pre></div>";
+ return form_head + form + form_msg;
+ }
+ },
+ name: "Errors"
+ }
+ ],
+ "columnDefs": [
+ { "visible": false, "targets": 11 },
+ { "visible": false, "targets": 12 },
+ { "visible": false, "targets": 13 },
+ { "visible": false, "targets": 14 },
+ { "visible": false, "targets": 15 },
+ { "visible": false, "targets": 16 },
+ { "visible": false, "targets": 17 },
+ { "visible": false, "targets": 18 }
+ ],
+ };
+ var taskTableSelector = $(taskTable).DataTable(task_conf);
+ $('#active-tasks-table_filter input').unbind();
+ $('#active-tasks-table_filter input').bind('keyup', function(e) {
+ taskTableSelector.search( this.value ).draw();
+ });
+
+ var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+ var allChecked = true;
+ for(k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ document.getElementById("box-"+optionalColumns[k]).checked = true;
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ document.getElementById("box-0").checked = true;
+ }
+
+ // hide or show columns dynamically event
+ $('input.toggle-vis').on('click', function(e){
+ // Get the column
+ var para = $(this).attr('data-column');
+ if(para == "0"){
+ var column = taskTableSelector.column([11, 12, 13, 14, 15, 16, 17]);
+ if($(this).is(":checked")){
+ $(".toggle-vis").prop('checked', true);
+ column.visible(true);
+ } else {
+ $(".toggle-vis").prop('checked', false);
+ column.visible(false);
+ }
+ } else {
+ var column = taskTableSelector.column($(this).attr('data-column'));
--- End diff --
identation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97007 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97007/testReport)** for PR 21688 at commit [`a4ed19d`](https://github.com/apache/spark/commit/a4ed19da4ebd0e3689f9fcaf3e847fb50111f884).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r225558809
--- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js ---
@@ -74,3 +74,99 @@ function getTimeZone() {
return new Date().toString().match(/\((.*)\)/)[1];
}
}
+
+function formatLogsCells(execLogs, type) {
+ if (type !== 'display') return Object.keys(execLogs);
+ if (!execLogs) return;
+ var result = '';
+ $.each(execLogs, function (logName, logUrl) {
+ result += '<div><a href=' + logUrl + '>' + logName + '</a></div>'
+ });
+ return result;
+}
+
+function getStandAloneAppId(cb) {
+ var words = document.baseURI.split('/');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ cb(appId);
+ return;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ cb(appId);
+ return;
+ }
+ //Looks like Web UI is running in standalone mode
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98745/testReport)** for PR 21688 at commit [`271de2d`](https://github.com/apache/spark/commit/271de2d186bcd776105a419a0c4f2b8e26498e35).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219220733
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
--- End diff --
check to see if really need id because we are copying array here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #92517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92517/testReport)** for PR 21688 at commit [`132fce9`](https://github.com/apache/spark/commit/132fce983a3c23798e9141e36b3ab06a42814a24).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21688
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231931747
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,869 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
--- End diff --
Yep makes sense, have removed it. Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #97947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97947/testReport)** for PR 21688 at commit [`052f706`](https://github.com/apache/spark/commit/052f70683db032783901fcce1e981a5392c92ea1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219301332
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
--- End diff --
We aren't handling sorting here. If I search on somethign and then sort on column it should sort here as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r229438719
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ if (typeof taskTableSelector !== 'undefined' && task_summary_metrics_table_current_state_array.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ task_summary_metrics_table_current_state_array.push(task_summary_metrics_table_array.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ task_summary_metrics_table_current_filtered_array = task_summary_metrics_table_current_state_array.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_current_filtered_array);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorSummaryMap = new Map();
--- End diff --
The `Map` class has some incompatibilities in IE, though I conceptually like the idea of using objects like a `Map`, in this case, I think you are looking for a regular-old JS map `{}`. Additionally, I've seen cases where there can be undefined cases, so I suggest to rewrite this as:
```
var executorDetailsMap = {};
executorDetailsResponse.forEach(function(executorDetail){
executorDetailsMap[executorDetail.id] = executorDetail;
});
var executorSummaryTable = [];
stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) {
var executorSummary = responseBody.executorSummary[columnKeyIndex];
var executorDetail = executorDetailsMap[columnKeyIndex.toString()];
executorSummary.id = columnKeyIndex;
executorSummary.executorLogs = {}
executorSummary.hostPort = "CANNOT FIND ADDRESS";
if (executorDetail) {
if (executorDetail["executorLogs"]) {
responseBody.executorSummary[columnKeyIndex].executorLogs =
executorDetail["executorLogs"];
}
if (executorDetail["hostPort"]) {
responseBody.executorSummary[columnKeyIndex].hostPort =
executorDetail["hostPort"];
}
}
executorSummaryTable.push(responseBody.executorSummary[columnKeyIndex]);
});
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98741/testReport)** for PR 21688 at commit [`73a5451`](https://github.com/apache/spark/commit/73a5451e99b073de220b0a5f149bc83437e1bcb3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231163762
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTable) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(taskSummaryMetricsTable).draw();
+ } else {
+ var taskConf = {
+ "data": taskSummaryMetricsTable,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false,
+ "bAutoWidth": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(taskConf);
+ }
+ taskSummaryMetricsTableCurrentStateArray = taskSummaryMetricsTable.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice();
+ if (typeof taskTableSelector !== 'undefined' && taskSummaryMetricsTableCurrentStateArray.length > 0) {
+ for (k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ taskSummaryMetricsTableCurrentStateArray.push(taskSummaryMetricsTableArray.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableCurrentFilteredArray);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ // We are using regex here to extract the stage attempt id as there might be certain url's with format
+ // like /proxy/application_1539986433979_27115/stages/stage/?id=0&attempt=0#tasksTitle
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var taskSummaryMetricsTableArray = [];
+var taskSummaryMetricsTableCurrentStateArray = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
--- End diff --
I don't know what `.tooltip(true)` does, but instead of calling `.tooltip` on each element, why not do this:
`$('[data-toggle="tooltip"]').tooltip()`
Maybe in a function that you can call for each block where you had individual `.tooltip`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216733959
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
+ return "Duration";
+ }
+ else if(columnKey == "jvmGcTime") {
+ return "GC Time";
+ }
+ else if(columnKey == "gettingResultTime") {
+ return "Getting Result Time";
+ }
+ else if(columnKey == "inputMetrics") {
+ return "Input Size / Records";
+ }
+ else if(columnKey == "outputMetrics") {
+ return "Output Size / Records";
+ }
+ else if(columnKey == "peakExecutionMemory") {
+ return "Peak Execution Memory";
+ }
+ else if(columnKey == "resultSerializationTime") {
+ return "Result Serialization Time";
+ }
+ else if(columnKey == "schedulerDelay") {
+ return "Scheduler Delay";
+ }
+ else if(columnKey == "diskBytesSpilled") {
+ return "Shuffle spill (disk)";
+ }
+ else if(columnKey == "memoryBytesSpilled") {
+ return "Shuffle spill (memory)";
+ }
+ else if(columnKey == "shuffleReadMetrics") {
+ return "Shuffle Read Size / Records";
+ }
+ else if(columnKey == "shuffleWriteMetrics") {
+ return "Shuffle Write Size / Records";
+ }
+ else if(columnKey == "executorDeserializeTime") {
+ return "Task Deserialization Time";
+ }
+ return "NA";
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(response[0].executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ response[0].executorSummary[ix].id = ix;
+ task_summary_table.push(response[0].executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(response[0].accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(response[0].accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {
--- End diff --
a lot of code happens inside of this callback. Could each of the tables be broken down into a function? With the callback here reduced to calls to a set of functions?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5233/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218889551
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
--- End diff --
write time shows up with 6 decimal points, we probably don't need that precision here:
36.213747 ms
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219297759
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ---
@@ -391,6 +391,41 @@ private[spark] class AppStatusStore(
ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq
}
+ // Filters task list based on search parameter
+ def filterTaskList(
--- End diff --
this function doesn't really have anything to do with the store, its simply filtering on the taskData so I think it should just live in StagesResource.scala
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98563/testReport)** for PR 21688 at commit [`a8b5862`](https://github.com/apache/spark/commit/a8b58629b0348e00a8ca5e83d124b29db4af656d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95902/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431683
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var stageAttemptId = getStageAttemptId();
+ var responseBody = response[stageAttemptId];
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0?true:false;
+ dataToShow.showOutputData = responseBody.outputBytes > 0?true:false;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0?true:false;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0?true:false;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0)?true:false;
+
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var executor_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ executor_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulator_table = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 3
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 13
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 14
+ };
+ if (dataToShow.showShuffleReadData) {
+ task_summary_metrics_table_array.push(row1);
+ }
+ task_summary_metrics_table_array.push(row2);
+ task_summary_metrics_table_array.push(row3);
+ }
+ else if (columnName == "Scheduler Delay") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 11
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Task Deserialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 12
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Result Serialization Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 15
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Getting Result Time") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 16
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Peak Execution Memory") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 17
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ else if (columnName == "Input Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 1
+ };
+ if (dataToShow.showInputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Output Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 2
+ };
+ if (dataToShow.showOutputData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle Write Size / Records") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 4
+ };
+ if (dataToShow.showShuffleWriteData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (disk)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 5
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName == "Shuffle spill (memory)") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 6
+ };
+ if (dataToShow.showBytesSpilledData) {
+ task_summary_metrics_table_array.push(row);
+ }
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix],
+ "checkboxId": 0
+ };
+ task_summary_metrics_table_array.push(row);
+ }
+ });
+ var task_summary_metrics_table_filtered_array =
+ task_summary_metrics_table_array.filter(row => row.checkboxId < 11);
+ createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table_filtered_array);
+ });
+
+ // building task aggregated metrics by executor table
+ var executorSummaryTable = "#summary-executor-table";
+ var executor_summary_conf = {
+ "data": executor_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ var executorSummaryTableSelector =
+ $(executorSummaryTable).DataTable(executor_summary_conf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var tasksToShow = JSON.parse(jsonStr);
+ return tasksToShow.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
--- End diff --
more javascripty + it removes on one your elses.
`if row.taskMetrics && row.taskMetrics.inputMetrics.bytesRead > 0`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219302290
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +124,87 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean, @Context uriInfo: UriInfo):
+ util.HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var _tasksToShow: Seq[TaskData] = null
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = ui.store.taskList(stageId, stageAttemptId, 0, totalRecords.toInt,
+ indexName("Index"), true)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
+ }
+ if (_tasksToShow.nonEmpty) {
+ val iterator = _tasksToShow.iterator
+ while(iterator.hasNext) {
+ val t1: TaskData = iterator.next()
+ val execId = t1.executorId
+ val executorLogs = ui.store.executorSummary(execId).executorLogs
+ t1.executorLogs = executorLogs
+ t1.schedulerDelay = AppStatusUtils.schedulerDelay(t1)
+ t1.gettingResultTime = AppStatusUtils.gettingResultTime(t1)
+ }
+ val ret = new util.HashMap[String, Object]()
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = ui.store.filterTaskList(_tasksToShow, searchValue)
+ if (filteredTaskList.length > 0) {
+ ret.put("aaData", filteredTaskList)
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId)
--- End diff --
although I think you still want to do the pagination stuff on the filtered list here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221431847
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
--- End diff --
we should use the "width" property to make the numeric columns equal in size if at all possible.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #95902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95902/testReport)** for PR 21688 at commit [`573390d`](https://github.com/apache/spark/commit/573390d0a933e7a2641f944046442a136bba6cd8).
* This patch **fails from timeout after a configured wait of \`400m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220260786
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[1]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[1], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[1], type) + " / " + str2arr[1];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[1], type) : (formatDuration(row.data[1]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[2]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[2], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[2], type) + " / " + str2arr[2];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[2], type) : (formatDuration(row.data[2]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[3]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[3], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[3], type) + " / " + str2arr[3];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[3], type) : (formatDuration(row.data[3]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[4]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[4], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[4], type) + " / " + str2arr[4];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[4], type) : (formatDuration(row.data[4]));
+
+ }
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "file-size", "targets": 1 },
+ { "type": "file-size", "targets": 2 },
+ { "type": "file-size", "targets": 3 },
+ { "type": "file-size", "targets": 4 },
+ { "type": "file-size", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ $(taskMetricsTable).DataTable(task_conf);
+ });
+
+ // building task aggregated metric table
+ var tasksSummarytable = "#summary-stages-table";
+ var task_summary_conf = {
+ "data": task_summary_table,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : 0;
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]]
+ }
+ $(tasksSummarytable).DataTable(task_summary_conf);
+ $('#active-tasks [data-toggle="tooltip"]').tooltip();
+
+ // building accumulator update table
+ var accumulatorTable = "#accumulator-table";
+ var accumulator_conf = {
+ "data": accumulator_table,
+ "columns": [
+ {data : "id"},
+ {data : "name"},
+ {data : "value"}
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ }
+ $(accumulatorTable).DataTable(accumulator_conf);
+
+ // building tasks table that uses server side functionality
+ var taskTable = "#active-tasks-table";
+ var task_conf = {
+ "serverSide": true,
+ "paging": true,
+ "info": true,
+ "processing": true,
+ "ajax": {
+ "url": stageEndPoint(appId) + "/" + stageAttemptId + "/taskTable",
+ "data": {
+ "numTasks": responseBody.numTasks
+ },
+ "dataSrc": function ( jsons ) {
+ var jsonStr = JSON.stringify(jsons);
+ var marrr = JSON.parse(jsonStr);
+ return marrr.aaData;
+ }
+ },
+ "columns": [
+ {data: function (row, type) {
+ return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index;
+ },
+ name: "Index"
+ },
+ {data : "taskId", name: "ID"},
+ {data : "attempt", name: "Attempt"},
+ {data : "status", name: "Status"},
+ {data : "taskLocality", name: "Locality Level"},
+ {data : "executorId", name: "Executor ID"},
+ {data : "host", name: "Host"},
+ {data : "executorLogs", name: "Logs", render: formatLogsCells},
+ {data : "launchTime", name: "Launch Time", render: formatDate},
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorRunTime) : row.taskMetrics.executorRunTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Duration"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "GC Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay;
+ },
+ name: "Scheduler Delay"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Task Deserialization Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleReadMetrics !== 'undefined') {
+ return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime;
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Shuffle Read Blocked Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Shuffle Remote Reads"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Result Serialization Time"
+ },
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime;
+ },
+ name: "Getting Result Time"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory;
+ } else {
+ return "N/A";
+ }
+ },
+ name: "Peak Execution Memory"
+ },
+ {
+ data : function (row, type) {
+ if (accumulator_table.length > 0 && row.accumulatorUpdates.length > 0) {
+ var accIndex = row.accumulatorUpdates.length - 1;
+ return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update;
+ } else {
+ return "";
+ }
+ },
+ name: "Accumulators"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.inputMetrics.bytesRead > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ } else {
+ return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Input Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.outputMetrics.bytesWritten > 0) {
+ if (type === 'display') {
+ return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ } else {
+ return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten;
+ }
+ } else {
+ return "";
+ }
+ } else {
+ return "";
+ }
+ },
+ name: "Output Size / Records"
+ },
+ {
+ data : function (row, type) {
+ if ("taskMetrics" in row) {
+ if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+ return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r231932218
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,960 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = true;
+
+$(document).ajaxStop(function () {
+ if (blockUICount) {
+ $.unblockUI();
+ blockUICount = false;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var indexOfProxy = words.indexOf("proxy");
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (indexOfProxy > 0) {
+ var appId = words[indexOfProxy + 1];
+ var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ var indexOfHistory = words.indexOf("history");
+ if (indexOfHistory > 0) {
+ var appId = words[indexOfHistory + 1];
+ var appAttemptId = words[indexOfHistory + 2];
+ var newBaseURI = words.slice(0, words.indexOf("history")).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.columnKey) {
+ case 'inputMetrics':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'outputMetrics':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'shuffleReadMetrics':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'shuffleReadBlockedTime':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'shuffleRemoteReads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'shuffleWriteMetrics':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.columnKey == 'peakExecutionMemory' || row.columnKey == 'memoryBytesSpilled'
+ || row.columnKey == 'diskBytesSpilled') ? formatBytes(
+ row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTable) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(taskSummaryMetricsTable).draw();
+ } else {
+ var taskConf = {
+ "data": taskSummaryMetricsTable,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]],
+ "bSort": false,
+ "bAutoWidth": false
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(taskConf);
+ }
+ taskSummaryMetricsTableCurrentStateArray = taskSummaryMetricsTable.slice();
+}
+
+function createRowMetadataForColumn(columnName, data, checkboxId, colKey) {
+ var row = {
+ "metric": columnName,
+ "data": data,
+ "checkboxId": checkboxId,
+ "columnKey": colKey
+ };
+ return row;
+}
+
+function reselectCheckboxesBasedOnTaskTableState() {
+ var allChecked = true;
+ var taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice();
+ if (typeof taskTableSelector !== 'undefined' && taskSummaryMetricsTableCurrentStateArray.length > 0) {
+ for (var k = 0; k < optionalColumns.length; k++) {
+ if (taskTableSelector.column(optionalColumns[k]).visible()) {
+ $("#box-"+optionalColumns[k]).prop('checked', true);
+ taskSummaryMetricsTableCurrentStateArray.push(taskSummaryMetricsTableArray.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]);
+ taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice();
+ } else {
+ allChecked = false;
+ }
+ }
+ if (allChecked) {
+ $("#box-0").prop('checked', true);
+ }
+ createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableCurrentFilteredArray);
+ }
+}
+
+function getStageAttemptId() {
+ var words = document.baseURI.split('?');
+ var attemptIdStr = words[1].split('&')[1];
+ var digitsRegex = /[0-9]+/;
+ // We are using regex here to extract the stage attempt id as there might be certain url's with format
+ // like /proxy/application_1539986433979_27115/stages/stage/?id=0&attempt=0#tasksTitle
+ var stgAttemptId = words[1].split("&").filter(
+ word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex);
+ return stgAttemptId;
+}
+
+var taskSummaryMetricsTableArray = [];
+var taskSummaryMetricsTableCurrentStateArray = [];
+var taskSummaryMetricsDataTable;
+var optionalColumns = [11, 12, 13, 14, 15, 16, 17];
+var taskTableSelector;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "right")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('[data-toggle="tooltip"]').tooltip();
+ tasksSummary = $("#parent-container");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ var stageAttemptId = getStageAttemptId();
+ $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) {
+
+ var responseBody = response;
+ var dataToShow = {};
+ dataToShow.showInputData = responseBody.inputBytes > 0;
+ dataToShow.showOutputData = responseBody.outputBytes > 0;
+ dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0;
+ dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0;
+ dataToShow.showBytesSpilledData =
+ (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);
+
+ if (!dataToShow.showShuffleReadData) {
+ $('#shuffle_read_blocked_time').remove();
+ $('#shuffle_remote_reads').remove();
+ optionalColumns.splice(2, 2);
+ }
+
+ // prepare data for executor summary table
+ stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary);
+ $.getJSON(createRESTEndPointForExecutorsPage(appId),
+ function(executorSummaryResponse, status, jqXHR) {
+ var executorDetailsMap = {};
+ executorSummaryResponse.forEach(function (executorDetail) {
+ executorDetailsMap[executorDetail.id] = executorDetail;
+ });
+
+ var executorSummaryTable = [];
+ stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) {
+ var executorSummary = responseBody.executorSummary[columnKeyIndex];
+ var executorDetail = executorDetailsMap[columnKeyIndex.toString()];
+ executorSummary.id = columnKeyIndex;
+ executorSummary.executorLogs = {};
+ executorSummary.hostPort = "CANNOT FIND ADDRESS";
+
+ if (executorDetail) {
+ if (executorDetail["executorLogs"]) {
+ responseBody.executorSummary[columnKeyIndex].executorLogs =
+ executorDetail["executorLogs"];
+ }
+ if (executorDetail["hostPort"]) {
+ responseBody.executorSummary[columnKeyIndex].hostPort =
+ executorDetail["hostPort"];
+ }
+ }
+ executorSummaryTable.push(responseBody.executorSummary[columnKeyIndex]);
+ });
+ // building task aggregated metrics by executor table
+ var executorSummaryConf = {
+ "data": executorSummaryTable,
+ "columns": [
+ {data : "id"},
+ {data : "executorLogs", render: formatLogsCells},
+ {data : "hostPort"},
+ {
+ data : function (row, type) {
+ return type === 'display' ? formatDuration(row.taskTime) : row.taskTime;
+ }
+ },
+ {
+ data : function (row, type) {
+ var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks;
+ return type === 'display' ? totaltasks : totaltasks.toString();
+ }
+ },
+ {data : "failedTasks"},
+ {data : "killedTasks"},
+ {data : "succeededTasks"},
+ {data : "isBlacklistedForStage"},
+ {
+ data : function (row, type) {
+ return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : "";
+ }
+ },
+ {
+ data : function (row, type) {
+ return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : "";
+ }
+ }
+ ],
+ "order": [[0, "asc"]],
+ "bAutoWidth": false
+ }
+ var executorSummaryTableSelector =
+ $("#summary-executor-table").DataTable(executorSummaryConf);
+ $('#parent-container [data-toggle="tooltip"]').tooltip();
+
+ executorSummaryTableSelector.column(9).visible(dataToShow.showInputData);
+ if (dataToShow.showInputData) {
+ $('#executor-summary-input').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Bytes and records read from Hadoop or from Spark storage.");
+ $('#executor-summary-input').tooltip(true);
+ }
+ executorSummaryTableSelector.column(10).visible(dataToShow.showOutputData);
+ if (dataToShow.showOutputData) {
+ $('#executor-summary-output').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Bytes and records written to Hadoop.");
+ $('#executor-summary-output').tooltip(true);
+ }
+ executorSummaryTableSelector.column(11).visible(dataToShow.showShuffleReadData);
+ if (dataToShow.showShuffleReadData) {
+ $('#executor-summary-shuffle-read').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Total shuffle bytes and records read (includes both data read locally and data read from remote executors).");
+ $('#executor-summary-shuffle-read').tooltip(true);
+ }
+ executorSummaryTableSelector.column(12).visible(dataToShow.showShuffleWriteData);
+ if (dataToShow.showShuffleWriteData) {
+ $('#executor-summary-shuffle-write').attr("data-toggle", "tooltip")
+ .attr("data-placement", "top")
+ .attr("title", "Bytes and records written to disk in order to be read by a shuffle in a future stage.");
+ $('#executor-summary-shuffle-write').tooltip(true);
+ }
+ executorSummaryTableSelector.column(13).visible(dataToShow.showBytesSpilledData);
+ executorSummaryTableSelector.column(14).visible(dataToShow.showBytesSpilledData);
+ });
+
+ // prepare data for accumulatorUpdates
+ var accumulatorTable = responseBody.accumulatorUpdates.filter(accumUpdate =>
+ !(accumUpdate.name).toString().includes("internal."));
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#additionalMetrics").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ if (window.localStorage) {
+ window.localStorage.setItem("arrowtoggle1class", $("#arrowtoggle1").attr('class'));
+ }
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ if (window.localStorage) {
+ window.localStorage.setItem("arrowtoggle2class", $("#arrowtoggle2").attr('class'));
+ }
+ });
+
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(endPoint + "/" + stageAttemptId + "/taskSummary?quantiles=" + quantiles,
+ function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricKeys = Object.keys(taskMetricsResponse);
+ taskMetricKeys.forEach(function (columnKey) {
+ switch(columnKey) {
+ case "shuffleReadMetrics":
+ var row1 = createRowMetadataForColumn(
+ getColumnNameForTaskMetricSummary(columnKey), taskMetricsResponse[columnKey], 3, columnKey);
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224856923
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
I don't think the executor stage summary should contain this info.
Is it really that difficult to do the join? The current stage page already does that:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala#L710
Is the concern that doing something like that from JS would be expensive or too difficult?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218876136
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
--- End diff --
the summary metrics Metric used to have tooltips on hover
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98236/testReport)** for PR 21688 at commit [`0ed3309`](https://github.com/apache/spark/commit/0ed3309debeb73575e5fdb1b342ce70a8b05f9db).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224850397
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
@vanzin ideas on how to better handle this?
I don't see a real clean way to populate these fields from the AppstatusListener before being written. For context, in this PR these are currently being populated in the AppStatusStore.executorSummary call before going back to user.
We could potentially split into separate api or on UI side query both that and the executor info and join but seems like a lot more data.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219251713
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -278,7 +198,7 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
}
}
val currentTime = System.currentTimeMillis()
- val (taskTable, taskTableHTML) = try {
+ val taskTable = try {
val _taskTable = new TaskPagedTable(
--- End diff --
seems like this shouldn't be needed anymore. Feels like a bunch of stuff in here should be able to be removed now?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r221671401
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,906 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfProxy = words.indexOf("proxy");
+ var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = urlArray.indexOf("history");
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var appAttemptId = urlArray[ind + 2];
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ var indexOfHistory = words.indexOf("history");
+ var newBaseURI = words.slice(0, indexOfHistory).join('/');
+ if (isNaN(appAttemptId) || appAttemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId;
+ }
+ }
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+function displayRowsForSummaryMetricsTable(row, type, columnIndex) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " +
+ row.data.recordsRead[columnIndex];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " +
+ row.data.recordsWritten[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " +
+ row.data.readRecords[columnIndex];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str = formatDuration(row.data.fetchWaitTime[columnIndex]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str = formatBytes(row.data.remoteBytesRead[columnIndex], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " +
+ row.data.writeRecords[columnIndex];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[columnIndex], type) : (formatDuration(row.data[columnIndex]));
+
+ }
+}
+
+function createDataTableForTaskSummaryMetricsTable(task_summary_metrics_table) {
+ var taskMetricsTable = "#summary-metrics-table";
+ if ($.fn.dataTable.isDataTable(taskMetricsTable)) {
+ taskSummaryMetricsDataTable.clear().draw();
+ taskSummaryMetricsDataTable.rows.add(task_summary_metrics_table).draw();
+ } else {
+ var task_conf = {
+ "data": task_summary_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ // Min
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 0);
+ }
+ },
+ // 25th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 1);
+ }
+ },
+ // Median
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 2);
+ }
+ },
+ // 75th percentile
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 3);
+ }
+ },
+ // Max
+ {
+ data: function (row, type) {
+ return displayRowsForSummaryMetricsTable(row, type, 4);
+ }
+ }
+ ],
+ "columnDefs": [
+ { "type": "duration", "targets": 1 },
+ { "type": "duration", "targets": 2 },
+ { "type": "duration", "targets": 3 },
+ { "type": "duration", "targets": 4 },
+ { "type": "duration", "targets": 5 }
+ ],
+ "paging": false,
+ "searching": false,
+ "order": [[0, "asc"]]
+ };
+ taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(task_conf);
+ }
+ task_summary_metrics_table_current_state_array = task_summary_metrics_table.slice();
+}
+
+var task_summary_metrics_table_array = [];
+var task_summary_metrics_table_current_state_array = [];
+var taskSummaryMetricsDataTable;
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='additionalMetrics'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
--- End diff --
The show additional metrics checkbox selections are note getting saved, the prior version of the UI would keep the choices.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21688
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r220260841
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224163278
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
+ var queryString = document.baseURI.split('?');
+ var words = document.baseURI.split('/');
+ var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1];
+ if (ind > 0) {
+ var appId = urlArray[2];
+ var indexOfProxy = words.indexOf("proxy");
--- End diff --
we have index of proxy in 2 places can we get rid of one
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #95902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95902/testReport)** for PR 21688 at commit [`573390d`](https://github.com/apache/spark/commit/573390d0a933e7a2641f944046442a136bba6cd8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232832492
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -162,26 +165,29 @@ private[v1] class StagesResource extends BaseAppResource {
// Performs pagination on the server side
def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
- val queryParams = queryParameters.keySet()
var columnNameToSort = queryParameters.getFirst("columnNameToSort")
+ // Sorting on Logs column will default to Index column sort
if (columnNameToSort.equalsIgnoreCase("Logs")) {
columnNameToSort = "Index"
}
val isAscendingStr = queryParameters.getFirst("order[0][dir]")
var pageStartIndex = 0
var pageLength = totalRecords
+ // We fetch only the desired rows upto the specified page length for all cases except when a
+ // search query is present, in that case, we need to fetch all the rows to perform the search
+ // on the entire table
if (!isSearch) {
pageStartIndex = queryParameters.getFirst("start").toInt
pageLength = queryParameters.getFirst("length").toInt
}
- return withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
+ withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
}
// Filters task list based on search parameter
def filterTaskList(
taskDataList: Seq[TaskData],
- searchValue: String): Seq[TaskData] = {
+ searchValue: String): Option[Seq[TaskData]] = {
--- End diff --
sorry my comment was confusing don't make the return value Option
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224217796
--- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js ---
@@ -74,3 +74,99 @@ function getTimeZone() {
return new Date().toString().match(/\((.*)\)/)[1];
}
}
+
+function formatLogsCells(execLogs, type) {
+ if (type !== 'display') return Object.keys(execLogs);
+ if (!execLogs) return;
+ var result = '';
+ $.each(execLogs, function (logName, logUrl) {
+ result += '<div><a href=' + logUrl + '>' + logName + '</a></div>'
+ });
+ return result;
+}
+
+function getStandAloneAppId(cb) {
+ var words = document.baseURI.split('/');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ cb(appId);
+ return;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ cb(appId);
+ return;
+ }
+ //Looks like Web UI is running in standalone mode
--- End diff --
add a space after //
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #92515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92515/testReport)** for PR 21688 at commit [`d79eb80`](https://github.com/apache/spark/commit/d79eb800bd8169cd063e1083d0444f755ffdb036).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224162463
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,872 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+ if (blockUICount == 0) {
+ $.unblockUI();
+ blockUICount++;
+ }
+});
+
+$(document).ajaxStart(function () {
+ if (blockUICount == 0) {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+ }
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "duration-pre": ConvertDurationString,
+
+ "duration-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "duration-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1
+// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1
+function stageEndPoint(appId) {
+ var urlRegex = /https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+ var urlArray = urlRegex.exec(document.baseURI);
+ var ind = urlArray.indexOf("proxy");
--- End diff --
use val and name it proxyInd or something, use different variable for history
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224864545
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
We can look into it more, I can't image its that hard to join, its just that much more data that is going over the wire, you are sending the entire ExecutorSummary for all executors when you really just need 2 fields out of it of some executors. Previously this was happening on the driver side so it didn't have to transfer the data. We can look at the data size and if it doesn't seem to bad we can do that for now and can always revisit if needed. The stage page executor table which uses this I wouldn't expect to be as used as the task one so probably not bad for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r217799929
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -962,7 +709,7 @@ private[ui] class TaskPagedTable(
}
}
-private[ui] object ApiHelper {
+private[spark] object ApiHelper {
--- End diff --
In order to use the function indexName in StagesResource.scala.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #99076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99076/testReport)** for PR 21688 at commit [`b3e2370`](https://github.com/apache/spark/commit/b3e23707869d87794992c12c7e782fd582a25ab7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96912/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #98563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98563/testReport)** for PR 21688 at commit [`a8b5862`](https://github.com/apache/spark/commit/a8b58629b0348e00a8ca5e83d124b29db4af656d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98357/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r218881945
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
+ <h4 class="title-table">Aggregated Metrics by Executor</h4>
+ </a>
+ <br>
+ <div class="container-fluid" id="toggle-aggregatedMetrics" hidden>
+ <div class="container-fluid">
+ <table id="summary-stages-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Executor ID</th>
+ <th>Logs</th>
+ <th>Address</th>
+ <th>Task Time</th>
+ <th>Total Tasks</th>
+ <th>Failed Tasks</th>
+ <th>Killed Tasks</th>
+ <th>Succeeded Tasks</th>
+ <th>
+ <span data-toggle="tooltip" data-placement="top"
+ title="Shows if this executor has been blacklisted by the scheduler due to task failures.">
+ Blacklisted</span>
+ </th>
+ <th>Input Size / Records</th>
+ <th>Output:</th>
+ <th>Shuffle Read: </th>
+ <th>Shuffle Write: </th>
+ <th>Shuffle Spill (Memory): </th>
--- End diff --
tooltips missing from this and a few of the others
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r232738700
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource {
withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
}
+ // This api needs to stay formatted exactly as it is below, since, it is being used by the
+ // datatables for the stages page.
+ @GET
+ @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+ def taskTable(
+ @PathParam("stageId") stageId: Int,
+ @PathParam("stageAttemptId") stageAttemptId: Int,
+ @QueryParam("details") @DefaultValue("true") details: Boolean,
+ @Context uriInfo: UriInfo):
+ HashMap[String, Object] = {
+ withUI { ui =>
+ val uriQueryParameters = uriInfo.getQueryParameters(true)
+ val totalRecords = uriQueryParameters.getFirst("numTasks")
+ var isSearch = false
+ var searchValue: String = null
+ var filteredRecords = totalRecords
+ var _tasksToShow: Seq[TaskData] = null
+ // The datatables client API sends a list of query parameters to the server which contain
+ // information like the columns to be sorted, search value typed by the user in the search
+ // box, pagination index etc. For more information on these query parameters,
+ // refer https://datatables.net/manual/server-side.
+ if (uriQueryParameters.getFirst("search[value]") != null &&
+ uriQueryParameters.getFirst("search[value]").length > 0) {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, true,
+ totalRecords.toInt)
+ isSearch = true
+ searchValue = uriQueryParameters.getFirst("search[value]")
+ } else {
+ _tasksToShow = doPagination(uriQueryParameters, stageId, stageAttemptId, false,
+ totalRecords.toInt)
+ }
+ val ret = new HashMap[String, Object]()
+ if (_tasksToShow.nonEmpty) {
+ // Performs server-side search based on input from user
+ if (isSearch) {
+ val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+ filteredRecords = filteredTaskList.length.toString
+ if (filteredTaskList.length > 0) {
+ val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+ val pageLength = uriQueryParameters.getFirst("length").toInt
+ ret.put("aaData", filteredTaskList.slice(pageStartIndex, pageStartIndex + pageLength))
+ } else {
+ ret.put("aaData", filteredTaskList)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ } else {
+ ret.put("aaData", _tasksToShow)
+ }
+ ret.put("recordsTotal", totalRecords)
+ ret.put("recordsFiltered", filteredRecords)
+ ret
+ }
+ }
+
+ // Performs pagination on the server side
+ def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int,
+ stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = {
+ val queryParams = queryParameters.keySet()
+ var columnNameToSort = queryParameters.getFirst("columnNameToSort")
+ if (columnNameToSort.equalsIgnoreCase("Logs")) {
+ columnNameToSort = "Index"
+ }
+ val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+ var pageStartIndex = 0
+ var pageLength = totalRecords
+ if (!isSearch) {
+ pageStartIndex = queryParameters.getFirst("start").toInt
+ pageLength = queryParameters.getFirst("length").toInt
+ }
+ return withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
+ indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+ }
+
+ // Filters task list based on search parameter
+ def filterTaskList(
+ taskDataList: Seq[TaskData],
+ searchValue: String): Seq[TaskData] = {
--- End diff --
change to an Option and you can just do a map on it to handle the case it might be empty
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r219230988
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ switch(columnKey) {
+ case "executorRunTime":
+ return "Duration";
+ break;
+
+ case "jvmGcTime":
+ return "GC Time";
+ break;
+
+ case "gettingResultTime":
+ return "Getting Result Time";
+ break;
+
+ case "inputMetrics":
+ return "Input Size / Records";
+ break;
+
+ case "outputMetrics":
+ return "Output Size / Records";
+ break;
+
+ case "peakExecutionMemory":
+ return "Peak Execution Memory";
+ break;
+
+ case "resultSerializationTime":
+ return "Result Serialization Time";
+ break;
+
+ case "schedulerDelay":
+ return "Scheduler Delay";
+ break;
+
+ case "diskBytesSpilled":
+ return "Shuffle spill (disk)";
+ break;
+
+ case "memoryBytesSpilled":
+ return "Shuffle spill (memory)";
+ break;
+
+ case "shuffleReadMetrics":
+ return "Shuffle Read Size / Records";
+ break;
+
+ case "shuffleWriteMetrics":
+ return "Shuffle Write Size / Records";
+ break;
+
+ case "executorDeserializeTime":
+ return "Task Deserialization Time";
+ break;
+
+ default:
+ return "NA";
+ }
+}
+
+$(document).ready(function () {
+ setDataTableDefaults();
+
+ $("#showAdditionalMetrics").append(
+ "<div><a id='taskMetric'>" +
+ "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+ " Show Additional Metrics" +
+ "</a></div>" +
+ "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" +
+ "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" +
+ "<div id='scheduler_delay'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" +
+ "<div id='task_deserialization_time'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" +
+ "<div id='shuffle_read_blocked_time'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" +
+ "<div id='shuffle_remote_reads'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" +
+ "<div id='result_serialization_time'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" +
+ "<div id='getting_result_time'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" +
+ "<div id='peak_execution_memory'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" +
+ "</div>");
+
+ $('#scheduler_delay').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " +
+ "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results.");
+ $('#task_deserialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task.");
+ $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines.");
+ $('#shuffle_remote_reads').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. ");
+ $('#result_serialization_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver.");
+ $('#getting_result_time').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task.");
+ $('#peak_execution_memory').attr("data-toggle", "tooltip")
+ .attr("data-placement", "bottom")
+ .attr("title", "Execution memory refers to the memory used by internal data structures created during " +
+ "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " +
+ "should be approximately the sum of the peak sizes across all such data structures created " +
+ "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " +
+ "external sort.");
+ $('#scheduler_delay').tooltip(true);
+ $('#task_deserialization_time').tooltip(true);
+ $('#shuffle_read_blocked_time').tooltip(true);
+ $('#shuffle_remote_reads').tooltip(true);
+ $('#result_serialization_time').tooltip(true);
+ $('#getting_result_time').tooltip(true);
+ $('#peak_execution_memory').tooltip(true);
+ tasksSummary = $("#active-tasks");
+ getStandAloneAppId(function (appId) {
+
+ var endPoint = stageEndPoint(appId);
+ $.getJSON(endPoint, function(response, status, jqXHR) {
+
+ var responseBody = response[0];
+ // prepare data for task aggregated metrics table
+ indices = Object.keys(responseBody.executorSummary);
+ var task_summary_table = [];
+ indices.forEach(function (ix) {
+ responseBody.executorSummary[ix].id = ix;
+ task_summary_table.push(responseBody.executorSummary[ix]);
+ });
+
+ // prepare data for accumulatorUpdates
+ var indices = Object.keys(responseBody.accumulatorUpdates);
+ var accumulator_table_all = [];
+ var accumulator_table = [];
+ indices.forEach(function (ix) {
+ accumulator_table_all.push(responseBody.accumulatorUpdates[ix]);
+ });
+
+ accumulator_table_all.forEach(function (x){
+ var name = (x.name).toString();
+ if(name.includes("internal.") == false){
+ accumulator_table.push(x);
+ }
+ });
+
+ // rendering the UI page
+ var data = {"executors": response};
+ $.get(createTemplateURI(appId, "stagespage"), function(template) {
+ tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data));
+
+ $("#taskMetric").click(function(){
+ $("#arrowtoggle1").toggleClass("arrow-open arrow-closed");
+ $("#toggle-metrics").toggle();
+ });
+
+ $("#aggregatedMetrics").click(function(){
+ $("#arrowtoggle2").toggleClass("arrow-open arrow-closed");
+ $("#toggle-aggregatedMetrics").toggle();
+ });
+
+ var task_metrics_table = [];
+ var stageAttemptId = getStageAttemptId();
+ var quantiles = "0,0.25,0.5,0.75,1.0";
+ $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+ var taskMetricIndices = Object.keys(taskMetricsResponse);
+ taskMetricIndices.forEach(function (ix) {
+ var columnName = getColumnNameForTaskMetricSummary(ix);
+ if (columnName == "Shuffle Read Size / Records") {
+ var row1 = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ var row2 = {
+ "metric": "Shuffle Read Blocked Time",
+ "data": taskMetricsResponse[ix]
+ };
+ var row3 = {
+ "metric": "Shuffle Remote Reads",
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row1);
+ task_metrics_table.push(row2);
+ task_metrics_table.push(row3);
+ }
+ else if (columnName != "NA") {
+ var row = {
+ "metric": columnName,
+ "data": taskMetricsResponse[ix]
+ };
+ task_metrics_table.push(row);
+ }
+ });
+
+ var taskMetricsTable = "#summary-metrics-table";
+ var task_conf = {
+ "data": task_metrics_table,
+ "columns": [
+ {data : 'metric'},
+ {
+ data: function (row, type) {
+ switch(row.metric) {
+ case 'Input Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesRead));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsRead));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Output Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.bytesWritten));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.recordsWritten));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.readBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.readRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ case 'Shuffle Read Blocked Time':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.fetchWaitTime));
+ var str = formatDuration(str1arr[0]);
+ return str;
+ break;
+
+ case 'Shuffle Remote Reads':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.remoteBytesRead));
+ var str = formatBytes(str1arr[0], type);
+ return str;
+ break;
+
+ case 'Shuffle Write Size / Records':
+ var str1arr = extractDataFromArrayString(JSON.stringify(row.data.writeBytes));
+ var str2arr = extractDataFromArrayString(JSON.stringify(row.data.writeRecords));
+ var str = formatBytes(str1arr[0], type) + " / " + str2arr[0];
+ return str;
+ break;
+
+ default:
+ return (row.metric == 'Peak Execution Memory' || row.metric == 'Shuffle spill (memory)'
+ || row.metric == 'Shuffle spill (disk)') ? formatBytes(row.data[0], type) : (formatDuration(row.data[0]));
+
+ }
+ }
+ },
+ {
+ data: function (row, type) {
--- End diff --
make this iterative if we can
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224176616
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---
@@ -0,0 +1,124 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<script id="stages-summary-template" type="text/html">
+ <h4 id="summaryMetricsTitle" class="title-table"></h4>
+ <div class="container-fluid">
+ <div class="container-fluid">
+ <table id="summary-metrics-table" class="table table-striped compact table-dataTable">
+ <thead>
+ <th>Metric</th>
+ <th>Min</th>
+ <th>25th percentile</th>
+ <th>Median</th>
+ <th>75th percentile</th>
+ <th>Max</th>
+ </thead>
+ <tbody>
+ </tbody>
+ </table>
+ </div>
+ </div>
+ <a id="aggregatedMetrics">
+ <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span>
--- End diff --
this table doesn't seem to save the state of twisty arrow on reload and it used to, is that something we can easily do?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97312/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...
Posted by abellina <gi...@git.apache.org>.
Github user abellina commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r216483202
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+ $.blockUI({message: '<h3>Loading Stage Page...</h3>'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+ "file-size-pre": ConvertDurationString,
+
+ "file-size-asc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+ },
+
+ "file-size-desc": function ( a, b ) {
+ a = ConvertDurationString( a );
+ b = ConvertDurationString( b );
+ return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+ }
+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+ var words = document.baseURI.split('/');
+ var words2 = document.baseURI.split('?');
+ var ind = words.indexOf("proxy");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind + 2).join('/');
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ }
+ ind = words.indexOf("history");
+ if (ind > 0) {
+ var appId = words[ind + 1];
+ var attemptId = words[ind + 2];
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ var newBaseURI = words.slice(0, ind).join('/');
+ if (isNaN(attemptId) || attemptId == "0") {
+ return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+ } else {
+ return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/stages/" + stageId;
+ }
+ }
+ var stageIdLen = words2[1].indexOf('&');
+ var stageId = words2[1].substr(3, stageIdLen - 3);
+ return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+ if(columnKey == "executorRunTime") {
--- End diff --
I'd replace with a switch statement.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21688
**[Test build #96517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96517/testReport)** for PR 21688 at commit [`af11138`](https://github.com/apache/spark/commit/af111380f9fad8b109842e53dbd7737b7d5c19bb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org