You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/06/06 02:51:45 UTC
(spark) branch branch-4.0 updated: [SPARK-52386][CORE][TESTS][FOLLOWUP] Add field sorting to JSON strings to ensure the consistency of the content output to golden files
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new d123aaf0736a [SPARK-52386][CORE][TESTS][FOLLOWUP] Add field sorting to JSON strings to ensure the consistency of the content output to golden files
d123aaf0736a is described below
commit d123aaf0736aa930c7b5d7c171f76c51cad22466
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Fri Jun 6 10:51:16 2025 +0800
[SPARK-52386][CORE][TESTS][FOLLOWUP] Add field sorting to JSON strings to ensure the consistency of the content output to golden files
### What changes were proposed in this pull request?
This pr adds sorting to the JSON content when regenerating golden files in `HistoryServerSuite` to ensure that the JSON strings output to the golden files are stable. Additionally, this pr refreshes the `HistoryServerExpectations` directory with the new approach for all golden files.
### Why are the changes needed?
https://github.com/apache/spark/pull/51072 has fixed and enhanced the way `HistoryServerSuite` generates golden files. However, the current approach is susceptible to the execution environment and cannot guarantee strong consistency in the content output to golden files every time. For instance, when I switched to a different computer to run the command for refreshing golden files, I noticed that the results had changed:
```
diff --git a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
index e485c0a3a85..500a74817fb 100644
--- a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
-9,9 +9,9
"sparkUser" : "lijunqing",
"completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "startTimeEpoch" : 1642039450519,
"endTimeEpoch" : 1642039536564,
- "lastUpdatedEpoch" : 0,
- "startTimeEpoch" : 1642039450519
+ "lastUpdatedEpoch" : 0
} ]
}, {
"id" : "application_1628109047826_1317105",
-24,9 +24,9
"sparkUser" : "john",
"completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "startTimeEpoch" : 1628637895333,
"endTimeEpoch" : 1628638170208,
- "lastUpdatedEpoch" : 0,
- "startTimeEpoch" : 1628637895333
+ "lastUpdatedEpoch" : 0
} ]
}, {
...
```
As can be seen, due to differences in the execution environment, the ordering of the output fields has changed. Therefore, field sorting has been added to the output JSON to ensure that the output content is not affected by the environment.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass Github Actions
- Executing
```
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "core/testOnly org.apache.spark.deploy.history.RocksDBBackendHistoryServerSuite"
```
in multiple environments has verified the consistency of the output content.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #51093 from LuciferYang/SPARK-52386-FOLLOWUP.
Lead-authored-by: yangjie01 <ya...@baidu.com>
Co-authored-by: YangJie <ya...@baidu.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
(cherry picked from commit ced7b03222335bbd8660a2716e5f5dfa45316453)
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../app_environment_expectation.json | 12 +-
.../application_list_json_expectation.json | 312 +--
.../complete_stage_list_json_expectation.json | 372 +--
.../completed_app_list_json_expectation.json | 312 +--
.../excludeOnFailure_for_stage_expectation.json | 1124 ++++----
...xcludeOnFailure_node_for_stage_expectation.json | 1550 +++++------
.../executor_list_json_expectation.json | 76 +-
...ist_with_executor_metrics_json_expectation.json | 392 +--
.../executor_memory_usage_expectation.json | 376 +--
...executor_node_excludeOnFailure_expectation.json | 376 +--
...e_excludeOnFailure_unexcluding_expectation.json | 340 +--
.../executor_resource_information_expectation.json | 194 +-
.../failed_stage_list_json_expectation.json | 130 +-
...from_multi_attempt_app_json_1__expectation.json | 20 +-
...from_multi_attempt_app_json_2__expectation.json | 20 +-
.../job_list_json_expectation.json | 60 +-
.../limit_app_list_json_expectation.json | 54 +-
.../maxDate2_app_list_json_expectation.json | 18 +-
.../maxDate_app_list_json_expectation.json | 36 +-
.../maxEndDate_app_list_json_expectation.json | 114 +-
...e_and_maxEndDate_app_list_json_expectation.json | 60 +-
.../minDate_app_list_json_expectation.json | 276 +-
...e_and_maxEndDate_app_list_json_expectation.json | 60 +-
.../minEndDate_app_list_json_expectation.json | 228 +-
.../miscellaneous_process_expectation.json | 12 +-
.../multiple_resource_profiles_expectation.json | 78 +-
.../one_app_json_expectation.json | 18 +-
.../one_app_multi_attempt_json_expectation.json | 30 +-
.../one_job_json_expectation.json | 20 +-
.../one_rdd_storage_json_expectation.json | 10 +-
..._json_details_with_failed_task_expectation.json | 190 +-
.../one_stage_attempt_json_expectation.json | 700 ++---
.../one_stage_json_expectation.json | 700 ++---
.../one_stage_json_with_details_expectation.json | 700 ++---
...ne_stage_json_with_partitionId_expectation.json | 816 +++---
.../stage_list_json_expectation.json | 496 ++--
...age_list_with_accumulable_json_expectation.json | 136 +-
.../stage_list_with_peak_metrics_expectation.json | 376 +--
.../stage_task_list_expectation.json | 1120 ++++----
...from_multi_attempt_app_json_1__expectation.json | 528 ++--
...from_multi_attempt_app_json_2__expectation.json | 528 ++--
...e_task_list_w__offset___length_expectation.json | 2800 ++++++++++----------
.../stage_task_list_w__sortBy_expectation.json | 1120 ++++----
...__sortBy_short_names___runtime_expectation.json | 1120 ++++----
...w__sortBy_short_names__runtime_expectation.json | 1120 ++++----
...st_w__status___offset___length_expectation.json | 112 +-
...___sortBy_short_names__runtime_expectation.json | 1120 ++++----
.../stage_task_list_w__status_expectation.json | 638 ++---
...age_task_list_with_partitionId_expectation.json | 448 ++--
...sk_summary_w__custom_quantiles_expectation.json | 36 +-
...ge_task_summary_w_shuffle_read_expectation.json | 36 +-
...e_task_summary_w_shuffle_write_expectation.json | 36 +-
.../stage_with_accumulable_json_expectation.json | 784 +++---
.../stage_with_peak_metrics_expectation.json | 1398 +++++-----
...stage_with_speculation_summary_expectation.json | 808 +++---
.../stage_with_summaries_expectation.json | 1576 +++++------
...succeeded_failed_job_list_json_expectation.json | 60 +-
.../succeeded_job_list_json_expectation.json | 40 +-
.../spark/deploy/history/HistoryServerSuite.scala | 8 +-
59 files changed, 13116 insertions(+), 13114 deletions(-)
diff --git a/core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json b/core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json
index 91f274f40278..51cbe21b8844 100644
--- a/core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json
@@ -1,13 +1,13 @@
{
+ "classpathEntries" : [ [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/RoaringBitmap-0.5.11.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.5.3.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/aopalliance-1.0.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar", "System Classpath" ], [...]
+ "hadoopProperties" : [ [ "mapreduce.jobhistory.client.thread-count", "10" ], [ "mapreduce.jobtracker.address", "local" ], [ "yarn.resourcemanager.scheduler.monitor.policies", "org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy" ] ],
+ "metricsProperties" : [ [ "*.sink.servlet.class", "org.apache.spark.metrics.sink.MetricsServlet" ], [ "*.sink.servlet.path", "/metrics/json" ], [ "applications.sink.servlet.path", "/metrics/applications/json" ], [ "master.sink.servlet.path", "/metrics/master/json" ] ],
+ "resourceProfiles" : [ ],
"runtime" : {
- "javaVersion" : "1.8.0_92 (Oracle Corporation)",
"javaHome" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre",
+ "javaVersion" : "1.8.0_92 (Oracle Corporation)",
"scalaVersion" : "version 2.11.8"
},
"sparkProperties" : [ [ "spark.app.id", "app-20161116163331-0000" ], [ "spark.app.name", "Spark shell" ], [ "spark.blacklist.application.maxFailedExecutorsPerNode", "2" ], [ "spark.blacklist.application.maxFailedTasksPerExecutor", "1" ], [ "spark.blacklist.enabled", "TRUE" ], [ "spark.blacklist.stage.maxFailedExecutorsPerNode", "3" ], [ "spark.blacklist.stage.maxFailedTasksPerExecutor", "3" ], [ "spark.blacklist.task.maxTaskAttemptsPerExecutor", "3" ], [ "spark.blacklist.task.maxTaskAt [...]
- "hadoopProperties" : [ [ "mapreduce.jobhistory.client.thread-count", "10" ], [ "mapreduce.jobtracker.address", "local" ], [ "yarn.resourcemanager.scheduler.monitor.policies", "org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy" ] ],
- "systemProperties" : [ [ "SPARK_SUBMIT", "true" ], [ "awt.toolkit", "sun.lwawt.macosx.LWCToolkit" ], [ "file.encoding", "UTF-8" ], [ "file.encoding.pkg", "sun.io" ], [ "file.separator", "/" ], [ "ftp.nonProxyHosts", "local|*.local|169.254/16|*.169.254/16" ], [ "gopherProxySet", "false" ], [ "http.nonProxyHosts", "local|*.local|169.254/16|*.169.254/16" ], [ "io.netty.maxDirectMemory", "0" ], [ "java.awt.graphicsenv", "sun.awt.CGraphicsEnvironment" ], [ "java.awt.printerjob", "sun.lwawt. [...]
- "metricsProperties" : [ [ "*.sink.servlet.class", "org.apache.spark.metrics.sink.MetricsServlet" ], [ "*.sink.servlet.path", "/metrics/json" ], [ "applications.sink.servlet.path", "/metrics/applications/json" ], [ "master.sink.servlet.path", "/metrics/master/json" ] ],
- "classpathEntries" : [ [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/RoaringBitmap-0.5.11.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.5.3.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/aopalliance-1.0.jar", "System Classpath" ], [ "/Users/Jose/IdeaProjects/spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar", "System Classpath" ], [...]
- "resourceProfiles" : [ ]
+ "systemProperties" : [ [ "SPARK_SUBMIT", "true" ], [ "awt.toolkit", "sun.lwawt.macosx.LWCToolkit" ], [ "file.encoding", "UTF-8" ], [ "file.encoding.pkg", "sun.io" ], [ "file.separator", "/" ], [ "ftp.nonProxyHosts", "local|*.local|169.254/16|*.169.254/16" ], [ "gopherProxySet", "false" ], [ "http.nonProxyHosts", "local|*.local|169.254/16|*.169.254/16" ], [ "io.netty.maxDirectMemory", "0" ], [ "java.awt.graphicsenv", "sun.awt.CGraphicsEnvironment" ], [ "java.awt.printerjob", "sun.lwawt. [...]
}
diff --git a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
index e485c0a3a851..eb05c2cab171 100644
--- a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
@@ -1,267 +1,267 @@
[ {
- "id" : "local-1642039451826",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2022-01-13T02:04:10.519GMT",
- "endTime" : "2022-01-13T02:05:36.564GMT",
- "lastUpdated" : "",
- "duration" : 86045,
- "sparkUser" : "lijunqing",
- "completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 86045,
+ "endTime" : "2022-01-13T02:05:36.564GMT",
"endTimeEpoch" : 1642039536564,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "lijunqing",
+ "startTime" : "2022-01-13T02:04:10.519GMT",
"startTimeEpoch" : 1642039450519
- } ]
+ } ],
+ "id" : "local-1642039451826",
+ "name" : "Spark shell"
}, {
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "lastUpdated" : "",
- "duration" : 274875,
- "sparkUser" : "john",
- "completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "completed" : true,
+ "duration" : 274875,
+ "endTime" : "2021-08-10T23:29:30.208GMT",
"endTimeEpoch" : 1628638170208,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "john",
+ "startTime" : "2021-08-10T23:24:55.333GMT",
"startTimeEpoch" : 1628637895333
- } ]
+ } ],
+ "id" : "application_1628109047826_1317105",
+ "name" : "Spark shell"
}, {
- "id" : "app-20200706201101-0003",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-07-07T03:11:00.235GMT",
- "endTime" : "2020-07-07T03:17:04.231GMT",
- "lastUpdated" : "",
- "duration" : 363996,
- "sparkUser" : "terryk",
- "completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 363996,
+ "endTime" : "2020-07-07T03:17:04.231GMT",
"endTimeEpoch" : 1594091824231,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "terryk",
+ "startTime" : "2020-07-07T03:11:00.235GMT",
"startTimeEpoch" : 1594091460235
- } ]
+ } ],
+ "id" : "app-20200706201101-0003",
+ "name" : "Spark shell"
}, {
- "id" : "application_1578436911597_0052",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-01-11T17:44:22.851GMT",
- "endTime" : "2020-01-11T17:46:42.615GMT",
- "lastUpdated" : "",
- "duration" : 139764,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 139764,
+ "endTime" : "2020-01-11T17:46:42.615GMT",
"endTimeEpoch" : 1578764802615,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2020-01-11T17:44:22.851GMT",
"startTimeEpoch" : 1578764662851
- } ]
+ } ],
+ "id" : "application_1578436911597_0052",
+ "name" : "Spark shell"
}, {
- "id" : "application_1555004656427_0144",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2019-07-02T21:02:17.180GMT",
- "endTime" : "2019-07-02T21:02:35.974GMT",
- "lastUpdated" : "",
- "duration" : 18794,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 18794,
+ "endTime" : "2019-07-02T21:02:35.974GMT",
"endTimeEpoch" : 1562101355974,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2019-07-02T21:02:17.180GMT",
"startTimeEpoch" : 1562101337180
- } ]
+ } ],
+ "id" : "application_1555004656427_0144",
+ "name" : "Spark shell"
}, {
- "id" : "application_1553914137147_0018",
- "name" : "LargeBlocks",
"attempts" : [ {
- "startTime" : "2019-04-08T20:39:44.286GMT",
- "endTime" : "2019-04-08T20:40:46.454GMT",
- "lastUpdated" : "",
- "duration" : 62168,
- "sparkUser" : "systest",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 62168,
+ "endTime" : "2019-04-08T20:40:46.454GMT",
"endTimeEpoch" : 1554756046454,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "systest",
+ "startTime" : "2019-04-08T20:39:44.286GMT",
"startTimeEpoch" : 1554755984286
- } ]
+ } ],
+ "id" : "application_1553914137147_0018",
+ "name" : "LargeBlocks"
}, {
- "id" : "application_1516285256255_0012",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-18T18:30:35.119GMT",
- "endTime" : "2018-01-18T18:38:27.938GMT",
- "lastUpdated" : "",
- "duration" : 472819,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 472819,
+ "endTime" : "2018-01-18T18:38:27.938GMT",
"endTimeEpoch" : 1516300707938,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-18T18:30:35.119GMT",
"startTimeEpoch" : 1516300235119
- } ]
+ } ],
+ "id" : "application_1516285256255_0012",
+ "name" : "Spark shell"
}, {
- "id" : "app-20180109111548-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-09T10:15:42.372GMT",
- "endTime" : "2018-01-09T10:24:37.606GMT",
- "lastUpdated" : "",
- "duration" : 535234,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 535234,
+ "endTime" : "2018-01-09T10:24:37.606GMT",
"endTimeEpoch" : 1515493477606,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-09T10:15:42.372GMT",
"startTimeEpoch" : 1515492942372
- } ]
+ } ],
+ "id" : "app-20180109111548-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161116163331-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-16T22:33:29.916GMT",
- "endTime" : "2016-11-16T22:33:40.587GMT",
- "lastUpdated" : "",
- "duration" : 10671,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10671,
+ "endTime" : "2016-11-16T22:33:40.587GMT",
"endTimeEpoch" : 1479335620587,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-16T22:33:29.916GMT",
"startTimeEpoch" : 1479335609916
- } ]
+ } ],
+ "id" : "app-20161116163331-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161115172038-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-15T23:20:37.079GMT",
- "endTime" : "2016-11-15T23:22:18.874GMT",
- "lastUpdated" : "",
- "duration" : 101795,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 101795,
+ "endTime" : "2016-11-15T23:22:18.874GMT",
"endTimeEpoch" : 1479252138874,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-15T23:20:37.079GMT",
"startTimeEpoch" : 1479252037079
- } ]
+ } ],
+ "id" : "app-20161115172038-0000",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381534",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:11.398GMT",
- "lastUpdated" : "",
- "duration" : 10505,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "1.4.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10505,
+ "endTime" : "2015-05-06T13:03:11.398GMT",
"endTimeEpoch" : 1430917391398,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
- } ]
+ } ],
+ "id" : "local-1430917381534",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
}, {
- "id" : "local-1425081759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-28T00:02:38.277GMT",
- "endTime" : "2015-02-28T00:02:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-28T00:02:46.912GMT",
"endTimeEpoch" : 1425081766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-28T00:02:38.277GMT",
"startTimeEpoch" : 1425081758277
- } ]
+ } ],
+ "id" : "local-1425081759269",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981780767",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:59.720GMT",
- "endTime" : "2015-02-03T16:43:08.731GMT",
- "lastUpdated" : "",
- "duration" : 9011,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 9011,
+ "endTime" : "2015-02-03T16:43:08.731GMT",
"endTimeEpoch" : 1422981788731,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:59.720GMT",
"startTimeEpoch" : 1422981779720
- } ]
+ } ],
+ "id" : "local-1422981780767",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:38.277GMT",
- "endTime" : "2015-02-03T16:42:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-03T16:42:46.912GMT",
"endTimeEpoch" : 1422981766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:38.277GMT",
"startTimeEpoch" : 1422981758277
- } ]
+ } ],
+ "id" : "local-1422981759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json
index 3fc669e11d9e..9812d879132a 100644
--- a/core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json
@@ -1,247 +1,247 @@
[ {
- "status" : "COMPLETE",
- "stageId" : 3,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 8,
- "numFailedTasks" : 0,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "submissionTime" : "2015-02-03T16:43:07.191GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:07.191GMT",
"completionTime" : "2015-02-03T16:43:07.226GMT",
- "executorDeserializeTime" : 36,
+ "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$line19.$read$$iwC$$iwC$$iwC.<init>(<console>:22)\n$line19.$read$$iwC$$iwC.<init>(<console>:24)\n$line19.$read$$iwC.<init>(<console>:26)\n$line19.$read.<init>(<console>:28)\n$line19.$read$.<init>(<console>:32)\n$line19.$read$.<clinit>(<console>)\n$line19.$eval$.<init>(<console>:7)\n$line19.$eval$.<clinit>(<console>)\n$line19.$eval.$print(<console>)\nsun.reflect.NativeMeth [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 36,
"executorRunTime" : 162,
- "executorCpuTime" : 0,
- "resultSize" : 14496,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:07.191GMT",
"inputBytes" : 160,
"inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 0,
+ "killedTasksSummary" : { },
+ "memoryBytesSpilled" : 0,
+ "name" : "count at <console>:17",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 8,
+ "numCompletedIndices" : 8,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "peakExecutionMemory" : 0,
+ "peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "rddIds" : [ 6, 5 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 14496,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 0,
- "shuffleWriteTime" : 0,
"shuffleWriteRecords" : 0,
- "name" : "count at <console>:17",
- "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$line19.$read$$iwC$$iwC$$iwC.<init>(<console>:22)\n$line19.$read$$iwC$$iwC.<init>(<console>:24)\n$line19.$read$$iwC.<init>(<console>:26)\n$line19.$read.<init>(<console>:28)\n$line19.$read$.<init>(<console>:32)\n$line19.$read$.<clinit>(<console>)\n$line19.$eval$.<init>(<console>:7)\n$line19.$eval$.<clinit>(<console>)\n$line19.$eval.$print(<console>)\nsun.reflect.NativeMeth [...]
- "schedulingPool" : "default",
- "rddIds" : [ 6, 5 ],
+ "shuffleWriteTime" : 0,
+ "stageId" : 3,
+ "status" : "COMPLETE",
+ "submissionTime" : "2015-02-03T16:43:07.191GMT"
+}, {
"accumulatorUpdates" : [ ],
+ "attemptId" : 0,
+ "completionTime" : "2015-02-03T16:43:06.286GMT",
+ "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 13,
+ "executorRunTime" : 3476,
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
+ "inputBytes" : 28000128,
+ "inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 152,
"killedTasksSummary" : { },
- "resourceProfileId" : 0,
+ "memoryBytesSpilled" : 0,
+ "name" : "map at <console>:14",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 8,
+ "numCompletedIndices" : 8,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakExecutionMemory" : 0,
"peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
-}, {
- "status" : "COMPLETE",
- "stageId" : 1,
- "attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 8,
- "numFailedTasks" : 0,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "submissionTime" : "2015-02-03T16:43:05.829GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
- "completionTime" : "2015-02-03T16:43:06.286GMT",
- "executorDeserializeTime" : 13,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 3476,
- "executorCpuTime" : 0,
- "resultSize" : 15216,
- "jvmGcTime" : 152,
+ "rddIds" : [ 1, 0 ],
+ "resourceProfileId" : 0,
"resultSerializationTime" : 9,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
- "inputBytes" : 28000128,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "resultSize" : 15216,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 13180,
- "shuffleWriteTime" : 692000,
"shuffleWriteRecords" : 0,
- "name" : "map at <console>:14",
- "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
- "schedulingPool" : "default",
- "rddIds" : [ 1, 0 ],
+ "shuffleWriteTime" : 692000,
+ "stageId" : 1,
+ "status" : "COMPLETE",
+ "submissionTime" : "2015-02-03T16:43:05.829GMT"
+}, {
"accumulatorUpdates" : [ ],
+ "attemptId" : 0,
+ "completionTime" : "2015-02-03T16:43:04.819GMT",
+ "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$line9.$read$$iwC$$iwC$$iwC.<init>(<console>:20)\n$line9.$read$$iwC$$iwC.<init>(<console>:22)\n$line9.$read$$iwC.<init>(<console>:24)\n$line9.$read.<init>(<console>:26)\n$line9.$read$.<init>(<console>:30)\n$line9.$read$.<clinit>(<console>)\n$line9.$eval$.<init>(<console>:7)\n$line9.$eval$.<clinit>(<console>)\n$line9.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessor [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 91,
+ "executorRunTime" : 4338,
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:04.234GMT",
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 200,
"killedTasksSummary" : { },
- "resourceProfileId" : 0,
+ "memoryBytesSpilled" : 0,
+ "name" : "count at <console>:15",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 8,
+ "numCompletedIndices" : 8,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakExecutionMemory" : 0,
"peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
-}, {
- "status" : "COMPLETE",
- "stageId" : 0,
- "attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 8,
- "numFailedTasks" : 0,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "submissionTime" : "2015-02-03T16:43:04.228GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:04.234GMT",
- "completionTime" : "2015-02-03T16:43:04.819GMT",
- "executorDeserializeTime" : 91,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 4338,
- "executorCpuTime" : 0,
- "resultSize" : 10144,
- "jvmGcTime" : 200,
+ "rddIds" : [ 0 ],
+ "resourceProfileId" : 0,
"resultSerializationTime" : 5,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "resultSize" : 10144,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 0,
- "shuffleWriteTime" : 0,
"shuffleWriteRecords" : 0,
- "name" : "count at <console>:15",
- "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$line9.$read$$iwC$$iwC$$iwC.<init>(<console>:20)\n$line9.$read$$iwC$$iwC.<init>(<console>:22)\n$line9.$read$$iwC.<init>(<console>:24)\n$line9.$read.<init>(<console>:26)\n$line9.$read$.<init>(<console>:30)\n$line9.$read$.<clinit>(<console>)\n$line9.$eval$.<init>(<console>:7)\n$line9.$eval$.<clinit>(<console>)\n$line9.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessor [...]
- "schedulingPool" : "default",
- "rddIds" : [ 0 ],
- "accumulatorUpdates" : [ ],
- "killedTasksSummary" : { },
- "resourceProfileId" : 0,
- "peakExecutorMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
+ "shuffleWriteTime" : 0,
+ "stageId" : 0,
+ "status" : "COMPLETE",
+ "submissionTime" : "2015-02-03T16:43:04.228GMT"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
index e485c0a3a851..eb05c2cab171 100644
--- a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
@@ -1,267 +1,267 @@
[ {
- "id" : "local-1642039451826",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2022-01-13T02:04:10.519GMT",
- "endTime" : "2022-01-13T02:05:36.564GMT",
- "lastUpdated" : "",
- "duration" : 86045,
- "sparkUser" : "lijunqing",
- "completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 86045,
+ "endTime" : "2022-01-13T02:05:36.564GMT",
"endTimeEpoch" : 1642039536564,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "lijunqing",
+ "startTime" : "2022-01-13T02:04:10.519GMT",
"startTimeEpoch" : 1642039450519
- } ]
+ } ],
+ "id" : "local-1642039451826",
+ "name" : "Spark shell"
}, {
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "lastUpdated" : "",
- "duration" : 274875,
- "sparkUser" : "john",
- "completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "completed" : true,
+ "duration" : 274875,
+ "endTime" : "2021-08-10T23:29:30.208GMT",
"endTimeEpoch" : 1628638170208,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "john",
+ "startTime" : "2021-08-10T23:24:55.333GMT",
"startTimeEpoch" : 1628637895333
- } ]
+ } ],
+ "id" : "application_1628109047826_1317105",
+ "name" : "Spark shell"
}, {
- "id" : "app-20200706201101-0003",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-07-07T03:11:00.235GMT",
- "endTime" : "2020-07-07T03:17:04.231GMT",
- "lastUpdated" : "",
- "duration" : 363996,
- "sparkUser" : "terryk",
- "completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 363996,
+ "endTime" : "2020-07-07T03:17:04.231GMT",
"endTimeEpoch" : 1594091824231,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "terryk",
+ "startTime" : "2020-07-07T03:11:00.235GMT",
"startTimeEpoch" : 1594091460235
- } ]
+ } ],
+ "id" : "app-20200706201101-0003",
+ "name" : "Spark shell"
}, {
- "id" : "application_1578436911597_0052",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-01-11T17:44:22.851GMT",
- "endTime" : "2020-01-11T17:46:42.615GMT",
- "lastUpdated" : "",
- "duration" : 139764,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 139764,
+ "endTime" : "2020-01-11T17:46:42.615GMT",
"endTimeEpoch" : 1578764802615,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2020-01-11T17:44:22.851GMT",
"startTimeEpoch" : 1578764662851
- } ]
+ } ],
+ "id" : "application_1578436911597_0052",
+ "name" : "Spark shell"
}, {
- "id" : "application_1555004656427_0144",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2019-07-02T21:02:17.180GMT",
- "endTime" : "2019-07-02T21:02:35.974GMT",
- "lastUpdated" : "",
- "duration" : 18794,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 18794,
+ "endTime" : "2019-07-02T21:02:35.974GMT",
"endTimeEpoch" : 1562101355974,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2019-07-02T21:02:17.180GMT",
"startTimeEpoch" : 1562101337180
- } ]
+ } ],
+ "id" : "application_1555004656427_0144",
+ "name" : "Spark shell"
}, {
- "id" : "application_1553914137147_0018",
- "name" : "LargeBlocks",
"attempts" : [ {
- "startTime" : "2019-04-08T20:39:44.286GMT",
- "endTime" : "2019-04-08T20:40:46.454GMT",
- "lastUpdated" : "",
- "duration" : 62168,
- "sparkUser" : "systest",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 62168,
+ "endTime" : "2019-04-08T20:40:46.454GMT",
"endTimeEpoch" : 1554756046454,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "systest",
+ "startTime" : "2019-04-08T20:39:44.286GMT",
"startTimeEpoch" : 1554755984286
- } ]
+ } ],
+ "id" : "application_1553914137147_0018",
+ "name" : "LargeBlocks"
}, {
- "id" : "application_1516285256255_0012",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-18T18:30:35.119GMT",
- "endTime" : "2018-01-18T18:38:27.938GMT",
- "lastUpdated" : "",
- "duration" : 472819,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 472819,
+ "endTime" : "2018-01-18T18:38:27.938GMT",
"endTimeEpoch" : 1516300707938,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-18T18:30:35.119GMT",
"startTimeEpoch" : 1516300235119
- } ]
+ } ],
+ "id" : "application_1516285256255_0012",
+ "name" : "Spark shell"
}, {
- "id" : "app-20180109111548-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-09T10:15:42.372GMT",
- "endTime" : "2018-01-09T10:24:37.606GMT",
- "lastUpdated" : "",
- "duration" : 535234,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 535234,
+ "endTime" : "2018-01-09T10:24:37.606GMT",
"endTimeEpoch" : 1515493477606,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-09T10:15:42.372GMT",
"startTimeEpoch" : 1515492942372
- } ]
+ } ],
+ "id" : "app-20180109111548-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161116163331-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-16T22:33:29.916GMT",
- "endTime" : "2016-11-16T22:33:40.587GMT",
- "lastUpdated" : "",
- "duration" : 10671,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10671,
+ "endTime" : "2016-11-16T22:33:40.587GMT",
"endTimeEpoch" : 1479335620587,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-16T22:33:29.916GMT",
"startTimeEpoch" : 1479335609916
- } ]
+ } ],
+ "id" : "app-20161116163331-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161115172038-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-15T23:20:37.079GMT",
- "endTime" : "2016-11-15T23:22:18.874GMT",
- "lastUpdated" : "",
- "duration" : 101795,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 101795,
+ "endTime" : "2016-11-15T23:22:18.874GMT",
"endTimeEpoch" : 1479252138874,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-15T23:20:37.079GMT",
"startTimeEpoch" : 1479252037079
- } ]
+ } ],
+ "id" : "app-20161115172038-0000",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381534",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:11.398GMT",
- "lastUpdated" : "",
- "duration" : 10505,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "1.4.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10505,
+ "endTime" : "2015-05-06T13:03:11.398GMT",
"endTimeEpoch" : 1430917391398,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
- } ]
+ } ],
+ "id" : "local-1430917381534",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
}, {
- "id" : "local-1425081759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-28T00:02:38.277GMT",
- "endTime" : "2015-02-28T00:02:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-28T00:02:46.912GMT",
"endTimeEpoch" : 1425081766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-28T00:02:38.277GMT",
"startTimeEpoch" : 1425081758277
- } ]
+ } ],
+ "id" : "local-1425081759269",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981780767",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:59.720GMT",
- "endTime" : "2015-02-03T16:43:08.731GMT",
- "lastUpdated" : "",
- "duration" : 9011,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 9011,
+ "endTime" : "2015-02-03T16:43:08.731GMT",
"endTimeEpoch" : 1422981788731,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:59.720GMT",
"startTimeEpoch" : 1422981779720
- } ]
+ } ],
+ "id" : "local-1422981780767",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:38.277GMT",
- "endTime" : "2015-02-03T16:42:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-03T16:42:46.912GMT",
"endTimeEpoch" : 1422981766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:38.277GMT",
"startTimeEpoch" : 1422981758277
- } ]
+ } ],
+ "id" : "local-1422981759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json b/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
index 2472dec11740..fc666fa2928c 100644
--- a/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
@@ -1,967 +1,967 @@
{
- "status" : "COMPLETE",
- "stageId" : 0,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 10,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 10,
- "numFailedTasks" : 2,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 10,
- "submissionTime" : "2018-01-09T10:21:18.152GMT",
- "firstTaskLaunchedTime" : "2018-01-09T10:21:18.347GMT",
"completionTime" : "2018-01-09T10:21:19.062GMT",
- "executorDeserializeTime" : 327,
+ "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:26)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)\n$line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:40)\n$line17.$read$$iw$$iw$$iw.<init>(<console>:42)\n$line17.$read$$iw$$iw.<init>(<console>:44)\n$line17.$read$$iw.<init>(<console>:46)\n [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 269916000,
"executorDeserializeCpuTime" : 225900000,
+ "executorDeserializeTime" : 327,
"executorRunTime" : 761,
- "executorCpuTime" : 269916000,
- "resultSize" : 10376,
- "jvmGcTime" : 27,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorSummary" : {
+ "0" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 2,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : true,
+ "isExcludedForStage" : true,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 0,
+ "shuffleWriteRecords" : 0,
+ "succeededTasks" : 0,
+ "taskTime" : 589
+ },
+ "1" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : false,
+ "isExcludedForStage" : false,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 460,
+ "shuffleWriteRecords" : 10,
+ "succeededTasks" : 10,
+ "taskTime" : 708
+ }
+ },
+ "firstTaskLaunchedTime" : "2018-01-09T10:21:18.347GMT",
"inputBytes" : 0,
"inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 27,
+ "killedTasksSummary" : { },
+ "memoryBytesSpilled" : 0,
+ "name" : "map at <console>:26",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 10,
+ "numCompletedIndices" : 10,
+ "numFailedTasks" : 2,
+ "numKilledTasks" : 0,
+ "numTasks" : 10,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "peakExecutionMemory" : 0,
+ "peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "rddIds" : [ 1, 0 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 10376,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 460,
- "shuffleWriteTime" : 8711515,
"shuffleWriteRecords" : 10,
- "name" : "map at <console>:26",
- "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:26)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)\n$line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:40)\n$line17.$read$$iw$$iw$$iw.<init>(<console>:42)\n$line17.$read$$iw$$iw.<init>(<console>:44)\n$line17.$read$$iw.<init>(<console>:46)\n [...]
- "schedulingPool" : "default",
- "rddIds" : [ 1, 0 ],
- "accumulatorUpdates" : [ ],
+ "shuffleWriteTime" : 8711515,
+ "stageId" : 0,
+ "status" : "COMPLETE",
+ "submissionTime" : "2018-01-09T10:21:18.152GMT",
"tasks" : {
"0" : {
- "taskId" : 0,
- "index" : 0,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.347GMT",
"duration" : 562,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:26)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
"executorId" : "0",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stderr",
+ "stdout" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 0,
+ "launchTime" : "2018-01-09T10:21:18.347GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 102,
+ "speculative" : false,
"status" : "FAILED",
+ "taskId" : 0,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:26)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
"taskMetrics" : {
- "executorDeserializeTime" : 0,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
"executorRunTime" : 460,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 14,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 14,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 0,
- "writeTime" : 3873006,
- "recordsWritten" : 0
+ "recordsWritten" : 0,
+ "writeTime" : 3873006
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stdout",
- "stderr" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stderr"
- },
- "schedulerDelay" : 102,
- "gettingResultTime" : 0
+ }
},
- "5" : {
- "taskId" : 5,
- "index" : 3,
+ "1" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.958GMT",
- "duration" : 22,
+ "duration" : 565,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 1,
+ "launchTime" : "2018-01-09T10:21:18.364GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 51,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 1,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 3,
- "executorDeserializeCpuTime" : 2586000,
- "executorRunTime" : 9,
- "executorCpuTime" : 9635000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 198479000,
+ "executorDeserializeCpuTime" : 200029000,
+ "executorDeserializeTime" : 301,
+ "executorRunTime" : 212,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 13,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1115,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 262919,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 2409488
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 10,
- "gettingResultTime" : 0
+ }
},
"10" : {
- "taskId" : 10,
- "index" : 8,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:19.034GMT",
"duration" : 12,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 8,
+ "launchTime" : "2018-01-09T10:21:19.034GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 4,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 10,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 6157000,
"executorDeserializeCpuTime" : 1803000,
+ "executorDeserializeTime" : 2,
"executorRunTime" : 6,
- "executorCpuTime" : 6157000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 243647,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 243647
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 4,
- "gettingResultTime" : 0
+ }
},
- "1" : {
- "taskId" : 1,
- "index" : 1,
+ "11" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.364GMT",
- "duration" : 565,
+ "duration" : 15,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 9,
+ "launchTime" : "2018-01-09T10:21:19.045GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 6,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 11,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 301,
- "executorDeserializeCpuTime" : 200029000,
- "executorRunTime" : 212,
- "executorCpuTime" : 198479000,
- "resultSize" : 1115,
- "jvmGcTime" : 13,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 6676000,
+ "executorDeserializeCpuTime" : 2017000,
+ "executorDeserializeTime" : 3,
+ "executorRunTime" : 6,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 2409488,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 233652
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 51,
- "gettingResultTime" : 0
+ }
},
- "6" : {
- "taskId" : 6,
- "index" : 4,
+ "2" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.980GMT",
- "duration" : 16,
- "executorId" : "1",
+ "duration" : 27,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:26)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "executorId" : "0",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stderr",
+ "stdout" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
- "status" : "SUCCESS",
- "taskLocality" : "PROCESS_LOCAL",
+ "index" : 2,
+ "launchTime" : "2018-01-09T10:21:18.899GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 11,
"speculative" : false,
- "accumulatorUpdates" : [ ],
+ "status" : "FAILED",
+ "taskId" : 2,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 3,
- "executorDeserializeCpuTime" : 2610000,
- "executorRunTime" : 10,
- "executorCpuTime" : 9622000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
+ "executorRunTime" : 16,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 46,
- "writeTime" : 385110,
- "recordsWritten" : 1
+ "bytesWritten" : 0,
+ "recordsWritten" : 0,
+ "writeTime" : 126128
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 3,
- "gettingResultTime" : 0
+ }
},
- "9" : {
- "taskId" : 9,
- "index" : 7,
- "attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:19.022GMT",
- "duration" : 12,
+ "3" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 1,
+ "duration" : 24,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 0,
+ "launchTime" : "2018-01-09T10:21:18.919GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 6,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 3,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
- "executorDeserializeCpuTime" : 1981000,
- "executorRunTime" : 7,
- "executorCpuTime" : 6335000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 9364000,
+ "executorDeserializeCpuTime" : 8878000,
+ "executorDeserializeTime" : 8,
+ "executorRunTime" : 10,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 259354,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 207014
}
- },
+ }
+ },
+ "4" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 1,
+ "duration" : 16,
+ "executorId" : "1",
"executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
},
- "schedulerDelay" : 3,
- "gettingResultTime" : 0
- },
- "2" : {
- "taskId" : 2,
+ "gettingResultTime" : 0,
+ "host" : "172.30.65.138",
"index" : 2,
- "attempt" : 0,
+ "launchTime" : "2018-01-09T10:21:18.943GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.899GMT",
- "duration" : 27,
- "executorId" : "0",
- "host" : "172.30.65.138",
- "status" : "FAILED",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 5,
"speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:26)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "status" : "SUCCESS",
+ "taskId" : 4,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 0,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 16,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 9207000,
+ "executorDeserializeCpuTime" : 2211000,
+ "executorDeserializeTime" : 2,
+ "executorRunTime" : 9,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 0,
- "writeTime" : 126128,
- "recordsWritten" : 0
+ "bytesWritten" : 46,
+ "recordsWritten" : 1,
+ "writeTime" : 292381
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stdout",
- "stderr" : "http://172.30.65.138:64279/logPage/?appId=app-20180109111548-0000&executorId=0&logType=stderr"
- },
- "schedulerDelay" : 11,
- "gettingResultTime" : 0
+ }
},
- "7" : {
- "taskId" : 7,
- "index" : 5,
+ "5" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.996GMT",
- "duration" : 15,
+ "duration" : 22,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 3,
+ "launchTime" : "2018-01-09T10:21:18.958GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 10,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 5,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
- "executorDeserializeCpuTime" : 2231000,
- "executorRunTime" : 9,
- "executorCpuTime" : 8407000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 9635000,
+ "executorDeserializeCpuTime" : 2586000,
+ "executorDeserializeTime" : 3,
+ "executorRunTime" : 9,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 205520,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 262919
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 4,
- "gettingResultTime" : 0
+ }
},
- "3" : {
- "taskId" : 3,
- "index" : 0,
- "attempt" : 1,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.919GMT",
- "duration" : 24,
+ "6" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 0,
+ "duration" : 16,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 4,
+ "launchTime" : "2018-01-09T10:21:18.980GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 3,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 6,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 8,
- "executorDeserializeCpuTime" : 8878000,
- "executorRunTime" : 10,
- "executorCpuTime" : 9364000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 9622000,
+ "executorDeserializeCpuTime" : 2610000,
+ "executorDeserializeTime" : 3,
+ "executorRunTime" : 10,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 207014,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 385110
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 6,
- "gettingResultTime" : 0
+ }
},
- "11" : {
- "taskId" : 11,
- "index" : 9,
+ "7" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:19.045GMT",
"duration" : 15,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
- "status" : "SUCCESS",
- "taskLocality" : "PROCESS_LOCAL",
+ "index" : 5,
+ "launchTime" : "2018-01-09T10:21:18.996GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 4,
"speculative" : false,
- "accumulatorUpdates" : [ ],
+ "status" : "SUCCESS",
+ "taskId" : 7,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 3,
- "executorDeserializeCpuTime" : 2017000,
- "executorRunTime" : 6,
- "executorCpuTime" : 6676000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 8407000,
+ "executorDeserializeCpuTime" : 2231000,
+ "executorDeserializeTime" : 2,
+ "executorRunTime" : 9,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 233652,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 205520
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 6,
- "gettingResultTime" : 0
+ }
},
"8" : {
- "taskId" : 8,
- "index" : 6,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:19.011GMT",
"duration" : 11,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 6,
+ "launchTime" : "2018-01-09T10:21:19.011GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 3,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 8,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 1,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 6034000,
"executorDeserializeCpuTime" : 1554000,
+ "executorDeserializeTime" : 1,
"executorRunTime" : 7,
- "executorCpuTime" : 6034000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 213296,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 213296
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 3,
- "gettingResultTime" : 0
+ }
},
- "4" : {
- "taskId" : 4,
- "index" : 2,
- "attempt" : 1,
- "partitionId" : -1,
- "launchTime" : "2018-01-09T10:21:18.943GMT",
- "duration" : 16,
+ "9" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 0,
+ "duration" : 12,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr",
+ "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout"
+ },
+ "gettingResultTime" : 0,
"host" : "172.30.65.138",
+ "index" : 7,
+ "launchTime" : "2018-01-09T10:21:19.022GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 3,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 9,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
- "executorDeserializeCpuTime" : 2211000,
- "executorRunTime" : 9,
- "executorCpuTime" : 9207000,
- "resultSize" : 1029,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 6335000,
+ "executorDeserializeCpuTime" : 1981000,
+ "executorDeserializeTime" : 2,
+ "executorRunTime" : 7,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1029,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 46,
- "writeTime" : 292381,
- "recordsWritten" : 1
+ "recordsWritten" : 1,
+ "writeTime" : 259354
}
- },
- "executorLogs" : {
- "stdout" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stdout",
- "stderr" : "http://172.30.65.138:64278/logPage/?appId=app-20180109111548-0000&executorId=1&logType=stderr"
- },
- "schedulerDelay" : 5,
- "gettingResultTime" : 0
- }
- },
- "executorSummary" : {
- "0" : {
- "taskTime" : 589,
- "failedTasks" : 2,
- "succeededTasks" : 0,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 0,
- "shuffleWriteRecords" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : true,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : true
- },
- "1" : {
- "taskTime" : 708,
- "failedTasks" : 0,
- "succeededTasks" : 10,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 460,
- "shuffleWriteRecords" : 10,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : false,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : false
+ }
}
- },
- "killedTasksSummary" : { },
- "resourceProfileId" : 0,
- "peakExecutorMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
+ }
}
diff --git a/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json b/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json
index 5b1597d4544a..309809d64553 100644
--- a/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json
@@ -1,1230 +1,1230 @@
{
- "status" : "COMPLETE",
- "stageId" : 0,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 10,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 10,
- "numFailedTasks" : 4,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 10,
- "submissionTime" : "2018-01-18T18:33:12.658GMT",
- "firstTaskLaunchedTime" : "2018-01-18T18:33:12.816GMT",
"completionTime" : "2018-01-18T18:33:15.279GMT",
- "executorDeserializeTime" : 3679,
+ "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:35)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:37)\n$line15.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:39)\n$line15.$read$$iw$$iw$$iw$$iw.<init>(<console>:41)\n$line15.$read$$iw$$iw$$iw.<init>(<console>:43)\n$line15.$read$$iw$$iw.<init>(<console>:45)\n$line15.$read$$iw.<init>(<console>:47)\n [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 1163210819,
"executorDeserializeCpuTime" : 1029819716,
+ "executorDeserializeTime" : 3679,
"executorRunTime" : 5080,
- "executorCpuTime" : 1163210819,
- "resultSize" : 10824,
- "jvmGcTime" : 370,
- "resultSerializationTime" : 5,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorSummary" : {
+ "1" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : false,
+ "isExcludedForStage" : false,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 585,
+ "shuffleWriteRecords" : 12,
+ "succeededTasks" : 4,
+ "taskTime" : 2411
+ },
+ "2" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : false,
+ "isExcludedForStage" : false,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 732,
+ "shuffleWriteRecords" : 15,
+ "succeededTasks" : 5,
+ "taskTime" : 2446
+ },
+ "3" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 0,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : true,
+ "isExcludedForStage" : true,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 144,
+ "shuffleWriteRecords" : 3,
+ "succeededTasks" : 1,
+ "taskTime" : 1774
+ },
+ "4" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 2,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : true,
+ "isExcludedForStage" : true,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 0,
+ "shuffleWriteRecords" : 0,
+ "succeededTasks" : 0,
+ "taskTime" : 1589
+ },
+ "5" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 2,
+ "inputBytes" : 0,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : true,
+ "isExcludedForStage" : true,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 0,
+ "shuffleWriteRecords" : 0,
+ "succeededTasks" : 0,
+ "taskTime" : 1579
+ }
+ },
+ "firstTaskLaunchedTime" : "2018-01-18T18:33:12.816GMT",
"inputBytes" : 0,
"inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 370,
+ "killedTasksSummary" : { },
+ "memoryBytesSpilled" : 0,
+ "name" : "map at <console>:27",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 10,
+ "numCompletedIndices" : 10,
+ "numFailedTasks" : 4,
+ "numKilledTasks" : 0,
+ "numTasks" : 10,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "peakExecutionMemory" : 0,
+ "peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "rddIds" : [ 1, 0 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 5,
+ "resultSize" : 10824,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 1461,
- "shuffleWriteTime" : 33251697,
"shuffleWriteRecords" : 30,
- "name" : "map at <console>:27",
- "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:35)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:37)\n$line15.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:39)\n$line15.$read$$iw$$iw$$iw$$iw.<init>(<console>:41)\n$line15.$read$$iw$$iw$$iw.<init>(<console>:43)\n$line15.$read$$iw$$iw.<init>(<console>:45)\n$line15.$read$$iw.<init>(<console>:47)\n [...]
- "schedulingPool" : "default",
- "rddIds" : [ 1, 0 ],
- "accumulatorUpdates" : [ ],
+ "shuffleWriteTime" : 33251697,
+ "stageId" : 0,
+ "status" : "COMPLETE",
+ "submissionTime" : "2018-01-18T18:33:12.658GMT",
"tasks" : {
"0" : {
- "taskId" : 0,
- "index" : 0,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:12.816GMT",
"duration" : 2064,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
"host" : "apiros-3.gce.test.com",
+ "index" : 0,
+ "launchTime" : "2018-01-18T18:33:12.816GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 68,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 0,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 1081,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 368865439,
"executorDeserializeCpuTime" : 353981050,
+ "executorDeserializeTime" : 1081,
"executorRunTime" : 914,
- "executorCpuTime" : 368865439,
- "resultSize" : 1134,
- "jvmGcTime" : 75,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 75,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1134,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 144,
- "writeTime" : 3662221,
- "recordsWritten" : 3
+ "recordsWritten" : 3,
+ "writeTime" : 3662221
}
- },
- "executorLogs" : {
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
- },
- "schedulerDelay" : 68,
- "gettingResultTime" : 0
+ }
},
- "5" : {
- "taskId" : 5,
- "index" : 5,
+ "1" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:14.320GMT",
- "duration" : 73,
+ "duration" : 1506,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
"executorId" : "5",
+ "executorLogs" : {
+ "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
"host" : "apiros-2.gce.test.com",
+ "index" : 1,
+ "launchTime" : "2018-01-18T18:33:12.832GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 174,
+ "speculative" : false,
"status" : "FAILED",
+ "taskId" : 1,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
"taskMetrics" : {
- "executorDeserializeTime" : 0,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 27,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
+ "executorRunTime" : 1332,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 33,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 0,
- "writeTime" : 191901,
- "recordsWritten" : 0
+ "recordsWritten" : 0,
+ "writeTime" : 3075188
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 46,
- "gettingResultTime" : 0
+ }
},
"10" : {
- "taskId" : 10,
- "index" : 1,
+ "accumulatorUpdates" : [ ],
"attempt" : 1,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:15.069GMT",
"duration" : 132,
"executorId" : "2",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
"host" : "apiros-3.gce.test.com",
+ "index" : 1,
+ "launchTime" : "2018-01-18T18:33:15.069GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 50,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 10,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 5,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 20826337,
"executorDeserializeCpuTime" : 4598966,
+ "executorDeserializeTime" : 5,
"executorRunTime" : 76,
- "executorCpuTime" : 20826337,
- "resultSize" : 1091,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1091,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 144,
- "writeTime" : 301705,
- "recordsWritten" : 3
+ "recordsWritten" : 3,
+ "writeTime" : 301705
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 50,
- "gettingResultTime" : 0
+ }
},
- "1" : {
- "taskId" : 1,
- "index" : 1,
+ "11" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 93,
+ "executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-3.gce.test.com",
+ "index" : 7,
+ "launchTime" : "2018-01-18T18:33:15.072GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:12.832GMT",
- "duration" : 1506,
- "executorId" : "5",
- "host" : "apiros-2.gce.test.com",
- "status" : "FAILED",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 12,
"speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "status" : "SUCCESS",
+ "taskId" : 11,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 0,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 1332,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 33,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 21689428,
+ "executorDeserializeCpuTime" : 4239884,
+ "executorDeserializeTime" : 4,
+ "executorRunTime" : 77,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1048,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 0,
- "writeTime" : 3075188,
- "recordsWritten" : 0
+ "bytesWritten" : 147,
+ "recordsWritten" : 3,
+ "writeTime" : 323898
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 174,
- "gettingResultTime" : 0
+ }
},
- "6" : {
- "taskId" : 6,
- "index" : 6,
+ "12" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 60,
+ "executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-3.gce.test.com",
+ "index" : 8,
+ "launchTime" : "2018-01-18T18:33:15.165GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:14.323GMT",
- "duration" : 67,
- "executorId" : "4",
- "host" : "apiros-2.gce.test.com",
- "status" : "FAILED",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 22,
"speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "status" : "SUCCESS",
+ "taskId" : 12,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 0,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 51,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 21657558,
+ "executorDeserializeCpuTime" : 4010338,
+ "executorDeserializeTime" : 4,
+ "executorRunTime" : 34,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1048,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 0,
- "writeTime" : 183718,
- "recordsWritten" : 0
+ "bytesWritten" : 147,
+ "recordsWritten" : 3,
+ "writeTime" : 319101
}
- },
- "executorLogs" : {
- "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stdout?start=-4096"
- },
- "schedulerDelay" : 16,
- "gettingResultTime" : 0
+ }
},
- "9" : {
- "taskId" : 9,
- "index" : 4,
- "attempt" : 1,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:14.973GMT",
- "duration" : 96,
+ "13" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 0,
+ "duration" : 76,
"executorId" : "2",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
"host" : "apiros-3.gce.test.com",
+ "index" : 9,
+ "launchTime" : "2018-01-18T18:33:15.200GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 26,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 13,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 5,
- "executorDeserializeCpuTime" : 4793905,
- "executorRunTime" : 48,
- "executorCpuTime" : 25678331,
- "resultSize" : 1091,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 20585619,
+ "executorDeserializeCpuTime" : 5860574,
+ "executorDeserializeTime" : 25,
+ "executorRunTime" : 25,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1048,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 147,
- "writeTime" : 366050,
- "recordsWritten" : 3
+ "recordsWritten" : 3,
+ "writeTime" : 369513
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 42,
- "gettingResultTime" : 0
+ }
},
- "13" : {
- "taskId" : 13,
- "index" : 9,
+ "2" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 1774,
+ "executorId" : "3",
+ "executorLogs" : {
+ "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000004/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000004/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-2.gce.test.com",
+ "index" : 2,
+ "launchTime" : "2018-01-18T18:33:12.832GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:15.200GMT",
- "duration" : 76,
- "executorId" : "2",
- "host" : "apiros-3.gce.test.com",
+ "schedulerDelay" : 74,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 2,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 25,
- "executorDeserializeCpuTime" : 5860574,
- "executorRunTime" : 25,
- "executorCpuTime" : 20585619,
- "resultSize" : 1048,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 278399617,
+ "executorDeserializeCpuTime" : 263386625,
+ "executorDeserializeTime" : 1206,
+ "executorRunTime" : 493,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 78,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1134,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 147,
- "writeTime" : 369513,
- "recordsWritten" : 3
+ "bytesWritten" : 144,
+ "recordsWritten" : 3,
+ "writeTime" : 3322956
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 26,
- "gettingResultTime" : 0
+ }
},
- "2" : {
- "taskId" : 2,
- "index" : 2,
+ "3" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 2027,
+ "executorId" : "2",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-3.gce.test.com",
+ "index" : 3,
+ "launchTime" : "2018-01-18T18:33:12.833GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:12.832GMT",
- "duration" : 1774,
- "executorId" : "3",
- "host" : "apiros-2.gce.test.com",
+ "schedulerDelay" : 63,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 3,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 1206,
- "executorDeserializeCpuTime" : 263386625,
- "executorRunTime" : 493,
- "executorCpuTime" : 278399617,
- "resultSize" : 1134,
- "jvmGcTime" : 78,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 349920830,
+ "executorDeserializeCpuTime" : 365807898,
+ "executorDeserializeTime" : 1282,
+ "executorRunTime" : 681,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 102,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1134,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 144,
- "writeTime" : 3322956,
- "recordsWritten" : 3
+ "bytesWritten" : 147,
+ "recordsWritten" : 3,
+ "writeTime" : 3587839
}
- },
- "executorLogs" : {
- "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000004/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000004/attilapiros/stdout?start=-4096"
- },
- "schedulerDelay" : 74,
- "gettingResultTime" : 0
+ }
},
- "12" : {
- "taskId" : 12,
- "index" : 8,
+ "4" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 1522,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "executorId" : "4",
+ "executorLogs" : {
+ "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-2.gce.test.com",
+ "index" : 4,
+ "launchTime" : "2018-01-18T18:33:12.833GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:15.165GMT",
- "duration" : 60,
- "executorId" : "1",
- "host" : "apiros-3.gce.test.com",
- "status" : "SUCCESS",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 338,
"speculative" : false,
- "accumulatorUpdates" : [ ],
+ "status" : "FAILED",
+ "taskId" : 4,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 4,
- "executorDeserializeCpuTime" : 4010338,
- "executorRunTime" : 34,
- "executorCpuTime" : 21657558,
- "resultSize" : 1048,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
+ "executorRunTime" : 1184,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 82,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 147,
- "writeTime" : 319101,
- "recordsWritten" : 3
+ "bytesWritten" : 0,
+ "recordsWritten" : 0,
+ "writeTime" : 16858066
}
- },
+ }
+ },
+ "5" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 0,
+ "duration" : 73,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "executorId" : "5",
"executorLogs" : {
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000007/attilapiros/stdout?start=-4096"
},
- "schedulerDelay" : 22,
- "gettingResultTime" : 0
- },
- "7" : {
- "taskId" : 7,
+ "gettingResultTime" : 0,
+ "host" : "apiros-2.gce.test.com",
"index" : 5,
- "attempt" : 1,
+ "launchTime" : "2018-01-18T18:33:14.320GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:14.859GMT",
- "duration" : 115,
- "executorId" : "2",
- "host" : "apiros-3.gce.test.com",
- "status" : "SUCCESS",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 46,
"speculative" : false,
- "accumulatorUpdates" : [ ],
+ "status" : "FAILED",
+ "taskId" : 5,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 11,
- "executorDeserializeCpuTime" : 10894331,
- "executorRunTime" : 84,
- "executorCpuTime" : 28283110,
- "resultSize" : 1048,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
+ "executorRunTime" : 27,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 147,
- "writeTime" : 377601,
- "recordsWritten" : 3
+ "bytesWritten" : 0,
+ "recordsWritten" : 0,
+ "writeTime" : 191901
}
- },
- "executorLogs" : {
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096"
- },
- "schedulerDelay" : 20,
- "gettingResultTime" : 0
+ }
},
- "3" : {
- "taskId" : 3,
- "index" : 3,
+ "6" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
+ "duration" : 67,
+ "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "executorId" : "4",
+ "executorLogs" : {
+ "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
+ "host" : "apiros-2.gce.test.com",
+ "index" : 6,
+ "launchTime" : "2018-01-18T18:33:14.323GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:12.833GMT",
- "duration" : 2027,
- "executorId" : "2",
- "host" : "apiros-3.gce.test.com",
- "status" : "SUCCESS",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 16,
"speculative" : false,
- "accumulatorUpdates" : [ ],
+ "status" : "FAILED",
+ "taskId" : 6,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 1282,
- "executorDeserializeCpuTime" : 365807898,
- "executorRunTime" : 681,
- "executorCpuTime" : 349920830,
- "resultSize" : 1134,
- "jvmGcTime" : 102,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 0,
+ "executorRunTime" : 51,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 0,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 147,
- "writeTime" : 3587839,
- "recordsWritten" : 3
+ "bytesWritten" : 0,
+ "recordsWritten" : 0,
+ "writeTime" : 183718
}
- },
+ }
+ },
+ "7" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 1,
+ "duration" : 115,
+ "executorId" : "2",
"executorLogs" : {
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096",
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096"
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096"
},
- "schedulerDelay" : 63,
- "gettingResultTime" : 0
- },
- "11" : {
- "taskId" : 11,
- "index" : 7,
- "attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:15.072GMT",
- "duration" : 93,
- "executorId" : "1",
+ "gettingResultTime" : 0,
"host" : "apiros-3.gce.test.com",
+ "index" : 5,
+ "launchTime" : "2018-01-18T18:33:14.859GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 20,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 7,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 4,
- "executorDeserializeCpuTime" : 4239884,
- "executorRunTime" : 77,
- "executorCpuTime" : 21689428,
- "resultSize" : 1048,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 28283110,
+ "executorDeserializeCpuTime" : 10894331,
+ "executorDeserializeTime" : 11,
+ "executorRunTime" : 84,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1048,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 147,
- "writeTime" : 323898,
- "recordsWritten" : 3
+ "recordsWritten" : 3,
+ "writeTime" : 377601
}
- },
- "executorLogs" : {
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
- },
- "schedulerDelay" : 12,
- "gettingResultTime" : 0
+ }
},
"8" : {
- "taskId" : 8,
- "index" : 6,
+ "accumulatorUpdates" : [ ],
"attempt" : 1,
- "partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:14.879GMT",
"duration" : 194,
"executorId" : "1",
+ "executorLogs" : {
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ },
+ "gettingResultTime" : 0,
"host" : "apiros-3.gce.test.com",
+ "index" : 6,
+ "launchTime" : "2018-01-18T18:33:14.879GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 84,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 8,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 56,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 27304550,
"executorDeserializeCpuTime" : 12246145,
+ "executorDeserializeTime" : 56,
"executorRunTime" : 54,
- "executorCpuTime" : 27304550,
- "resultSize" : 1048,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 1048,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 147,
- "writeTime" : 311940,
- "recordsWritten" : 3
+ "recordsWritten" : 3,
+ "writeTime" : 311940
}
- },
+ }
+ },
+ "9" : {
+ "accumulatorUpdates" : [ ],
+ "attempt" : 1,
+ "duration" : 96,
+ "executorId" : "2",
"executorLogs" : {
- "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000002/attilapiros/stdout?start=-4096"
+ "stderr" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stderr?start=-4096",
+ "stdout" : "http://apiros-3.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000003/attilapiros/stdout?start=-4096"
},
- "schedulerDelay" : 84,
- "gettingResultTime" : 0
- },
- "4" : {
- "taskId" : 4,
+ "gettingResultTime" : 0,
+ "host" : "apiros-3.gce.test.com",
"index" : 4,
- "attempt" : 0,
+ "launchTime" : "2018-01-18T18:33:14.973GMT",
"partitionId" : -1,
- "launchTime" : "2018-01-18T18:33:12.833GMT",
- "duration" : 1522,
- "executorId" : "4",
- "host" : "apiros-2.gce.test.com",
- "status" : "FAILED",
- "taskLocality" : "PROCESS_LOCAL",
+ "schedulerDelay" : 42,
"speculative" : false,
- "accumulatorUpdates" : [ ],
- "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:28)\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$2.apply(<console>:27)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)\n\tat org.apache. [...]
+ "status" : "SUCCESS",
+ "taskId" : 9,
+ "taskLocality" : "PROCESS_LOCAL",
"taskMetrics" : {
- "executorDeserializeTime" : 0,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 1184,
- "executorCpuTime" : 0,
- "resultSize" : 0,
- "jvmGcTime" : 82,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 25678331,
+ "executorDeserializeCpuTime" : 4793905,
+ "executorDeserializeTime" : 5,
+ "executorRunTime" : 48,
"inputMetrics" : {
"bytesRead" : 0,
"recordsRead" : 0
},
+ "jvmGcTime" : 0,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1091,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 0,
- "writeTime" : 16858066,
- "recordsWritten" : 0
+ "bytesWritten" : 147,
+ "recordsWritten" : 3,
+ "writeTime" : 366050
}
- },
- "executorLogs" : {
- "stderr" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stderr?start=-4096",
- "stdout" : "http://apiros-2.gce.test.com:8042/node/containerlogs/container_1516285256255_0012_01_000005/attilapiros/stdout?start=-4096"
- },
- "schedulerDelay" : 338,
- "gettingResultTime" : 0
- }
- },
- "executorSummary" : {
- "4" : {
- "taskTime" : 1589,
- "failedTasks" : 2,
- "succeededTasks" : 0,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 0,
- "shuffleWriteRecords" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : true,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : true
- },
- "5" : {
- "taskTime" : 1579,
- "failedTasks" : 2,
- "succeededTasks" : 0,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 0,
- "shuffleWriteRecords" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : true,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : true
- },
- "1" : {
- "taskTime" : 2411,
- "failedTasks" : 0,
- "succeededTasks" : 4,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 585,
- "shuffleWriteRecords" : 12,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : false,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : false
- },
- "2" : {
- "taskTime" : 2446,
- "failedTasks" : 0,
- "succeededTasks" : 5,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 732,
- "shuffleWriteRecords" : 15,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : false,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : false
- },
- "3" : {
- "taskTime" : 1774,
- "failedTasks" : 0,
- "succeededTasks" : 1,
- "killedTasks" : 0,
- "inputBytes" : 0,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 144,
- "shuffleWriteRecords" : 3,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : true,
- "peakMemoryMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isExcludedForStage" : true
+ }
}
- },
- "killedTasksSummary" : { },
- "resourceProfileId" : 0,
- "peakExecutorMetrics" : {
- "JVMHeapMemory" : 0,
- "JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
- "OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
- "OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
- "OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
- "ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
- "ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
+ }
}
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json
index a860682ca2e2..5f0b7fbe2742 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json
@@ -1,54 +1,54 @@
[ {
- "id" : "driver",
- "hostPort" : "localhost:57971",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 0,
- "maxTasks" : 0,
"activeTasks" : 0,
- "failedTasks" : 1,
+ "addTime" : "2015-02-03T16:43:00.906GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
"completedTasks" : 31,
- "totalTasks" : 32,
- "totalDuration" : 9011,
- "totalGCTime" : 0,
- "totalInputBytes" : 28000288,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 13180,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 1,
+ "hostPort" : "localhost:57971",
+ "id" : "driver",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 278302556,
- "addTime" : "2015-02-03T16:43:00.906GMT",
- "executorLogs" : { },
- "blacklistedInStages" : [ ],
+ "maxTasks" : 0,
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
+ "rddBlocks" : 0,
"resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
+ "resources" : { },
+ "totalCores" : 0,
+ "totalDuration" : 9011,
+ "totalGCTime" : 0,
+ "totalInputBytes" : 28000288,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 13180,
+ "totalTasks" : 32
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_list_with_executor_metrics_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_list_with_executor_metrics_json_expectation.json
index 79f20de25691..3881e0c882bb 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_list_with_executor_metrics_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_list_with_executor_metrics_json_expectation.json
@@ -1,276 +1,276 @@
[ {
- "id" : "driver",
- "hostPort" : "test-1.vpc.company.com:44768",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 0,
- "maxTasks" : 0,
"activeTasks" : 0,
- "failedTasks" : 0,
+ "addTime" : "2019-04-08T20:39:49.747GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
"completedTasks" : 0,
- "totalTasks" : 0,
- "totalDuration" : 62168,
- "totalGCTime" : 0,
- "totalInputBytes" : 0,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 0,
+ "hostPort" : "test-1.vpc.company.com:44768",
+ "id" : "driver",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 956615884,
- "addTime" : "2019-04-08T20:39:49.747GMT",
- "executorLogs" : { },
+ "maxTasks" : 0,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 956615884,
- "totalOffHeapStorageMemory" : 0
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 157182,
"JVMHeapMemory" : 209883992,
"JVMOffHeapMemory" : 91926448,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 3,
+ "MajorGCTime" : 144,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 7,
+ "MinorGCTime" : 55,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 19560,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 19560,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 157182,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 8296026112,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 19560,
+ "OnHeapUnifiedMemory" : 19560,
"ProcessTreeJVMRSSMemory" : 526491648,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 8296026112,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 7,
- "MinorGCTime" : 55,
- "MajorGCCount" : 3,
- "MajorGCTime" : 144,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "3",
- "hostPort" : "test-3.vpc.company.com:37641",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 1,
- "maxTasks" : 1,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 2,
- "totalTasks" : 2,
- "totalDuration" : 31015,
- "totalGCTime" : 151,
- "totalInputBytes" : 2500050000,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 0,
+ "totalDuration" : 62168,
+ "totalGCTime" : 0,
+ "totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 3820172083,
+ "totalTasks" : 0
+}, {
+ "activeTasks" : 0,
"addTime" : "2019-04-08T20:39:55.362GMT",
+ "attributes" : {
+ "CLUSTER_ID" : "",
+ "CONTAINER_ID" : "container_1553914137147_0018_01_000004",
+ "HTTP_SCHEME" : "http://",
+ "LOG_FILES" : "stderr,stdout",
+ "NM_HOST" : "test-3.vpc.company.com",
+ "NM_HTTP_ADDRESS" : "test-3.vpc.company.com:8042",
+ "NM_HTTP_PORT" : "8042",
+ "NM_PORT" : "8041",
+ "USER" : "systest"
+ },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 2,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
- "stdout" : "http://test-3.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000004/systest/stdout?start=-4096",
- "stderr" : "http://test-3.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000004/systest/stderr?start=-4096"
+ "stderr" : "http://test-3.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000004/systest/stderr?start=-4096",
+ "stdout" : "http://test-3.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000004/systest/stdout?start=-4096"
},
+ "failedTasks" : 0,
+ "hostPort" : "test-3.vpc.company.com:37641",
+ "id" : "3",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 3820172083,
+ "maxTasks" : 1,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 3820172083,
- "totalOffHeapStorageMemory" : 0
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 10865,
"JVMHeapMemory" : 759479480,
"JVMOffHeapMemory" : 61343616,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 2,
+ "MajorGCTime" : 63,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 24,
+ "MinorGCTime" : 145,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 12537,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 12537,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 10865,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 9691553792,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 12537,
+ "OnHeapUnifiedMemory" : 12537,
"ProcessTreeJVMRSSMemory" : 964222976,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 9691553792,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 24,
- "MinorGCTime" : 145,
- "MajorGCCount" : 2,
- "MajorGCTime" : 63,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 1,
+ "totalDuration" : 31015,
+ "totalGCTime" : 151,
+ "totalInputBytes" : 2500050000,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 2
+}, {
+ "activeTasks" : 0,
+ "addTime" : "2019-04-08T20:39:55.291GMT",
"attributes" : {
- "NM_HTTP_ADDRESS" : "test-3.vpc.company.com:8042",
- "USER" : "systest",
+ "CLUSTER_ID" : "",
+ "CONTAINER_ID" : "container_1553914137147_0018_01_000003",
+ "HTTP_SCHEME" : "http://",
"LOG_FILES" : "stderr,stdout",
+ "NM_HOST" : "test-4.vpc.company.com",
+ "NM_HTTP_ADDRESS" : "test-4.vpc.company.com:8042",
"NM_HTTP_PORT" : "8042",
- "CLUSTER_ID" : "",
"NM_PORT" : "8041",
- "HTTP_SCHEME" : "http://",
- "NM_HOST" : "test-3.vpc.company.com",
- "CONTAINER_ID" : "container_1553914137147_0018_01_000004"
+ "USER" : "systest"
},
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "2",
- "hostPort" : "test-4.vpc.company.com:33179",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 1,
"diskUsed" : 0,
- "totalCores" : 1,
- "maxTasks" : 1,
- "activeTasks" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : {
+ "stderr" : "http://test-4.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000003/systest/stderr?start=-4096",
+ "stdout" : "http://test-4.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000003/systest/stdout?start=-4096"
+ },
"failedTasks" : 0,
- "completedTasks" : 1,
- "totalTasks" : 1,
- "totalDuration" : 30302,
- "totalGCTime" : 121,
- "totalInputBytes" : 2500050000,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
+ "hostPort" : "test-4.vpc.company.com:33179",
+ "id" : "2",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 3820172083,
- "addTime" : "2019-04-08T20:39:55.291GMT",
- "executorLogs" : {
- "stdout" : "http://test-4.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000003/systest/stdout?start=-4096",
- "stderr" : "http://test-4.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000003/systest/stderr?start=-4096"
- },
+ "maxTasks" : 1,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 3820172083,
- "totalOffHeapStorageMemory" : 0
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 12781,
"JVMHeapMemory" : 462135680,
"JVMOffHeapMemory" : 62717520,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 2,
+ "MajorGCTime" : 75,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 15,
+ "MinorGCTime" : 106,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 7023,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 7023,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 12781,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 9684373504,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 7023,
+ "OnHeapUnifiedMemory" : 7023,
"ProcessTreeJVMRSSMemory" : 703639552,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 9684373504,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 15,
- "MinorGCTime" : 106,
- "MajorGCCount" : 2,
- "MajorGCTime" : 75,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 1,
+ "totalDuration" : 30302,
+ "totalGCTime" : 121,
+ "totalInputBytes" : 2500050000,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 1
+}, {
+ "activeTasks" : 0,
+ "addTime" : "2019-04-08T20:39:54.596GMT",
"attributes" : {
- "NM_HTTP_ADDRESS" : "test-4.vpc.company.com:8042",
- "USER" : "systest",
+ "CLUSTER_ID" : "",
+ "CONTAINER_ID" : "container_1553914137147_0018_01_000002",
+ "HTTP_SCHEME" : "http://",
"LOG_FILES" : "stderr,stdout",
+ "NM_HOST" : "test-2.vpc.company.com",
+ "NM_HTTP_ADDRESS" : "test-2.vpc.company.com:8042",
"NM_HTTP_PORT" : "8042",
- "CLUSTER_ID" : "",
"NM_PORT" : "8041",
- "HTTP_SCHEME" : "http://",
- "NM_HOST" : "test-4.vpc.company.com",
- "CONTAINER_ID" : "container_1553914137147_0018_01_000003"
+ "USER" : "systest"
},
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "1",
- "hostPort" : "test-2.vpc.company.com:43764",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 1,
- "maxTasks" : 1,
- "activeTasks" : 0,
- "failedTasks" : 0,
+ "blacklistedInStages" : [ ],
"completedTasks" : 3,
- "totalTasks" : 3,
- "totalDuration" : 44900,
- "totalGCTime" : 147,
- "totalInputBytes" : 5000100000,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 3820172083,
- "addTime" : "2019-04-08T20:39:54.596GMT",
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://test-2.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000002/systest/stderr?start=-4096",
"stdout" : "http://test-2.vpc.company.com:8042/node/containerlogs/container_1553914137147_0018_01_000002/systest/stdout?start=-4096"
},
+ "failedTasks" : 0,
+ "hostPort" : "test-2.vpc.company.com:43764",
+ "id" : "1",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 3820172083,
+ "maxTasks" : 1,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 3820172083,
- "totalOffHeapStorageMemory" : 0
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 10742,
"JVMHeapMemory" : 984736560,
"JVMOffHeapMemory" : 60829472,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 2,
+ "MajorGCTime" : 60,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 27,
+ "MinorGCTime" : 140,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 19560,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 19560,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 10742,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 9678606336,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 19560,
+ "OnHeapUnifiedMemory" : 19560,
"ProcessTreeJVMRSSMemory" : 1210867712,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 9678606336,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 27,
- "MinorGCTime" : 140,
- "MajorGCCount" : 2,
- "MajorGCTime" : 60,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
- },
- "attributes" : {
- "NM_HTTP_ADDRESS" : "test-2.vpc.company.com:8042",
- "USER" : "systest",
- "LOG_FILES" : "stderr,stdout",
- "NM_HTTP_PORT" : "8042",
- "CLUSTER_ID" : "",
- "NM_PORT" : "8041",
- "HTTP_SCHEME" : "http://",
- "NM_HOST" : "test-2.vpc.company.com",
- "CONTAINER_ID" : "container_1553914137147_0018_01_000002"
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "resources" : { },
+ "rddBlocks" : 0,
"resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
+ "resources" : { },
+ "totalCores" : 1,
+ "totalDuration" : 44900,
+ "totalGCTime" : 147,
+ "totalInputBytes" : 5000100000,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 3
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json
index 19a4432ab17b..fc63185b4c4a 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json
@@ -1,283 +1,283 @@
[ {
- "id" : "driver",
- "hostPort" : "172.22.0.167:51475",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 0,
- "maxTasks" : 0,
"activeTasks" : 0,
- "failedTasks" : 0,
+ "addTime" : "2016-11-16T22:33:31.477GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
"completedTasks" : 0,
- "totalTasks" : 0,
- "totalDuration" : 10671,
- "totalGCTime" : 0,
- "totalInputBytes" : 0,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51475",
+ "id" : "driver",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 908381388,
- "addTime" : "2016-11-16T22:33:31.477GMT",
- "executorLogs" : { },
+ "maxTasks" : 0,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "3",
- "hostPort" : "172.22.0.167:51485",
- "isActive" : true,
- "rddBlocks" : 0,
"memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 12,
- "totalTasks" : 12,
- "totalDuration" : 2453,
- "totalGCTime" : 72,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 0,
+ "totalDuration" : 10671,
+ "totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 0
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.320GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 12,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51466/logPage/?appId=app-20161116163331-0000&executorId=3&logType=stderr",
"stdout" : "http://172.22.0.167:51466/logPage/?appId=app-20161116163331-0000&executorId=3&logType=stdout"
},
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51485",
+ "id" : "3",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "2",
- "hostPort" : "172.22.0.167:51487",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 4,
- "completedTasks" : 0,
- "totalTasks" : 4,
- "totalDuration" : 2537,
- "totalGCTime" : 88,
+ "totalDuration" : 2453,
+ "totalGCTime" : 72,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 12
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.393GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
- "stdout" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stdout",
- "stderr" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stderr"
+ "stderr" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stderr",
+ "stdout" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stdout"
},
+ "failedTasks" : 4,
+ "hostPort" : "172.22.0.167:51487",
+ "id" : "2",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "1",
- "hostPort" : "172.22.0.167:51490",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 4,
- "totalTasks" : 4,
- "totalDuration" : 3152,
- "totalGCTime" : 68,
+ "totalDuration" : 2537,
+ "totalGCTime" : 88,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 4
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.443GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 4,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51467/logPage/?appId=app-20161116163331-0000&executorId=1&logType=stderr",
"stdout" : "http://172.22.0.167:51467/logPage/?appId=app-20161116163331-0000&executorId=1&logType=stdout"
},
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51490",
+ "id" : "1",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "0",
- "hostPort" : "172.22.0.167:51491",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 4,
- "completedTasks" : 0,
- "totalTasks" : 4,
- "totalDuration" : 2551,
- "totalGCTime" : 116,
+ "totalDuration" : 3152,
+ "totalGCTime" : 68,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 4
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.462GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51465/logPage/?appId=app-20161116163331-0000&executorId=0&logType=stderr",
"stdout" : "http://172.22.0.167:51465/logPage/?appId=app-20161116163331-0000&executorId=0&logType=stdout"
},
+ "failedTasks" : 4,
+ "hostPort" : "172.22.0.167:51491",
+ "id" : "0",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
+ "rddBlocks" : 0,
"resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
+ "resources" : { },
+ "totalCores" : 4,
+ "totalDuration" : 2551,
+ "totalGCTime" : 116,
+ "totalInputBytes" : 0,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 4
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_expectation.json
index 19a4432ab17b..fc63185b4c4a 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_expectation.json
@@ -1,283 +1,283 @@
[ {
- "id" : "driver",
- "hostPort" : "172.22.0.167:51475",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 0,
- "maxTasks" : 0,
"activeTasks" : 0,
- "failedTasks" : 0,
+ "addTime" : "2016-11-16T22:33:31.477GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
"completedTasks" : 0,
- "totalTasks" : 0,
- "totalDuration" : 10671,
- "totalGCTime" : 0,
- "totalInputBytes" : 0,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51475",
+ "id" : "driver",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 908381388,
- "addTime" : "2016-11-16T22:33:31.477GMT",
- "executorLogs" : { },
+ "maxTasks" : 0,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "3",
- "hostPort" : "172.22.0.167:51485",
- "isActive" : true,
- "rddBlocks" : 0,
"memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 12,
- "totalTasks" : 12,
- "totalDuration" : 2453,
- "totalGCTime" : 72,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 0,
+ "totalDuration" : 10671,
+ "totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 0
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.320GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 12,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51466/logPage/?appId=app-20161116163331-0000&executorId=3&logType=stderr",
"stdout" : "http://172.22.0.167:51466/logPage/?appId=app-20161116163331-0000&executorId=3&logType=stdout"
},
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51485",
+ "id" : "3",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "2",
- "hostPort" : "172.22.0.167:51487",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 4,
- "completedTasks" : 0,
- "totalTasks" : 4,
- "totalDuration" : 2537,
- "totalGCTime" : 88,
+ "totalDuration" : 2453,
+ "totalGCTime" : 72,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 12
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.393GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
- "stdout" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stdout",
- "stderr" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stderr"
+ "stderr" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stderr",
+ "stdout" : "http://172.22.0.167:51469/logPage/?appId=app-20161116163331-0000&executorId=2&logType=stdout"
},
+ "failedTasks" : 4,
+ "hostPort" : "172.22.0.167:51487",
+ "id" : "2",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "1",
- "hostPort" : "172.22.0.167:51490",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 4,
- "totalTasks" : 4,
- "totalDuration" : 3152,
- "totalGCTime" : 68,
+ "totalDuration" : 2537,
+ "totalGCTime" : 88,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 4
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.443GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 4,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51467/logPage/?appId=app-20161116163331-0000&executorId=1&logType=stderr",
"stdout" : "http://172.22.0.167:51467/logPage/?appId=app-20161116163331-0000&executorId=1&logType=stdout"
},
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.167:51490",
+ "id" : "1",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
-}, {
- "id" : "0",
- "hostPort" : "172.22.0.167:51491",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 4,
- "completedTasks" : 0,
- "totalTasks" : 4,
- "totalDuration" : 2551,
- "totalGCTime" : 116,
+ "totalDuration" : 3152,
+ "totalGCTime" : 68,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : true,
- "maxMemory" : 908381388,
+ "totalTasks" : 4
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-16T22:33:35.462GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.167:51465/logPage/?appId=app-20161116163331-0000&executorId=0&logType=stderr",
"stdout" : "http://172.22.0.167:51465/logPage/?appId=app-20161116163331-0000&executorId=0&logType=stdout"
},
+ "failedTasks" : 4,
+ "hostPort" : "172.22.0.167:51491",
+ "id" : "0",
+ "isActive" : true,
+ "isBlacklisted" : true,
+ "isExcluded" : true,
+ "maxMemory" : 908381388,
+ "maxTasks" : 4,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 524288000,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 524288000
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
+ "rddBlocks" : 0,
"resourceProfileId" : 0,
- "isExcluded" : true,
- "excludedInStages" : [ ]
+ "resources" : { },
+ "totalCores" : 4,
+ "totalDuration" : 2551,
+ "totalGCTime" : 116,
+ "totalInputBytes" : 0,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 4
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_unexcluding_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_unexcluding_expectation.json
index feeb4eb19a13..f178457df79e 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_unexcluding_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_node_excludeOnFailure_unexcluding_expectation.json
@@ -1,253 +1,253 @@
[ {
- "id" : "driver",
+ "activeTasks" : 0,
+ "addTime" : "2016-11-15T23:20:38.836GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 0,
"hostPort" : "172.22.0.111:64527",
+ "id" : "driver",
"isActive" : true,
- "rddBlocks" : 0,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 0,
"memoryUsed" : 0,
- "diskUsed" : 0,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 0,
- "maxTasks" : 0,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 0,
- "totalTasks" : 0,
"totalDuration" : 101795,
"totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
- "addTime" : "2016-11-15T23:20:38.836GMT",
- "executorLogs" : { },
- "blacklistedInStages" : [ ],
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
+ "totalTasks" : 0
}, {
- "id" : "3",
- "hostPort" : "172.22.0.111:64543",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 4,
- "maxTasks" : 4,
"activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 4,
- "totalTasks" : 4,
- "totalDuration" : 3457,
- "totalGCTime" : 72,
- "totalInputBytes" : 0,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
"addTime" : "2016-11-15T23:20:42.711GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 4,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.111:64521/logPage/?appId=app-20161115172038-0000&executorId=3&logType=stderr",
"stdout" : "http://172.22.0.111:64521/logPage/?appId=app-20161115172038-0000&executorId=3&logType=stdout"
},
- "blacklistedInStages" : [ ],
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.111:64543",
+ "id" : "3",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 4,
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "2",
- "hostPort" : "172.22.0.111:64539",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 6,
- "completedTasks" : 0,
- "totalTasks" : 6,
- "totalDuration" : 2792,
- "totalGCTime" : 128,
+ "totalDuration" : 3457,
+ "totalGCTime" : 72,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
+ "totalTasks" : 4
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-15T23:20:42.589GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.111:64519/logPage/?appId=app-20161115172038-0000&executorId=2&logType=stderr",
"stdout" : "http://172.22.0.111:64519/logPage/?appId=app-20161115172038-0000&executorId=2&logType=stdout"
},
- "blacklistedInStages" : [ ],
+ "failedTasks" : 6,
+ "hostPort" : "172.22.0.111:64539",
+ "id" : "2",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 4,
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "1",
- "hostPort" : "172.22.0.111:64541",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 12,
- "totalTasks" : 12,
- "totalDuration" : 2613,
- "totalGCTime" : 84,
+ "totalDuration" : 2792,
+ "totalGCTime" : 128,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
+ "totalTasks" : 6
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-15T23:20:42.629GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 12,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://172.22.0.111:64518/logPage/?appId=app-20161115172038-0000&executorId=1&logType=stderr",
"stdout" : "http://172.22.0.111:64518/logPage/?appId=app-20161115172038-0000&executorId=1&logType=stdout"
},
- "blacklistedInStages" : [ ],
+ "failedTasks" : 0,
+ "hostPort" : "172.22.0.111:64541",
+ "id" : "1",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 4,
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "0",
- "hostPort" : "172.22.0.111:64540",
- "isActive" : true,
"rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
"totalCores" : 4,
- "maxTasks" : 4,
- "activeTasks" : 0,
- "failedTasks" : 4,
- "completedTasks" : 0,
- "totalTasks" : 4,
- "totalDuration" : 2741,
- "totalGCTime" : 120,
+ "totalDuration" : 2613,
+ "totalGCTime" : 84,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
+ "totalTasks" : 12
+}, {
+ "activeTasks" : 0,
"addTime" : "2016-11-15T23:20:42.593GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
- "stdout" : "http://172.22.0.111:64517/logPage/?appId=app-20161115172038-0000&executorId=0&logType=stdout",
- "stderr" : "http://172.22.0.111:64517/logPage/?appId=app-20161115172038-0000&executorId=0&logType=stderr"
+ "stderr" : "http://172.22.0.111:64517/logPage/?appId=app-20161115172038-0000&executorId=0&logType=stderr",
+ "stdout" : "http://172.22.0.111:64517/logPage/?appId=app-20161115172038-0000&executorId=0&logType=stdout"
},
- "blacklistedInStages" : [ ],
+ "failedTasks" : 4,
+ "hostPort" : "172.22.0.111:64540",
+ "id" : "0",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 4,
+ "memoryUsed" : 0,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "attributes" : { },
- "resources" : { },
+ "rddBlocks" : 0,
"resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
+ "resources" : { },
+ "totalCores" : 4,
+ "totalDuration" : 2741,
+ "totalGCTime" : 120,
+ "totalInputBytes" : 0,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 4
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/executor_resource_information_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_resource_information_expectation.json
index 49209521f064..ebfa88965157 100644
--- a/core/src/test/resources/HistoryServerExpectations/executor_resource_information_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/executor_resource_information_expectation.json
@@ -1,139 +1,139 @@
[ {
- "id" : "driver",
- "hostPort" : "10.28.9.112:37319",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 0,
- "maxTasks" : 0,
"activeTasks" : 0,
- "failedTasks" : 0,
+ "addTime" : "2019-07-02T21:02:25.595GMT",
+ "attributes" : { },
+ "blacklistedInStages" : [ ],
"completedTasks" : 0,
- "totalTasks" : 0,
- "totalDuration" : 18794,
- "totalGCTime" : 0,
- "totalInputBytes" : 0,
- "totalShuffleRead" : 0,
- "totalShuffleWrite" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : { },
+ "failedTasks" : 0,
+ "hostPort" : "10.28.9.112:37319",
+ "id" : "driver",
+ "isActive" : true,
"isBlacklisted" : false,
+ "isExcluded" : false,
"maxMemory" : 384093388,
- "addTime" : "2019-07-02T21:02:25.595GMT",
- "executorLogs" : { },
+ "maxTasks" : 0,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 0
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
- "blacklistedInStages" : [ ],
- "attributes" : { },
- "resources" : { },
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "2",
- "hostPort" : "tomg-test:46005",
- "isActive" : true,
- "rddBlocks" : 0,
"memoryUsed" : 0,
- "diskUsed" : 0,
- "totalCores" : 1,
- "maxTasks" : 1,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 0,
- "totalTasks" : 0,
- "totalDuration" : 0,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
+ "resources" : { },
+ "totalCores" : 0,
+ "totalDuration" : 18794,
"totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
+ "totalTasks" : 0
+}, {
+ "activeTasks" : 0,
"addTime" : "2019-07-02T21:02:29.256GMT",
- "executorLogs" : {
- "stdout" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000003/tgraves/stdout?start=-4096",
- "stderr" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000003/tgraves/stderr?start=-4096"
- },
- "memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
- "totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 0
- },
- "blacklistedInStages" : [ ],
"attributes" : {
- "NM_HTTP_ADDRESS" : "tomg-test:8042",
- "USER" : "tgraves",
- "LOG_FILES" : "stderr,stdout",
- "NM_HTTP_PORT" : "8042",
"CLUSTER_ID" : "",
- "NM_PORT" : "43125",
+ "CONTAINER_ID" : "container_1555004656427_0144_01_000003",
"HTTP_SCHEME" : "http://",
+ "LOG_FILES" : "stderr,stdout",
"NM_HOST" : "tomg-test",
- "CONTAINER_ID" : "container_1555004656427_0144_01_000003"
+ "NM_HTTP_ADDRESS" : "tomg-test:8042",
+ "NM_HTTP_PORT" : "8042",
+ "NM_PORT" : "43125",
+ "USER" : "tgraves"
+ },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
+ "executorLogs" : {
+ "stderr" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000003/tgraves/stderr?start=-4096",
+ "stdout" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000003/tgraves/stdout?start=-4096"
},
+ "failedTasks" : 0,
+ "hostPort" : "tomg-test:46005",
+ "id" : "2",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 1,
+ "memoryMetrics" : {
+ "totalOffHeapStorageMemory" : 0,
+ "totalOnHeapStorageMemory" : 384093388,
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
+ },
+ "memoryUsed" : 0,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
"resources" : {
"gpu" : {
- "name" : "gpu",
- "addresses" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12" ]
+ "addresses" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12" ],
+ "name" : "gpu"
}
},
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
-}, {
- "id" : "1",
- "hostPort" : "tomg-test:44873",
- "isActive" : true,
- "rddBlocks" : 0,
- "memoryUsed" : 0,
- "diskUsed" : 0,
"totalCores" : 1,
- "maxTasks" : 1,
- "activeTasks" : 0,
- "failedTasks" : 0,
- "completedTasks" : 0,
- "totalTasks" : 0,
"totalDuration" : 0,
"totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
- "isBlacklisted" : false,
- "maxMemory" : 384093388,
+ "totalTasks" : 0
+}, {
+ "activeTasks" : 0,
"addTime" : "2019-07-02T21:02:28.551GMT",
+ "attributes" : {
+ "CLUSTER_ID" : "",
+ "CONTAINER_ID" : "container_1555004656427_0144_01_000002",
+ "HTTP_SCHEME" : "http://",
+ "LOG_FILES" : "stderr,stdout",
+ "NM_HOST" : "tomg-test",
+ "NM_HTTP_ADDRESS" : "tomg-test:8042",
+ "NM_HTTP_PORT" : "8042",
+ "NM_PORT" : "43125",
+ "USER" : "tgraves"
+ },
+ "blacklistedInStages" : [ ],
+ "completedTasks" : 0,
+ "diskUsed" : 0,
+ "excludedInStages" : [ ],
"executorLogs" : {
"stderr" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000002/tgraves/stderr?start=-4096",
"stdout" : "http://tomg-test:8042/node/containerlogs/container_1555004656427_0144_01_000002/tgraves/stdout?start=-4096"
},
+ "failedTasks" : 0,
+ "hostPort" : "tomg-test:44873",
+ "id" : "1",
+ "isActive" : true,
+ "isBlacklisted" : false,
+ "isExcluded" : false,
+ "maxMemory" : 384093388,
+ "maxTasks" : 1,
"memoryMetrics" : {
- "usedOnHeapStorageMemory" : 0,
- "usedOffHeapStorageMemory" : 0,
+ "totalOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 384093388,
- "totalOffHeapStorageMemory" : 0
- },
- "blacklistedInStages" : [ ],
- "attributes" : {
- "NM_HTTP_ADDRESS" : "tomg-test:8042",
- "USER" : "tgraves",
- "LOG_FILES" : "stderr,stdout",
- "NM_HTTP_PORT" : "8042",
- "CLUSTER_ID" : "",
- "NM_PORT" : "43125",
- "HTTP_SCHEME" : "http://",
- "NM_HOST" : "tomg-test",
- "CONTAINER_ID" : "container_1555004656427_0144_01_000002"
+ "usedOffHeapStorageMemory" : 0,
+ "usedOnHeapStorageMemory" : 0
},
+ "memoryUsed" : 0,
+ "rddBlocks" : 0,
+ "resourceProfileId" : 0,
"resources" : {
"gpu" : {
- "name" : "gpu",
- "addresses" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12" ]
+ "addresses" : [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12" ],
+ "name" : "gpu"
}
},
- "resourceProfileId" : 0,
- "isExcluded" : false,
- "excludedInStages" : [ ]
+ "totalCores" : 1,
+ "totalDuration" : 0,
+ "totalGCTime" : 0,
+ "totalInputBytes" : 0,
+ "totalShuffleRead" : 0,
+ "totalShuffleWrite" : 0,
+ "totalTasks" : 0
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/failed_stage_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/failed_stage_list_json_expectation.json
index dc1bcd6a3962..fc5e0942edca 100644
--- a/core/src/test/resources/HistoryServerExpectations/failed_stage_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/failed_stage_list_json_expectation.json
@@ -1,84 +1,84 @@
[ {
- "status" : "FAILED",
- "stageId" : 2,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 7,
- "numFailedTasks" : 1,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 7,
- "submissionTime" : "2015-02-03T16:43:06.296GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:06.296GMT",
"completionTime" : "2015-02-03T16:43:06.347GMT",
- "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost task 3.0 in stage 2.0 (TID 19, localhost): java.lang.RuntimeException: got a 3, failing\n\tat $line11.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:18)\n\tat $line11.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:17)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\n\tat org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1311)\n\tat org. [...]
- "executorDeserializeTime" : 10,
+ "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$line11.$read$$iwC$$iwC$$iwC.<init>(<console>:25)\n$line11.$read$$iwC$$iwC.<init>(<console>:27)\n$line11.$read$$iwC.<init>(<console>:29)\n$line11.$read.<init>(<console>:31)\n$line11.$read$.<init>(<console>:35)\n$line11.$read$.<clinit>(<console>)\n$line11.$eval$.<init>(<console>:7)\n$line11.$eval$.<clinit>(<console>)\n$line11.$eval.$print(<console>)\nsun.reflect.NativeMeth [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 10,
"executorRunTime" : 278,
- "executorCpuTime" : 0,
- "resultSize" : 6034,
- "jvmGcTime" : 0,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost task 3.0 in stage 2.0 (TID 19, localhost): java.lang.RuntimeException: got a 3, failing\n\tat $line11.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:18)\n\tat $line11.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:17)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\n\tat org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1311)\n\tat org. [...]
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:06.296GMT",
"inputBytes" : 0,
"inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 0,
+ "killedTasksSummary" : { },
+ "memoryBytesSpilled" : 0,
+ "name" : "count at <console>:20",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 7,
+ "numCompletedIndices" : 7,
+ "numFailedTasks" : 1,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 64,
- "shuffleFetchWaitTime" : 1,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
- "shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
- "shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
- "shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
- "shuffleMergedLocalChunksFetched" : 0,
- "shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
- "shuffleMergedRemoteReqsDuration" : 0,
- "shuffleWriteBytes" : 0,
- "shuffleWriteTime" : 0,
- "shuffleWriteRecords" : 0,
- "name" : "count at <console>:20",
- "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$line11.$read$$iwC$$iwC$$iwC.<init>(<console>:25)\n$line11.$read$$iwC$$iwC.<init>(<console>:27)\n$line11.$read$$iwC.<init>(<console>:29)\n$line11.$read.<init>(<console>:31)\n$line11.$read$.<init>(<console>:35)\n$line11.$read$.<clinit>(<console>)\n$line11.$eval$.<init>(<console>:7)\n$line11.$eval$.<clinit>(<console>)\n$line11.$eval.$print(<console>)\nsun.reflect.NativeMeth [...]
- "schedulingPool" : "default",
- "rddIds" : [ 3, 2 ],
- "accumulatorUpdates" : [ ],
- "killedTasksSummary" : { },
- "resourceProfileId" : 0,
+ "peakExecutionMemory" : 0,
"peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
+ "rddIds" : [ 3, 2 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 0,
+ "resultSize" : 6034,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
+ "shuffleFetchWaitTime" : 1,
+ "shuffleLocalBlocksFetched" : 64,
+ "shuffleLocalBytesRead" : 0,
+ "shuffleMergedFetchFallbackCount" : 0,
+ "shuffleMergedLocalBlocksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
+ "shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
+ "shuffleMergedRemoteBytesRead" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
+ "shuffleWriteBytes" : 0,
+ "shuffleWriteRecords" : 0,
+ "shuffleWriteTime" : 0,
+ "stageId" : 2,
+ "status" : "FAILED",
+ "submissionTime" : "2015-02-03T16:43:06.296GMT"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_1__expectation.json b/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_1__expectation.json
index b7271d89e027..1d54a3da4b0d 100644
--- a/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_1__expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_1__expectation.json
@@ -1,19 +1,19 @@
[ {
"jobId" : 0,
- "name" : "foreach at <console>:15",
- "stageIds" : [ 0 ],
"jobTags" : [ ],
- "status" : "SUCCEEDED",
- "numTasks" : 8,
+ "killedTasksSummary" : { },
+ "name" : "foreach at <console>:15",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 8,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 8,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 0,
"numFailedTasks" : 0,
"numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 0,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 8,
+ "stageIds" : [ 0 ],
+ "status" : "SUCCEEDED"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_2__expectation.json b/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_2__expectation.json
index b7271d89e027..1d54a3da4b0d 100644
--- a/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_2__expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/job_list_from_multi_attempt_app_json_2__expectation.json
@@ -1,19 +1,19 @@
[ {
"jobId" : 0,
- "name" : "foreach at <console>:15",
- "stageIds" : [ 0 ],
"jobTags" : [ ],
- "status" : "SUCCEEDED",
- "numTasks" : 8,
+ "killedTasksSummary" : { },
+ "name" : "foreach at <console>:15",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 8,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 8,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 0,
"numFailedTasks" : 0,
"numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 0,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 8,
+ "stageIds" : [ 0 ],
+ "status" : "SUCCEEDED"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/job_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/job_list_json_expectation.json
index bb26bc47eac4..b45eb6b8696e 100644
--- a/core/src/test/resources/HistoryServerExpectations/job_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/job_list_json_expectation.json
@@ -1,55 +1,55 @@
[ {
"jobId" : 2,
- "name" : "count at <console>:17",
- "stageIds" : [ 3 ],
"jobTags" : [ ],
- "status" : "SUCCEEDED",
- "numTasks" : 8,
+ "killedTasksSummary" : { },
+ "name" : "count at <console>:17",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 8,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 8,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 0,
"numFailedTasks" : 0,
"numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 0,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 8,
+ "stageIds" : [ 3 ],
+ "status" : "SUCCEEDED"
}, {
"jobId" : 1,
- "name" : "count at <console>:20",
- "stageIds" : [ 1, 2 ],
"jobTags" : [ ],
- "status" : "FAILED",
- "numTasks" : 16,
+ "killedTasksSummary" : { },
+ "name" : "count at <console>:20",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 15,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 15,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 1,
"numFailedTasks" : 1,
"numKilledTasks" : 0,
- "numCompletedIndices" : 15,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 1,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 16,
+ "stageIds" : [ 1, 2 ],
+ "status" : "FAILED"
}, {
"jobId" : 0,
- "name" : "count at <console>:15",
- "stageIds" : [ 0 ],
"jobTags" : [ ],
- "status" : "SUCCEEDED",
- "numTasks" : 8,
+ "killedTasksSummary" : { },
+ "name" : "count at <console>:15",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 8,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 8,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 0,
"numFailedTasks" : 0,
"numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 0,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 8,
+ "stageIds" : [ 0 ],
+ "status" : "SUCCEEDED"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
index 31e4866b0a5b..7dbab594055a 100644
--- a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
@@ -1,46 +1,46 @@
[ {
- "id" : "local-1642039451826",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2022-01-13T02:04:10.519GMT",
- "endTime" : "2022-01-13T02:05:36.564GMT",
- "lastUpdated" : "",
- "duration" : 86045,
- "sparkUser" : "lijunqing",
- "completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 86045,
+ "endTime" : "2022-01-13T02:05:36.564GMT",
"endTimeEpoch" : 1642039536564,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "lijunqing",
+ "startTime" : "2022-01-13T02:04:10.519GMT",
"startTimeEpoch" : 1642039450519
- } ]
+ } ],
+ "id" : "local-1642039451826",
+ "name" : "Spark shell"
}, {
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "lastUpdated" : "",
- "duration" : 274875,
- "sparkUser" : "john",
- "completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "completed" : true,
+ "duration" : 274875,
+ "endTime" : "2021-08-10T23:29:30.208GMT",
"endTimeEpoch" : 1628638170208,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "john",
+ "startTime" : "2021-08-10T23:24:55.333GMT",
"startTimeEpoch" : 1628637895333
- } ]
+ } ],
+ "id" : "application_1628109047826_1317105",
+ "name" : "Spark shell"
}, {
- "id" : "app-20200706201101-0003",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-07-07T03:11:00.235GMT",
- "endTime" : "2020-07-07T03:17:04.231GMT",
- "lastUpdated" : "",
- "duration" : 363996,
- "sparkUser" : "terryk",
- "completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 363996,
+ "endTime" : "2020-07-07T03:17:04.231GMT",
"endTimeEpoch" : 1594091824231,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "terryk",
+ "startTime" : "2020-07-07T03:11:00.235GMT",
"startTimeEpoch" : 1594091460235
- } ]
+ } ],
+ "id" : "app-20200706201101-0003",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/maxDate2_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/maxDate2_app_list_json_expectation.json
index a760efe2ade3..ce928dca9838 100644
--- a/core/src/test/resources/HistoryServerExpectations/maxDate2_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/maxDate2_app_list_json_expectation.json
@@ -1,16 +1,16 @@
[ {
- "id" : "local-1422981759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:38.277GMT",
- "endTime" : "2015-02-03T16:42:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-03T16:42:46.912GMT",
"endTimeEpoch" : 1422981766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:38.277GMT",
"startTimeEpoch" : 1422981758277
- } ]
+ } ],
+ "id" : "local-1422981759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/maxDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/maxDate_app_list_json_expectation.json
index 797d69df7d4d..a747bc66844a 100644
--- a/core/src/test/resources/HistoryServerExpectations/maxDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/maxDate_app_list_json_expectation.json
@@ -1,31 +1,31 @@
[ {
- "id" : "local-1422981780767",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:59.720GMT",
- "endTime" : "2015-02-03T16:43:08.731GMT",
- "lastUpdated" : "",
- "duration" : 9011,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 9011,
+ "endTime" : "2015-02-03T16:43:08.731GMT",
"endTimeEpoch" : 1422981788731,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:59.720GMT",
"startTimeEpoch" : 1422981779720
- } ]
+ } ],
+ "id" : "local-1422981780767",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:38.277GMT",
- "endTime" : "2015-02-03T16:42:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-03T16:42:46.912GMT",
"endTimeEpoch" : 1422981766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:38.277GMT",
"startTimeEpoch" : 1422981758277
- } ]
+ } ],
+ "id" : "local-1422981759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/maxEndDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/maxEndDate_app_list_json_expectation.json
index 5f706c37f92b..5d65fff3ebad 100644
--- a/core/src/test/resources/HistoryServerExpectations/maxEndDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/maxEndDate_app_list_json_expectation.json
@@ -1,102 +1,102 @@
[ {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
}, {
- "id" : "local-1425081759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-28T00:02:38.277GMT",
- "endTime" : "2015-02-28T00:02:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-28T00:02:46.912GMT",
"endTimeEpoch" : 1425081766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-28T00:02:38.277GMT",
"startTimeEpoch" : 1425081758277
- } ]
+ } ],
+ "id" : "local-1425081759269",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981780767",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:59.720GMT",
- "endTime" : "2015-02-03T16:43:08.731GMT",
- "lastUpdated" : "",
- "duration" : 9011,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 9011,
+ "endTime" : "2015-02-03T16:43:08.731GMT",
"endTimeEpoch" : 1422981788731,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:59.720GMT",
"startTimeEpoch" : 1422981779720
- } ]
+ } ],
+ "id" : "local-1422981780767",
+ "name" : "Spark shell"
}, {
- "id" : "local-1422981759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:38.277GMT",
- "endTime" : "2015-02-03T16:42:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-03T16:42:46.912GMT",
"endTimeEpoch" : 1422981766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:38.277GMT",
"startTimeEpoch" : 1422981758277
- } ]
+ } ],
+ "id" : "local-1422981759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/minDate_and_maxEndDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/minDate_and_maxEndDate_app_list_json_expectation.json
index 48a5e62d42d4..8e09a800fae2 100644
--- a/core/src/test/resources/HistoryServerExpectations/minDate_and_maxEndDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/minDate_and_maxEndDate_app_list_json_expectation.json
@@ -1,57 +1,57 @@
[ {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
index ec5fc41d4cad..6df175062b5a 100644
--- a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
@@ -1,237 +1,237 @@
[ {
- "id" : "local-1642039451826",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2022-01-13T02:04:10.519GMT",
- "endTime" : "2022-01-13T02:05:36.564GMT",
- "lastUpdated" : "",
- "duration" : 86045,
- "sparkUser" : "lijunqing",
- "completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 86045,
+ "endTime" : "2022-01-13T02:05:36.564GMT",
"endTimeEpoch" : 1642039536564,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "lijunqing",
+ "startTime" : "2022-01-13T02:04:10.519GMT",
"startTimeEpoch" : 1642039450519
- } ]
+ } ],
+ "id" : "local-1642039451826",
+ "name" : "Spark shell"
}, {
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "lastUpdated" : "",
- "duration" : 274875,
- "sparkUser" : "john",
- "completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "completed" : true,
+ "duration" : 274875,
+ "endTime" : "2021-08-10T23:29:30.208GMT",
"endTimeEpoch" : 1628638170208,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "john",
+ "startTime" : "2021-08-10T23:24:55.333GMT",
"startTimeEpoch" : 1628637895333
- } ]
+ } ],
+ "id" : "application_1628109047826_1317105",
+ "name" : "Spark shell"
}, {
- "id" : "app-20200706201101-0003",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-07-07T03:11:00.235GMT",
- "endTime" : "2020-07-07T03:17:04.231GMT",
- "lastUpdated" : "",
- "duration" : 363996,
- "sparkUser" : "terryk",
- "completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 363996,
+ "endTime" : "2020-07-07T03:17:04.231GMT",
"endTimeEpoch" : 1594091824231,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "terryk",
+ "startTime" : "2020-07-07T03:11:00.235GMT",
"startTimeEpoch" : 1594091460235
- } ]
+ } ],
+ "id" : "app-20200706201101-0003",
+ "name" : "Spark shell"
}, {
- "id" : "application_1578436911597_0052",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-01-11T17:44:22.851GMT",
- "endTime" : "2020-01-11T17:46:42.615GMT",
- "lastUpdated" : "",
- "duration" : 139764,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 139764,
+ "endTime" : "2020-01-11T17:46:42.615GMT",
"endTimeEpoch" : 1578764802615,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2020-01-11T17:44:22.851GMT",
"startTimeEpoch" : 1578764662851
- } ]
+ } ],
+ "id" : "application_1578436911597_0052",
+ "name" : "Spark shell"
}, {
- "id" : "application_1555004656427_0144",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2019-07-02T21:02:17.180GMT",
- "endTime" : "2019-07-02T21:02:35.974GMT",
- "lastUpdated" : "",
- "duration" : 18794,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 18794,
+ "endTime" : "2019-07-02T21:02:35.974GMT",
"endTimeEpoch" : 1562101355974,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2019-07-02T21:02:17.180GMT",
"startTimeEpoch" : 1562101337180
- } ]
+ } ],
+ "id" : "application_1555004656427_0144",
+ "name" : "Spark shell"
}, {
- "id" : "application_1553914137147_0018",
- "name" : "LargeBlocks",
"attempts" : [ {
- "startTime" : "2019-04-08T20:39:44.286GMT",
- "endTime" : "2019-04-08T20:40:46.454GMT",
- "lastUpdated" : "",
- "duration" : 62168,
- "sparkUser" : "systest",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 62168,
+ "endTime" : "2019-04-08T20:40:46.454GMT",
"endTimeEpoch" : 1554756046454,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "systest",
+ "startTime" : "2019-04-08T20:39:44.286GMT",
"startTimeEpoch" : 1554755984286
- } ]
+ } ],
+ "id" : "application_1553914137147_0018",
+ "name" : "LargeBlocks"
}, {
- "id" : "application_1516285256255_0012",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-18T18:30:35.119GMT",
- "endTime" : "2018-01-18T18:38:27.938GMT",
- "lastUpdated" : "",
- "duration" : 472819,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 472819,
+ "endTime" : "2018-01-18T18:38:27.938GMT",
"endTimeEpoch" : 1516300707938,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-18T18:30:35.119GMT",
"startTimeEpoch" : 1516300235119
- } ]
+ } ],
+ "id" : "application_1516285256255_0012",
+ "name" : "Spark shell"
}, {
- "id" : "app-20180109111548-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-09T10:15:42.372GMT",
- "endTime" : "2018-01-09T10:24:37.606GMT",
- "lastUpdated" : "",
- "duration" : 535234,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 535234,
+ "endTime" : "2018-01-09T10:24:37.606GMT",
"endTimeEpoch" : 1515493477606,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-09T10:15:42.372GMT",
"startTimeEpoch" : 1515492942372
- } ]
+ } ],
+ "id" : "app-20180109111548-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161116163331-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-16T22:33:29.916GMT",
- "endTime" : "2016-11-16T22:33:40.587GMT",
- "lastUpdated" : "",
- "duration" : 10671,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10671,
+ "endTime" : "2016-11-16T22:33:40.587GMT",
"endTimeEpoch" : 1479335620587,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-16T22:33:29.916GMT",
"startTimeEpoch" : 1479335609916
- } ]
+ } ],
+ "id" : "app-20161116163331-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161115172038-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-15T23:20:37.079GMT",
- "endTime" : "2016-11-15T23:22:18.874GMT",
- "lastUpdated" : "",
- "duration" : 101795,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 101795,
+ "endTime" : "2016-11-15T23:22:18.874GMT",
"endTimeEpoch" : 1479252138874,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-15T23:20:37.079GMT",
"startTimeEpoch" : 1479252037079
- } ]
+ } ],
+ "id" : "app-20161115172038-0000",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381534",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:11.398GMT",
- "lastUpdated" : "",
- "duration" : 10505,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "1.4.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10505,
+ "endTime" : "2015-05-06T13:03:11.398GMT",
"endTimeEpoch" : 1430917391398,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
- } ]
+ } ],
+ "id" : "local-1430917381534",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
}, {
- "id" : "local-1425081759269",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-28T00:02:38.277GMT",
- "endTime" : "2015-02-28T00:02:46.912GMT",
- "lastUpdated" : "",
- "duration" : 8635,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 8635,
+ "endTime" : "2015-02-28T00:02:46.912GMT",
"endTimeEpoch" : 1425081766912,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-28T00:02:38.277GMT",
"startTimeEpoch" : 1425081758277
- } ]
+ } ],
+ "id" : "local-1425081759269",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/minEndDate_and_maxEndDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/minEndDate_and_maxEndDate_app_list_json_expectation.json
index 48a5e62d42d4..8e09a800fae2 100644
--- a/core/src/test/resources/HistoryServerExpectations/minEndDate_and_maxEndDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/minEndDate_and_maxEndDate_app_list_json_expectation.json
@@ -1,57 +1,57 @@
[ {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
}, {
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
index d423f5547c30..e084a5f9c29e 100644
--- a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
@@ -1,194 +1,194 @@
[ {
- "id" : "local-1642039451826",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2022-01-13T02:04:10.519GMT",
- "endTime" : "2022-01-13T02:05:36.564GMT",
- "lastUpdated" : "",
- "duration" : 86045,
- "sparkUser" : "lijunqing",
- "completed" : true,
"appSparkVersion" : "3.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 86045,
+ "endTime" : "2022-01-13T02:05:36.564GMT",
"endTimeEpoch" : 1642039536564,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "lijunqing",
+ "startTime" : "2022-01-13T02:04:10.519GMT",
"startTimeEpoch" : 1642039450519
- } ]
+ } ],
+ "id" : "local-1642039451826",
+ "name" : "Spark shell"
}, {
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "lastUpdated" : "",
- "duration" : 274875,
- "sparkUser" : "john",
- "completed" : true,
"appSparkVersion" : "3.1.1.119",
+ "completed" : true,
+ "duration" : 274875,
+ "endTime" : "2021-08-10T23:29:30.208GMT",
"endTimeEpoch" : 1628638170208,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "john",
+ "startTime" : "2021-08-10T23:24:55.333GMT",
"startTimeEpoch" : 1628637895333
- } ]
+ } ],
+ "id" : "application_1628109047826_1317105",
+ "name" : "Spark shell"
}, {
- "id" : "app-20200706201101-0003",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-07-07T03:11:00.235GMT",
- "endTime" : "2020-07-07T03:17:04.231GMT",
- "lastUpdated" : "",
- "duration" : 363996,
- "sparkUser" : "terryk",
- "completed" : true,
"appSparkVersion" : "3.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 363996,
+ "endTime" : "2020-07-07T03:17:04.231GMT",
"endTimeEpoch" : 1594091824231,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "terryk",
+ "startTime" : "2020-07-07T03:11:00.235GMT",
"startTimeEpoch" : 1594091460235
- } ]
+ } ],
+ "id" : "app-20200706201101-0003",
+ "name" : "Spark shell"
}, {
- "id" : "application_1578436911597_0052",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2020-01-11T17:44:22.851GMT",
- "endTime" : "2020-01-11T17:46:42.615GMT",
- "lastUpdated" : "",
- "duration" : 139764,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 139764,
+ "endTime" : "2020-01-11T17:46:42.615GMT",
"endTimeEpoch" : 1578764802615,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2020-01-11T17:44:22.851GMT",
"startTimeEpoch" : 1578764662851
- } ]
+ } ],
+ "id" : "application_1578436911597_0052",
+ "name" : "Spark shell"
}, {
- "id" : "application_1555004656427_0144",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2019-07-02T21:02:17.180GMT",
- "endTime" : "2019-07-02T21:02:35.974GMT",
- "lastUpdated" : "",
- "duration" : 18794,
- "sparkUser" : "tgraves",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 18794,
+ "endTime" : "2019-07-02T21:02:35.974GMT",
"endTimeEpoch" : 1562101355974,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "tgraves",
+ "startTime" : "2019-07-02T21:02:17.180GMT",
"startTimeEpoch" : 1562101337180
- } ]
+ } ],
+ "id" : "application_1555004656427_0144",
+ "name" : "Spark shell"
}, {
- "id" : "application_1553914137147_0018",
- "name" : "LargeBlocks",
"attempts" : [ {
- "startTime" : "2019-04-08T20:39:44.286GMT",
- "endTime" : "2019-04-08T20:40:46.454GMT",
- "lastUpdated" : "",
- "duration" : 62168,
- "sparkUser" : "systest",
- "completed" : true,
"appSparkVersion" : "3.0.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 62168,
+ "endTime" : "2019-04-08T20:40:46.454GMT",
"endTimeEpoch" : 1554756046454,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "systest",
+ "startTime" : "2019-04-08T20:39:44.286GMT",
"startTimeEpoch" : 1554755984286
- } ]
+ } ],
+ "id" : "application_1553914137147_0018",
+ "name" : "LargeBlocks"
}, {
- "id" : "application_1516285256255_0012",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-18T18:30:35.119GMT",
- "endTime" : "2018-01-18T18:38:27.938GMT",
- "lastUpdated" : "",
- "duration" : 472819,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 472819,
+ "endTime" : "2018-01-18T18:38:27.938GMT",
"endTimeEpoch" : 1516300707938,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-18T18:30:35.119GMT",
"startTimeEpoch" : 1516300235119
- } ]
+ } ],
+ "id" : "application_1516285256255_0012",
+ "name" : "Spark shell"
}, {
- "id" : "app-20180109111548-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2018-01-09T10:15:42.372GMT",
- "endTime" : "2018-01-09T10:24:37.606GMT",
- "lastUpdated" : "",
- "duration" : 535234,
- "sparkUser" : "attilapiros",
- "completed" : true,
"appSparkVersion" : "2.3.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 535234,
+ "endTime" : "2018-01-09T10:24:37.606GMT",
"endTimeEpoch" : 1515493477606,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "attilapiros",
+ "startTime" : "2018-01-09T10:15:42.372GMT",
"startTimeEpoch" : 1515492942372
- } ]
+ } ],
+ "id" : "app-20180109111548-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161116163331-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-16T22:33:29.916GMT",
- "endTime" : "2016-11-16T22:33:40.587GMT",
- "lastUpdated" : "",
- "duration" : 10671,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10671,
+ "endTime" : "2016-11-16T22:33:40.587GMT",
"endTimeEpoch" : 1479335620587,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-16T22:33:29.916GMT",
"startTimeEpoch" : 1479335609916
- } ]
+ } ],
+ "id" : "app-20161116163331-0000",
+ "name" : "Spark shell"
}, {
- "id" : "app-20161115172038-0000",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2016-11-15T23:20:37.079GMT",
- "endTime" : "2016-11-15T23:22:18.874GMT",
- "lastUpdated" : "",
- "duration" : 101795,
- "sparkUser" : "jose",
- "completed" : true,
"appSparkVersion" : "2.1.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 101795,
+ "endTime" : "2016-11-15T23:22:18.874GMT",
"endTimeEpoch" : 1479252138874,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "jose",
+ "startTime" : "2016-11-15T23:20:37.079GMT",
"startTimeEpoch" : 1479252037079
- } ]
+ } ],
+ "id" : "app-20161115172038-0000",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381534",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:11.398GMT",
- "lastUpdated" : "",
- "duration" : 10505,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "1.4.0-SNAPSHOT",
+ "completed" : true,
+ "duration" : 10505,
+ "endTime" : "2015-05-06T13:03:11.398GMT",
"endTimeEpoch" : 1430917391398,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
- } ]
+ } ],
+ "id" : "local-1430917381534",
+ "name" : "Spark shell"
}, {
- "id" : "local-1430917381535",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "2",
- "startTime" : "2015-05-06T13:03:00.893GMT",
- "endTime" : "2015-05-06T13:03:00.950GMT",
- "lastUpdated" : "",
- "duration" : 57,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 57,
+ "endTime" : "2015-05-06T13:03:00.950GMT",
"endTimeEpoch" : 1430917380950,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.893GMT",
"startTimeEpoch" : 1430917380893
}, {
+ "appSparkVersion" : "1.4.0-SNAPSHOT",
"attemptId" : "1",
- "startTime" : "2015-05-06T13:03:00.880GMT",
- "endTime" : "2015-05-06T13:03:00.890GMT",
- "lastUpdated" : "",
- "duration" : 10,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "1.4.0-SNAPSHOT",
+ "duration" : 10,
+ "endTime" : "2015-05-06T13:03:00.890GMT",
"endTimeEpoch" : 1430917380890,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-05-06T13:03:00.880GMT",
"startTimeEpoch" : 1430917380880
- } ]
+ } ],
+ "id" : "local-1430917381535",
+ "name" : "Spark shell"
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/miscellaneous_process_expectation.json b/core/src/test/resources/HistoryServerExpectations/miscellaneous_process_expectation.json
index 93d674a1f921..92e2c9f886ec 100644
--- a/core/src/test/resources/HistoryServerExpectations/miscellaneous_process_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/miscellaneous_process_expectation.json
@@ -1,11 +1,11 @@
[ {
- "id" : "yarn-am",
+ "addTime" : "2021-04-19T15:35:50.218GMT",
"hostPort" : "192.168.1.19:8042",
+ "id" : "yarn-am",
"isActive" : true,
- "totalCores" : 1,
- "addTime" : "2021-04-19T15:35:50.218GMT",
"processLogs" : {
- "stdout" : "http://192.168.1.19:8042/node/containerlogs/container_1555004656427_0144_01_000001/test/stdout?start=-4096",
- "stderr" : "http://192.168.1.19:8042/node/containerlogs/container_1555004656427_0144_01_000001/test/stderr?start=-4096"
- }
+ "stderr" : "http://192.168.1.19:8042/node/containerlogs/container_1555004656427_0144_01_000001/test/stderr?start=-4096",
+ "stdout" : "http://192.168.1.19:8042/node/containerlogs/container_1555004656427_0144_01_000001/test/stdout?start=-4096"
+ },
+ "totalCores" : 1
} ]
diff --git a/core/src/test/resources/HistoryServerExpectations/multiple_resource_profiles_expectation.json b/core/src/test/resources/HistoryServerExpectations/multiple_resource_profiles_expectation.json
index 41e91ea2b929..26f5c0b0f161 100644
--- a/core/src/test/resources/HistoryServerExpectations/multiple_resource_profiles_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/multiple_resource_profiles_expectation.json
@@ -1,113 +1,113 @@
{
- "runtime" : {
- "javaVersion" : "1.8.0_232 (Private Build)",
- "javaHome" : "/usr/lib/jvm/java-8-openjdk-amd64/jre",
- "scalaVersion" : "version 2.12.10"
- },
- "sparkProperties" : [ ],
+ "classpathEntries" : [ ],
"hadoopProperties" : [ ],
- "systemProperties" : [ ],
"metricsProperties" : [ ],
- "classpathEntries" : [ ],
"resourceProfiles" : [ {
- "id" : 0,
"executorResources" : {
- "memory" : {
- "resourceName" : "memory",
- "amount" : 1024,
+ "cores" : {
+ "amount" : 1,
"discoveryScript" : "",
+ "resourceName" : "cores",
"vendor" : ""
},
"gpu" : {
- "resourceName" : "gpu",
"amount" : 1,
"discoveryScript" : "/home/tgraves/getGpus",
+ "resourceName" : "gpu",
"vendor" : ""
},
- "cores" : {
- "resourceName" : "cores",
- "amount" : 1,
+ "memory" : {
+ "amount" : 1024,
"discoveryScript" : "",
+ "resourceName" : "memory",
"vendor" : ""
}
},
+ "id" : 0,
"taskResources" : {
"cpus" : {
- "resourceName" : "cpus",
- "amount" : 1.0
+ "amount" : 1.0,
+ "resourceName" : "cpus"
},
"gpu" : {
- "resourceName" : "gpu",
- "amount" : 1.0
+ "amount" : 1.0,
+ "resourceName" : "gpu"
}
}
}, {
- "id" : 1,
"executorResources" : {
"cores" : {
- "resourceName" : "cores",
"amount" : 4,
"discoveryScript" : "",
+ "resourceName" : "cores",
"vendor" : ""
},
"gpu" : {
- "resourceName" : "gpu",
"amount" : 1,
"discoveryScript" : "./getGpus",
+ "resourceName" : "gpu",
"vendor" : ""
}
},
+ "id" : 1,
"taskResources" : {
"cpus" : {
- "resourceName" : "cpus",
- "amount" : 1.0
+ "amount" : 1.0,
+ "resourceName" : "cpus"
},
"gpu" : {
- "resourceName" : "gpu",
- "amount" : 1.0
+ "amount" : 1.0,
+ "resourceName" : "gpu"
}
}
}, {
- "id" : 2,
"executorResources" : {
"cores" : {
- "resourceName" : "cores",
"amount" : 2,
"discoveryScript" : "",
+ "resourceName" : "cores",
"vendor" : ""
}
},
+ "id" : 2,
"taskResources" : {
"cpus" : {
- "resourceName" : "cpus",
- "amount" : 2.0
+ "amount" : 2.0,
+ "resourceName" : "cpus"
}
}
}, {
- "id" : 3,
"executorResources" : {
"cores" : {
- "resourceName" : "cores",
"amount" : 4,
"discoveryScript" : "",
+ "resourceName" : "cores",
"vendor" : ""
},
"gpu" : {
- "resourceName" : "gpu",
"amount" : 1,
"discoveryScript" : "./getGpus",
+ "resourceName" : "gpu",
"vendor" : ""
}
},
+ "id" : 3,
"taskResources" : {
"cpus" : {
- "resourceName" : "cpus",
- "amount" : 2.0
+ "amount" : 2.0,
+ "resourceName" : "cpus"
},
"gpu" : {
- "resourceName" : "gpu",
- "amount" : 1.0
+ "amount" : 1.0,
+ "resourceName" : "gpu"
}
}
- } ]
+ } ],
+ "runtime" : {
+ "javaHome" : "/usr/lib/jvm/java-8-openjdk-amd64/jre",
+ "javaVersion" : "1.8.0_232 (Private Build)",
+ "scalaVersion" : "version 2.12.10"
+ },
+ "sparkProperties" : [ ],
+ "systemProperties" : [ ]
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_app_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_app_json_expectation.json
index bead190da365..7e375af0ee9e 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_app_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_app_json_expectation.json
@@ -1,16 +1,16 @@
{
- "id" : "local-1422981780767",
- "name" : "Spark shell",
"attempts" : [ {
- "startTime" : "2015-02-03T16:42:59.720GMT",
- "endTime" : "2015-02-03T16:43:08.731GMT",
- "lastUpdated" : "",
- "duration" : 9011,
- "sparkUser" : "irashid",
- "completed" : true,
"appSparkVersion" : "",
+ "completed" : true,
+ "duration" : 9011,
+ "endTime" : "2015-02-03T16:43:08.731GMT",
"endTimeEpoch" : 1422981788731,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-02-03T16:42:59.720GMT",
"startTimeEpoch" : 1422981779720
- } ]
+ } ],
+ "id" : "local-1422981780767",
+ "name" : "Spark shell"
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_app_multi_attempt_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_app_multi_attempt_json_expectation.json
index 075f115790a1..0e265f11af64 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_app_multi_attempt_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_app_multi_attempt_json_expectation.json
@@ -1,29 +1,29 @@
{
- "id" : "local-1426533911241",
- "name" : "Spark shell",
"attempts" : [ {
+ "appSparkVersion" : "",
"attemptId" : "2",
- "startTime" : "2015-03-17T23:11:50.242GMT",
- "endTime" : "2015-03-17T23:12:25.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-17T23:12:25.177GMT",
"endTimeEpoch" : 1426633945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-17T23:11:50.242GMT",
"startTimeEpoch" : 1426633910242
}, {
+ "appSparkVersion" : "",
"attemptId" : "1",
- "startTime" : "2015-03-16T19:25:10.242GMT",
- "endTime" : "2015-03-16T19:25:45.177GMT",
- "lastUpdated" : "",
- "duration" : 34935,
- "sparkUser" : "irashid",
"completed" : true,
- "appSparkVersion" : "",
+ "duration" : 34935,
+ "endTime" : "2015-03-16T19:25:45.177GMT",
"endTimeEpoch" : 1426533945177,
+ "lastUpdated" : "",
"lastUpdatedEpoch" : 0,
+ "sparkUser" : "irashid",
+ "startTime" : "2015-03-16T19:25:10.242GMT",
"startTimeEpoch" : 1426533910242
- } ]
+ } ],
+ "id" : "local-1426533911241",
+ "name" : "Spark shell"
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_job_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_job_json_expectation.json
index 3bf4845ed1e0..71b8af83452f 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_job_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_job_json_expectation.json
@@ -1,19 +1,19 @@
{
"jobId" : 0,
- "name" : "count at <console>:15",
- "stageIds" : [ 0 ],
"jobTags" : [ ],
- "status" : "SUCCEEDED",
- "numTasks" : 8,
+ "killedTasksSummary" : { },
+ "name" : "count at <console>:15",
+ "numActiveStages" : 0,
"numActiveTasks" : 0,
+ "numCompletedIndices" : 8,
+ "numCompletedStages" : 1,
"numCompletedTasks" : 8,
- "numSkippedTasks" : 0,
+ "numFailedStages" : 0,
"numFailedTasks" : 0,
"numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "numActiveStages" : 0,
- "numCompletedStages" : 1,
"numSkippedStages" : 0,
- "numFailedStages" : 0,
- "killedTasksSummary" : { }
+ "numSkippedTasks" : 0,
+ "numTasks" : 8,
+ "stageIds" : [ 0 ],
+ "status" : "SUCCEEDED"
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
index 09afdf5da5b6..72cf178acad5 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
@@ -1,10 +1,10 @@
{
+ "diskUsed" : 0,
"id" : 0,
+ "memoryUsed" : 0,
"name" : "0",
- "numPartitions" : 8,
"numCachedPartitions" : 0,
- "storageLevel" : "Memory Deserialized 1x Replicated",
- "memoryUsed" : 0,
- "diskUsed" : 0,
- "partitions" : [ ]
+ "numPartitions" : 8,
+ "partitions" : [ ],
+ "storageLevel" : "Memory Deserialized 1x Replicated"
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_details_with_failed_task_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_details_with_failed_task_expectation.json
index c6dbd10f4f6a..f79816d20f25 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_details_with_failed_task_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_details_with_failed_task_expectation.json
@@ -1,129 +1,129 @@
{
- "status" : "COMPLETE",
- "stageId" : 1,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 8,
- "numFailedTasks" : 0,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "submissionTime" : "2015-02-03T16:43:05.829GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
"completionTime" : "2015-02-03T16:43:06.286GMT",
- "executorDeserializeTime" : 13,
+ "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 13,
"executorRunTime" : 3476,
- "executorCpuTime" : 0,
- "resultSize" : 15216,
- "jvmGcTime" : 152,
- "resultSerializationTime" : 9,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
- "inputBytes" : 28000128,
- "inputRecords" : 0,
- "outputBytes" : 0,
- "outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
- "shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
- "shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
- "shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
- "shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
- "shuffleMergedLocalChunksFetched" : 0,
- "shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
- "shuffleMergedRemoteReqsDuration" : 0,
- "shuffleWriteBytes" : 13180,
- "shuffleWriteTime" : 692000,
- "shuffleWriteRecords" : 0,
- "name" : "map at <console>:14",
- "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
- "schedulingPool" : "default",
- "rddIds" : [ 1, 0 ],
- "accumulatorUpdates" : [ ],
- "tasks" : { },
"executorSummary" : {
"driver" : {
- "taskTime" : 3624,
+ "diskBytesSpilled" : 0,
"failedTasks" : 0,
- "succeededTasks" : 8,
- "killedTasks" : 0,
"inputBytes" : 28000128,
"inputRecords" : 0,
+ "isBlacklistedForStage" : false,
+ "isExcludedForStage" : false,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRead" : 0,
- "shuffleReadRecords" : 0,
- "shuffleWrite" : 13180,
- "shuffleWriteRecords" : 0,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "isBlacklistedForStage" : false,
"peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "isExcludedForStage" : false
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 13180,
+ "shuffleWriteRecords" : 0,
+ "succeededTasks" : 8,
+ "taskTime" : 3624
}
},
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
+ "inputBytes" : 28000128,
+ "inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 152,
"killedTasksSummary" : { },
- "resourceProfileId" : 0,
+ "memoryBytesSpilled" : 0,
+ "name" : "map at <console>:14",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 8,
+ "numCompletedIndices" : 8,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakExecutionMemory" : 0,
"peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
- "OnHeapExecutionMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
"OffHeapExecutionMemory" : 0,
- "OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
- "OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
- "DirectPoolMemory" : 0,
- "MappedPoolMemory" : 0,
- "ProcessTreeJVMVMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
- "ProcessTreePythonVMemory" : 0,
- "ProcessTreePythonRSSMemory" : 0,
- "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
- "MinorGCCount" : 0,
- "MinorGCTime" : 0,
- "MajorGCCount" : 0,
- "MajorGCTime" : 0,
- "TotalGCTime" : 0,
- "ConcurrentGCCount" : 0,
- "ConcurrentGCTime" : 0
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
},
- "isShufflePushEnabled" : false,
- "shuffleMergersCount" : 0
+ "rddIds" : [ 1, 0 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 9,
+ "resultSize" : 15216,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
+ "shuffleFetchWaitTime" : 0,
+ "shuffleLocalBlocksFetched" : 0,
+ "shuffleLocalBytesRead" : 0,
+ "shuffleMergedFetchFallbackCount" : 0,
+ "shuffleMergedLocalBlocksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
+ "shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
+ "shuffleMergedRemoteBytesRead" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
+ "shuffleWriteBytes" : 13180,
+ "shuffleWriteRecords" : 0,
+ "shuffleWriteTime" : 692000,
+ "stageId" : 1,
+ "status" : "COMPLETE",
+ "submissionTime" : "2015-02-03T16:43:05.829GMT",
+ "tasks" : { }
}
diff --git a/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json
index 90db41b7ebe5..a4caf7da159d 100644
--- a/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json
+++ b/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json
@@ -1,634 +1,634 @@
{
- "status" : "COMPLETE",
- "stageId" : 1,
+ "accumulatorUpdates" : [ ],
"attemptId" : 0,
- "numTasks" : 8,
- "numActiveTasks" : 0,
- "numCompleteTasks" : 8,
- "numFailedTasks" : 0,
- "numKilledTasks" : 0,
- "numCompletedIndices" : 8,
- "submissionTime" : "2015-02-03T16:43:05.829GMT",
- "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
"completionTime" : "2015-02-03T16:43:06.286GMT",
- "executorDeserializeTime" : 13,
+ "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 13,
"executorRunTime" : 3476,
- "executorCpuTime" : 0,
- "resultSize" : 15216,
- "jvmGcTime" : 152,
- "resultSerializationTime" : 9,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorSummary" : {
+ "driver" : {
+ "diskBytesSpilled" : 0,
+ "failedTasks" : 0,
+ "inputBytes" : 28000128,
+ "inputRecords" : 0,
+ "isBlacklistedForStage" : false,
+ "isExcludedForStage" : false,
+ "killedTasks" : 0,
+ "memoryBytesSpilled" : 0,
+ "outputBytes" : 0,
+ "outputRecords" : 0,
+ "peakMemoryMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "shuffleRead" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleWrite" : 13180,
+ "shuffleWriteRecords" : 0,
+ "succeededTasks" : 8,
+ "taskTime" : 3624
+ }
+ },
+ "firstTaskLaunchedTime" : "2015-02-03T16:43:05.829GMT",
"inputBytes" : 28000128,
"inputRecords" : 0,
+ "isShufflePushEnabled" : false,
+ "jvmGcTime" : 152,
+ "killedTasksSummary" : { },
+ "memoryBytesSpilled" : 0,
+ "name" : "map at <console>:14",
+ "numActiveTasks" : 0,
+ "numCompleteTasks" : 8,
+ "numCompletedIndices" : 8,
+ "numFailedTasks" : 0,
+ "numKilledTasks" : 0,
+ "numTasks" : 8,
"outputBytes" : 0,
"outputRecords" : 0,
- "shuffleRemoteBlocksFetched" : 0,
- "shuffleLocalBlocksFetched" : 0,
+ "peakExecutionMemory" : 0,
+ "peakExecutorMetrics" : {
+ "ConcurrentGCCount" : 0,
+ "ConcurrentGCTime" : 0,
+ "DirectPoolMemory" : 0,
+ "JVMHeapMemory" : 0,
+ "JVMOffHeapMemory" : 0,
+ "MajorGCCount" : 0,
+ "MajorGCTime" : 0,
+ "MappedPoolMemory" : 0,
+ "MinorGCCount" : 0,
+ "MinorGCTime" : 0,
+ "OffHeapExecutionMemory" : 0,
+ "OffHeapStorageMemory" : 0,
+ "OffHeapUnifiedMemory" : 0,
+ "OnHeapExecutionMemory" : 0,
+ "OnHeapStorageMemory" : 0,
+ "OnHeapUnifiedMemory" : 0,
+ "ProcessTreeJVMRSSMemory" : 0,
+ "ProcessTreeJVMVMemory" : 0,
+ "ProcessTreeOtherRSSMemory" : 0,
+ "ProcessTreeOtherVMemory" : 0,
+ "ProcessTreePythonRSSMemory" : 0,
+ "ProcessTreePythonVMemory" : 0,
+ "TotalGCTime" : 0
+ },
+ "rddIds" : [ 1, 0 ],
+ "resourceProfileId" : 0,
+ "resultSerializationTime" : 9,
+ "resultSize" : 15216,
+ "schedulingPool" : "default",
+ "shuffleCorruptMergedBlockChunks" : 0,
"shuffleFetchWaitTime" : 0,
- "shuffleRemoteBytesRead" : 0,
- "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleLocalBlocksFetched" : 0,
"shuffleLocalBytesRead" : 0,
- "shuffleReadBytes" : 0,
- "shuffleReadRecords" : 0,
- "shuffleCorruptMergedBlockChunks" : 0,
"shuffleMergedFetchFallbackCount" : 0,
- "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedLocalBlocksFetched" : 0,
- "shuffleMergedRemoteChunksFetched" : 0,
+ "shuffleMergedLocalBytesRead" : 0,
"shuffleMergedLocalChunksFetched" : 0,
+ "shuffleMergedRemoteBlocksFetched" : 0,
"shuffleMergedRemoteBytesRead" : 0,
- "shuffleMergedLocalBytesRead" : 0,
- "shuffleRemoteReqsDuration" : 0,
+ "shuffleMergedRemoteChunksFetched" : 0,
"shuffleMergedRemoteReqsDuration" : 0,
+ "shuffleMergersCount" : 0,
+ "shuffleReadBytes" : 0,
+ "shuffleReadRecords" : 0,
+ "shuffleRemoteBlocksFetched" : 0,
+ "shuffleRemoteBytesRead" : 0,
+ "shuffleRemoteBytesReadToDisk" : 0,
+ "shuffleRemoteReqsDuration" : 0,
"shuffleWriteBytes" : 13180,
- "shuffleWriteTime" : 692000,
"shuffleWriteRecords" : 0,
- "name" : "map at <console>:14",
- "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$line10.$read$$iwC$$iwC$$iwC.<init>(<console>:19)\n$line10.$read$$iwC$$iwC.<init>(<console>:21)\n$line10.$read$$iwC.<init>(<console>:23)\n$line10.$read.<init>(<console>:25)\n$line10.$read$.<init>(<console>:29)\n$line10.$read$.<clinit>(<console>)\n$line10.$eval$.<init>(<console>:7)\n$line10.$eval$.<clinit>(<console>)\n$line10.$eval.$print(<console>)\nsun.reflect.NativeMethod [...]
- "schedulingPool" : "default",
- "rddIds" : [ 1, 0 ],
- "accumulatorUpdates" : [ ],
+ "shuffleWriteTime" : 692000,
+ "stageId" : 1,
+ "status" : "COMPLETE",
+ "submissionTime" : "2015-02-03T16:43:05.829GMT",
"tasks" : {
"10" : {
- "taskId" : 10,
- "index" : 2,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.830GMT",
"duration" : 456,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 2,
+ "launchTime" : "2015-02-03T16:43:05.830GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 19,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 10,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 2,
"executorRunTime" : 434,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 1648,
- "writeTime" : 76000,
- "recordsWritten" : 0
+ "recordsWritten" : 0,
+ "writeTime" : 76000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 19,
- "gettingResultTime" : 0
+ }
},
- "14" : {
- "taskId" : 14,
- "index" : 6,
+ "11" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.832GMT",
- "duration" : 450,
+ "duration" : 454,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 3,
+ "launchTime" : "2015-02-03T16:43:05.830GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 17,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 11,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 2,
"executorRunTime" : 434,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 1648,
- "writeTime" : 88000,
- "recordsWritten" : 0
+ "bytesWritten" : 1647,
+ "recordsWritten" : 0,
+ "writeTime" : 83000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 13,
- "gettingResultTime" : 0
+ }
},
- "9" : {
- "taskId" : 9,
- "index" : 1,
+ "12" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.830GMT",
"duration" : 454,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 4,
+ "launchTime" : "2015-02-03T16:43:05.831GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 17,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 12,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 1,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 436,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 0,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 2,
+ "executorRunTime" : 434,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 1648,
- "writeTime" : 98000,
- "recordsWritten" : 0
+ "bytesWritten" : 1645,
+ "recordsWritten" : 0,
+ "writeTime" : 101000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 17,
- "gettingResultTime" : 0
+ }
},
"13" : {
- "taskId" : 13,
- "index" : 5,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.831GMT",
"duration" : 452,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 5,
+ "launchTime" : "2015-02-03T16:43:05.831GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 14,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 13,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 2,
"executorRunTime" : 434,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 2,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 2,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
"bytesWritten" : 1648,
- "writeTime" : 73000,
- "recordsWritten" : 0
+ "recordsWritten" : 0,
+ "writeTime" : 73000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 14,
- "gettingResultTime" : 0
+ }
},
- "12" : {
- "taskId" : 12,
- "index" : 4,
+ "14" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.831GMT",
- "duration" : 454,
+ "duration" : 450,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 6,
+ "launchTime" : "2015-02-03T16:43:05.832GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 13,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 14,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 2,
"executorRunTime" : 434,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
- "diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 1645,
- "writeTime" : 101000,
- "recordsWritten" : 0
+ "bytesWritten" : 1648,
+ "recordsWritten" : 0,
+ "writeTime" : 88000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 17,
- "gettingResultTime" : 0
+ }
},
- "11" : {
- "taskId" : 11,
- "index" : 3,
+ "15" : {
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.830GMT",
- "duration" : 454,
+ "duration" : 450,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 7,
+ "launchTime" : "2015-02-03T16:43:05.833GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 13,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 15,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 2,
- "executorDeserializeCpuTime" : 0,
- "executorRunTime" : 434,
- "executorCpuTime" : 0,
- "resultSize" : 1902,
- "jvmGcTime" : 19,
- "resultSerializationTime" : 1,
- "memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
- "peakExecutionMemory" : 0,
+ "executorCpuTime" : 0,
+ "executorDeserializeCpuTime" : 0,
+ "executorDeserializeTime" : 1,
+ "executorRunTime" : 435,
"inputMetrics" : {
"bytesRead" : 3500016,
"recordsRead" : 0
},
+ "jvmGcTime" : 19,
+ "memoryBytesSpilled" : 0,
"outputMetrics" : {
"bytesWritten" : 0,
"recordsWritten" : 0
},
+ "peakExecutionMemory" : 0,
+ "resultSerializationTime" : 1,
+ "resultSize" : 1902,
"shuffleReadMetrics" : {
- "remoteBlocksFetched" : 0,
- "localBlocksFetched" : 0,
"fetchWaitTime" : 0,
- "remoteBytesRead" : 0,
- "remoteBytesReadToDisk" : 0,
+ "localBlocksFetched" : 0,
"localBytesRead" : 0,
"recordsRead" : 0,
+ "remoteBlocksFetched" : 0,
+ "remoteBytesRead" : 0,
+ "remoteBytesReadToDisk" : 0,
"remoteReqsDuration" : 0,
"shufflePushReadMetrics" : {
"corruptMergedBlockChunks" : 0,
- "mergedFetchFallbackCount" : 0,
- "remoteMergedBlocksFetched" : 0,
"localMergedBlocksFetched" : 0,
- "remoteMergedChunksFetched" : 0,
+ "localMergedBytesRead" : 0,
"localMergedChunksFetched" : 0,
+ "mergedFetchFallbackCount" : 0,
+ "remoteMergedBlocksFetched" : 0,
"remoteMergedBytesRead" : 0,
- "localMergedBytesRead" : 0,
+ "remoteMergedChunksFetched" : 0,
"remoteMergedReqsDuration" : 0
}
},
"shuffleWriteMetrics" : {
- "bytesWritten" : 1647,
- "writeTime" : 83000,
- "recordsWritten" : 0
+ "bytesWritten" : 1648,
+ "recordsWritten" : 0,
+ "writeTime" : 79000
}
- },
- "executorLogs" : { },
- "schedulerDelay" : 17,
- "gettingResultTime" : 0
+ }
},
"8" : {
- "taskId" : 8,
- "index" : 0,
+ "accumulatorUpdates" : [ ],
"attempt" : 0,
- "partitionId" : -1,
- "launchTime" : "2015-02-03T16:43:05.829GMT",
"duration" : 454,
"executorId" : "driver",
+ "executorLogs" : { },
+ "gettingResultTime" : 0,
"host" : "localhost",
+ "index" : 0,
+ "launchTime" : "2015-02-03T16:43:05.829GMT",
+ "partitionId" : -1,
+ "schedulerDelay" : 16,
+ "speculative" : false,
"status" : "SUCCESS",
+ "taskId" : 8,
"taskLocality" : "PROCESS_LOCAL",
- "speculative" : false,
- "accumulatorUpdates" : [ ],
"taskMetrics" : {
- "executorDeserializeTime" : 1,
+ "diskBytesSpilled" : 0,
+ "executorCpuTime" : 0,
"executorDeserializeCpuTime" : 0,
... 28137 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org