You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/04/19 15:12:51 UTC
[spark] branch master updated: [SPARK-27486][CORE][TEST] Enable
History server storage information test in the HistoryServerSuite
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 16bbe0f [SPARK-27486][CORE][TEST] Enable History server storage information test in the HistoryServerSuite
16bbe0f is described below
commit 16bbe0f798d2a75ba0a9f2daff709292f6971755
Author: Shahid <sh...@gmail.com>
AuthorDate: Fri Apr 19 08:12:20 2019 -0700
[SPARK-27486][CORE][TEST] Enable History server storage information test in the HistoryServerSuite
## What changes were proposed in this pull request?
We have disabled a test related to storage in the History server suite after SPARK-13845. But, after SPARK-22050, we can store the information about block updated events to eventLog, if we enable "spark.eventLog.logBlockUpdates.enabled=true".
So, we can enable the test, by adding an eventlog corresponding to the application, which has enabled the configuration, "spark.eventLog.logBlockUpdates.enabled=true"
## How was this patch tested?
Existing UTs
Closes #24390 from shahidki31/enableRddStorageTest.
Authored-by: Shahid <sh...@gmail.com>
Signed-off-by: Sean Owen <se...@databricks.com>
---
.../one_rdd_storage_json_expectation.json | 10 ++++++++++
.../org/apache/spark/deploy/history/HistoryServerSuite.scala | 8 +++++---
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
new file mode 100644
index 0000000..09afdf5
--- /dev/null
+++ b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
@@ -0,0 +1,10 @@
+{
+ "id" : 0,
+ "name" : "0",
+ "numPartitions" : 8,
+ "numCachedPartitions" : 0,
+ "storageLevel" : "Memory Deserialized 1x Replicated",
+ "memoryUsed" : 0,
+ "diskUsed" : 0,
+ "partitions" : [ ]
+}
diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index c99bcf2..5ba27a0 100644
--- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -174,9 +174,11 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
"executor node blacklisting unblacklisting" -> "applications/app-20161115172038-0000/executors",
"executor memory usage" -> "applications/app-20161116163331-0000/executors",
- "app environment" -> "applications/app-20161116163331-0000/environment"
- // Todo: enable this test when logging the even of onBlockUpdated. See: SPARK-13845
- // "one rdd storage json" -> "applications/local-1422981780767/storage/rdd/0"
+ "app environment" -> "applications/app-20161116163331-0000/environment",
+
+ // Enable "spark.eventLog.logBlockUpdates.enabled", to get the storage information
+ // in the history server.
+ "one rdd storage json" -> "applications/local-1422981780767/storage/rdd/0"
)
// run a bunch of characterization tests -- just verify the behavior is the same as what is saved
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org