You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/02/16 22:27:27 UTC
(spark) branch master updated: [SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2a7e8900ab5f [SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir
2a7e8900ab5f is described below
commit 2a7e8900ab5f3845c3ebe6f43ce9b5ccd95cec99
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Fri Feb 16 14:27:16 2024 -0800
[SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir
### What changes were proposed in this pull request?
This PR aims to fix `HistoryServerSuite`'s `incomplete apps get refreshed` test case to start with empty `storeDir`.
### Why are the changes needed?
To fix a flakiness in `LevelDBBackendHistoryServerSuite` and `RocksDBBackendHistoryServerSuite`.
```
LevelDBBackendHistoryServerSuite
...
info] - incomplete apps get refreshed *** FAILED *** (15 seconds, 397 milliseconds)
[info] The code passed to eventually never returned normally. Attempted 43 times over 10.213112356000002 seconds. Last failure message: 4 did not equal 0. (HistoryServerSuite.scala:564)
```
```
RocksDBBackendHistoryServerSuite
...
[info] - incomplete apps get refreshed *** FAILED *** (15 seconds, 339 milliseconds)
[info] The code passed to eventually never returned normally. Attempted 42 times over 10.135717083000001 seconds. Last failure message: 0 did not equal 4. (HistoryServerSuite.scala:564)
```
For the other test cases, `init` method recreates `storeDir`.
https://github.com/apache/spark/blob/5faa512bf8c7c268613220d93ae4733cdf734cd0/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala#L85-L87
Also, `init` is invoked if `stop` is used manually. Only `incomplete apps get refreshed` test case doesn't clean up.
https://github.com/apache/spark/blob/5faa512bf8c7c268613220d93ae4733cdf734cd0/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala#L611-L612
### Does this PR introduce _any_ user-facing change?
No. This is a single test-case change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #45143 from dongjoon-hyun/SPARK-47076.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../scala/org/apache/spark/deploy/history/HistoryServerSuite.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index c8ffbf68a0c5..b1738b4e742b 100644
--- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -390,6 +390,9 @@ abstract class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with
// a new conf is used with the background thread set and running at its fastest
// allowed refresh rate (1Hz)
stop()
+ // Like 'init()', we need to clear the store directory of previously stopped server.
+ Utils.deleteRecursively(storeDir)
+ assert(storeDir.mkdir())
val myConf = new SparkConf()
.set(HISTORY_LOG_DIR, logDir.getAbsolutePath)
.set(EVENT_LOG_DIR, logDir.getAbsolutePath)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org