You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2022/05/12 08:15:40 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #2636] Refine BatchesResourceSuite
This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new a5bb93e5d [KYUUBI #2636] Refine BatchesResourceSuite
a5bb93e5d is described below
commit a5bb93e5d66856e87a38b8c49a3ea17cbd113c7a
Author: Fu Chen <cf...@gmail.com>
AuthorDate: Thu May 12 16:15:27 2022 +0800
[KYUUBI #2636] Refine BatchesResourceSuite
### _Why are the changes needed?_
refine BatchesResourceSuite
the BatchesResourceSuite failed after upgrade spark to 3.3.0
https://github.com/apache/incubator-kyuubi/runs/6384637613?check_suite_focus=true
```
BatchesResourceSuite:
- open batch session *** FAILED ***
The code passed to eventually never returned normally. Attempted 20 times over 10.176988683 seconds. Last failure message: logs.exists(((x$2: String) => x$2.contains("/bin/spark-submit"))) was true, but logs.exists(((x$3: String) => x$3.contains(scala.StringContext.apply("spark.SparkContext: Submitted application: ", "").s(appName)))) was false. (BatchesResourceSuite.scala:94)
- get batch session list *** FAILED ***
List(Batch(d860929f-f1a8-4f0d-890d-d06d895b48d8,spark,Map(id -> 21071, name -> org.apache.spark.deploy.SparkSubmit --conf spark.master=local --conf spark.kyuubi.session.engine.spark.max.lifetime=5000 --conf spark.kyuubi.session.engine.check.interval=1000 --conf spark.yarn.tags=d860929f-f1a8-4f0d-890d-d06d895b48d8 --conf spark.app.name=spark-batch-submission --class org.apache.kyuubi.engine.spark.SparkSQLEngine --proxy-user anonymous /home/runner/work/incubator-kyuubi/incubator-kyuu [...]
```
- spark 3.3 has a different console layer pattern after migrating to log4j2.
Spark 3.3
```
22/05/12 11:19:57 INFO SparkContext: Submitted application: Spark Pi
```
Spark 3.2 and previous version
```
22/05/12 11:42:18 INFO spark.SparkContext: Submitted application: Spark Pi
```
> Actually, I can't figure out what causes this change.
- clean up batch job after each unit test.
### _How was this patch tested?_
Pass CI
Closes #2636 from cfmcgrady/fix-batch-suite.
Closes #2636
815fdf5a [Fu Chen] address comment
9878d1d0 [Fu Chen] address comment
db17e826 [Fu Chen] refine BatchesResourceSuite
Authored-by: Fu Chen <cf...@gmail.com>
Signed-off-by: ulysses-you <ul...@apache.org>
---
.../kyuubi/server/api/v1/BatchesResourceSuite.scala | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
index e2ff28ab5..a6160357f 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
@@ -33,9 +33,24 @@ import org.apache.kyuubi.config.KyuubiConf.{ENGINE_CHECK_INTERVAL, ENGINE_SPARK_
import org.apache.kyuubi.engine.spark.SparkProcessBuilder
import org.apache.kyuubi.server.http.authentication.AuthenticationHandler.AUTHORIZATION_HEADER
import org.apache.kyuubi.service.authentication.KyuubiAuthenticationFactory
-import org.apache.kyuubi.session.KyuubiSessionManager
+import org.apache.kyuubi.session.{KyuubiBatchSessionImpl, KyuubiSessionManager}
class BatchesResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
+
+ override def afterEach(): Unit = {
+ val sessionManager = fe.be.sessionManager
+ sessionManager.asInstanceOf[KyuubiSessionManager]
+ .getBatchSessionList(null, 0, Int.MaxValue)
+ .map(_.asInstanceOf[KyuubiBatchSessionImpl])
+ .foreach { session =>
+ try {
+ session.batchJobSubmissionOp.killBatchApplication()
+ } finally {
+ sessionManager.closeSession(session.handle)
+ }
+ }
+ }
+
test("open batch session") {
val sparkProcessBuilder = new SparkProcessBuilder("kyuubi", conf)
val appName = "spark-batch-submission"
@@ -104,7 +119,7 @@ class BatchesResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
// check both kyuubi log and engine log
assert(logs.exists(_.contains("/bin/spark-submit")) && logs.exists(
- _.contains(s"spark.SparkContext: Submitted application: $appName")))
+ _.contains(s"SparkContext: Submitted application: $appName")))
}
// invalid user name