You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2019/09/15 17:16:27 UTC
[spark] branch branch-2.4 updated: [SPARK-29087][CORE][STREAMING]
Use DelegatingServletContextHandler to avoid CCE
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new b41795a [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
b41795a is described below
commit b41795a29f413b36a29d0650a779accfb06142ef
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sun Sep 15 10:15:49 2019 -0700
[SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
### What changes were proposed in this pull request?
[SPARK-27122](https://github.com/apache/spark/pull/24088) fixes `ClassCastException` at `yarn` module by introducing `DelegatingServletContextHandler`. Initially, this was discovered with JDK9+, but the class path issues affected JDK8 environment, too. After [SPARK-28709](https://github.com/apache/spark/pull/25439), I also hit the similar issue at `streaming` module.
This PR aims to fix `streaming` module by adding `getContextPath` to `DelegatingServletContextHandler` and using it.
### Why are the changes needed?
Currently, when we test `streaming` module independently, it fails like the following.
```
$ build/mvn test -pl streaming
...
UISeleniumSuite:
- attaching and detaching a Streaming tab *** FAILED ***
java.lang.ClassCastException: org.sparkproject.jetty.servlet.ServletContextHandler cannot be cast to org.eclipse.jetty.servlet.ServletContextHandler
...
Tests: succeeded 337, failed 1, canceled 0, ignored 1, pending 0
*** 1 TEST FAILED ***
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the Jenkins with the modified tests. And do the following manually.
Since you can observe this when you run `streaming` module test only (instead of running all), you need to install the changed `core` module and use it.
```
$ java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.222-b10, mixed mode)
$ build/mvn install -DskipTests
$ build/mvn test -pl streaming
```
Closes #25791 from dongjoon-hyun/SPARK-29087.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 729b3180bcbaa5289cb9a5848a3cce9010e75515)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
core/src/main/scala/org/apache/spark/ui/WebUI.scala | 4 ++++
.../src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala | 4 ++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/ui/WebUI.scala b/core/src/main/scala/org/apache/spark/ui/WebUI.scala
index fa90129..da08d12 100644
--- a/core/src/main/scala/org/apache/spark/ui/WebUI.scala
+++ b/core/src/main/scala/org/apache/spark/ui/WebUI.scala
@@ -229,4 +229,8 @@ private[spark] class DelegatingServletContextHandler(handler: ServletContextHand
def filterCount(): Int = {
handler.getServletHandler.getFilters.length
}
+
+ def getContextPath(): String = {
+ handler.getContextPath
+ }
}
diff --git a/streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala b/streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala
index cee561a..ead7f92 100644
--- a/streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala
+++ b/streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala
@@ -96,7 +96,7 @@ class UISeleniumSuite
val sparkUI = ssc.sparkContext.ui.get
- sparkUI.getHandlers.count(_.getContextPath.contains("/streaming")) should be (5)
+ sparkUI.getDelegatingHandlers.count(_.getContextPath.contains("/streaming")) should be (5)
eventually(timeout(10 seconds), interval(50 milliseconds)) {
go to (sparkUI.webUrl.stripSuffix("/"))
@@ -197,7 +197,7 @@ class UISeleniumSuite
ssc.stop(false)
- sparkUI.getHandlers.count(_.getContextPath.contains("/streaming")) should be (0)
+ sparkUI.getDelegatingHandlers.count(_.getContextPath.contains("/streaming")) should be (0)
eventually(timeout(10 seconds), interval(50 milliseconds)) {
go to (sparkUI.webUrl.stripSuffix("/"))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org