You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/17 08:55:10 UTC

[GitHub] [spark] LuciferYang opened a new pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

LuciferYang opened a new pull request #29783:
URL: https://github.com/apache/spark/pull/29783


   ### What changes were proposed in this pull request?
   
   This pr fix failed and aborted cases in sql hive-thriftserver module in Scala 2.13, the main change of this pr as follow:
   
   - Use `s.c.Seq` instead of `Seq` in `HiveResult` because the input type maybe `mutable.ArraySeq`, but `Seq` represent `immutable.Seq` in Scala 2.13.
   
   - Reset classLoader after `HiveMetastoreLazyInitializationSuite` completed because context class loader is `NonClosableMutableURLClassLoader`  in `HiveMetastoreLazyInitializationSuite` running process, and it propagate to `HiveThriftServer2ListenerSuite` trigger following problems in Scala 2.13:
   
   ```
   HiveThriftServer2ListenerSuite:
   *** RUN ABORTED ***
     java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"
     at org.mockito.codegen.HiveThriftServer2$MockitoMock$1850222569.<clinit>(Unknown Source)
     at sun.reflect.GeneratedSerializationConstructorAccessor530.newInstance(Unknown Source)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
     at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:48)
     at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
     at org.mockito.internal.creation.instance.ObjenesisInstantiator.newInstance(ObjenesisInstantiator.java:19)
     at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:47)
     at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
     at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
     at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
     ...
   ```
   
   After this pr `HiveThriftServer2Suites` and `HiveThriftServer2ListenerSuite` was fixed and all 461 test passed
   
   ### Why are the changes needed?
   We need to support a Scala 2.13 build.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   - Scala 2.12: Pass the Jenkins or GitHub Action
   
   - Scala 2.13: All tests passed.
   
   Do the following:
   
   ```
   dev/change-scala-version.sh 2.13
   mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive-thriftserver -Pscala-2.13
   mvn test -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Pscala-2.13
   ```
   
   **Before**
   
   ```
   HiveThriftServer2ListenerSuite:
   *** RUN ABORTED ***
   ```
   
   **After**
   
   ```
   
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694509785


   **[Test build #128831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128831/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490323441



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       Thank you for investigation, @LuciferYang . It's interesting.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694511927






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490089526



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       cc @srowen I think this change is very Interesting, `HiveThriftServer2ListenerSuite ` will ABORTED without this change
   ```
   HiveThriftServer2ListenerSuite:
   *** RUN ABORTED ***
     java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"
   ```
   
   The Context Class Loader is `NonClosableMutableURLClassLoader` in `HiveMetastoreLazyInitializationSuite` running process and propagate to `HiveThriftServer2ListenerSuite`  , then 
   
   https://github.com/apache/spark/blob/92b75dc260eb43d906a425f9f9d8d63b78c48cee/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2ListenerSuite.scala#L172-L175
   
   will use `MultipleParentClassLoader(NonClosableMutableURLClassLoader and AppClassLoader)` to mock `HiveThriftServer2` instance in Scala 2.13, but mock `HiveThriftServer2` instance  use `AppClassLoader` only in Scala 2.12.
   
   Then I do some experiments:
   
   - Ignore `HiveMetastoreLazyInitializationSuite` all tests passed
   
   - Run `HiveThriftServer2ListenerSuite` only will passed
   
   - Copy test case `HiveMetastoreLazyInitializationSuite` to  `HiveThriftServer2ListenerSuite` then  run `HiveThriftServer2ListenerSuite` only, it aborted in both Scala 2.12 and Scala 2.13.
   
   I guess that `HiveMetastoreLazyInitializationSuite` and `HiveThriftServer2ListenerSuite` run in same thread in Scala 2.13 but different thread in Scala 2.12, but I don't know why.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694096704


   **[Test build #128811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128811/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694511927






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694307606






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490089526



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       This change is very Interesting, `HiveThriftServer2ListenerSuite ` will ABORTED without this change
   ```
   HiveThriftServer2ListenerSuite:
   *** RUN ABORTED ***
     java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"
     at org.mockito.codegen.HiveThriftServer2$MockitoMock$1850222569.<clinit>(Unknown Source)
     at sun.reflect.GeneratedSerializationConstructorAccessor530.newInstance(Unknown Source)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
     at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:48)
     at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
     at org.mockito.internal.creation.instance.ObjenesisInstantiator.newInstance(ObjenesisInstantiator.java:19)
     at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:47)
     at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
     at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
     at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
     ...
   
   ```
    The Context Class Loader is `NonClosableMutableURLClassLoader` in `HiveMetastoreLazyInitializationSuite` running process and the `NonClosableMutableURLClassLoader` propagate to `HiveThriftServer2ListenerSuite`  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694307606






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490432301



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       Ya. I agree with @srowen . The patch itself looks good.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694285715






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694511670


   Thank you, @LuciferYang and @srowen .
   Merged to master for Apache Spark 3.1.0 on December 2020.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694096704


   **[Test build #128811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128811/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694285715






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694097314






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694283611


   **[Test build #128811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128811/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694306949


   **[Test build #128831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128831/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694305667


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694306949


   **[Test build #128831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128831/testReport)** for PR 29783 at commit [`0d20878`](https://github.com/apache/spark/commit/0d208781b7a3132a89dca75242d89008cb4dc66a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29783:
URL: https://github.com/apache/spark/pull/29783


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490089526



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       cc @srowen I think this change is very Interesting, `HiveThriftServer2ListenerSuite ` will ABORTED without this change
   ```
   HiveThriftServer2ListenerSuite:
   *** RUN ABORTED ***
     java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"
   ```
   
   The Context Class Loader is `NonClosableMutableURLClassLoader` in `HiveMetastoreLazyInitializationSuite` running process and propagate to `HiveThriftServer2ListenerSuite`  , then 
   
   https://github.com/apache/spark/blob/92b75dc260eb43d906a425f9f9d8d63b78c48cee/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2ListenerSuite.scala#L172-L175
   
   will use `MultipleParentClassLoader(NonClosableMutableURLClassLoader and AppClassLoader)` to mock `HiveThriftServer2` instance in Scala 2.13, but it's `AppClassLoader` only in Scala 2.12.
   
   Then I do some experiments:
   
   - Ignore `HiveMetastoreLazyInitializationSuite` all tests passed
   
   - Run `HiveThriftServer2ListenerSuite` only will passed
   
   - Copy test case `HiveMetastoreLazyInitializationSuite` to  `HiveThriftServer2ListenerSuite` then  run `HiveThriftServer2ListenerSuite` only, it aborted in both Scala 2.12 and Scala 2.13.
   
   I guess that `HiveMetastoreLazyInitializationSuite` and `HiveThriftServer2ListenerSuite` run in same thread in Scala 2.13 but different thread in Scala 2.12, but I don't know why.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29783: [SPARK-32909][SQL][test-java11] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29783:
URL: https://github.com/apache/spark/pull/29783#discussion_r490336387



##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreLazyInitializationSuite.scala
##########
@@ -31,6 +31,7 @@ class HiveMetastoreLazyInitializationSuite extends SparkFunSuite {
       .config("spark.hadoop.hive.metastore.uris", "thrift://127.0.0.1:11111")
       .getOrCreate()
     val originalLevel = org.apache.log4j.Logger.getRootLogger().getLevel
+    val originalClassLoader = Thread.currentThread().getContextClassLoader

Review comment:
       Interesting. I think it's fine, as this change just improves isolation in any event.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694097314






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29783: [SPARK-32909][SQL] Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29783:
URL: https://github.com/apache/spark/pull/29783#issuecomment-694607601


   Thank @srowen and @dongjoon-hyun ~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org