You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/23 03:33:21 UTC
[GitHub] [spark] LuciferYang opened a new pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option for Java 17 test introduced by SPARK-37070
LuciferYang opened a new pull request #34991:
URL: https://github.com/apache/spark/pull/34991
### What changes were proposed in this pull request?
In order to enable the UTs in `mllib-local` and `mllib` to use `mockito` to mock `j.u.Random`, `-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED` compile option is added for Java 17 in SPARK-37070 .
This pr do the following change to remove the `-add-exports` option:
1. Introduce test dependence of mockito-inline, the new test dependence used to specify the configuration of `mockito-extensions`
2. Upgrade org.scalatestplus:mockito from 3-4 to 3-12 and upgrade mockito from 3.4.x to 3.12.x
### Why are the changes needed?
Find a way to let `mockito` can mock `j.u.Random` directly and remove `-add-exports` compile option for Java 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test with Java 17
Use maven
```
mvn clean install -pl mllib-local -am -DskipTests
mvn test -pl mllib-local
mvn clean install -pl mllib -am -DskipTests
mvn test -pl mllib
```
Use sbt
```
sbt mllib-local/test
sbt mllib/test
```
There are no test failed related to ` org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class java.util.Random.` without `-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED`, all test can be successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r775228275
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
I will restart this work in a few days because I have something else to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000061011
**[Test build #146503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146503/testReport)** for PR 34991 at commit [`36679c0`](https://github.com/apache/spark/commit/36679c0774074076fff2be31b843a2ae72a875f9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #34991:
URL: https://github.com/apache/spark/pull/34991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r774390940
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
I guess there are another libraries which use `objenesis` and this PR exposes a new version to supercede the old version of that library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [WIP][SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000061011
**[Test build #146503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146503/testReport)** for PR 34991 at commit [`36679c0`](https://github.com/apache/spark/commit/36679c0774074076fff2be31b843a2ae72a875f9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r775228275
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
@dongjoon-hyun Thank you for changing this pr to draft, I will restart this work in a few days because I have something else to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000080369
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50979/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000110579
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50979/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000110558
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50979/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34991: [WIP][SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000060765
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1001531544
merge with master and re-trigger Jenkins CI again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003301172
@dongjoon-hyun @srowen
I found that `objenesis ` as `compile` scope level comes from `kryo`.
Take `common/unsafe` module as an example, the dependency chain is as follows:
- spark-unsafe_2.12 -> `com.twitter:chill_2.12:0.10.0` -> `com.esotericsoftware:kryo-shaded:4.0.2` -> `org.objenesis:objenesis:2.5.1:compile`
- spark-unsafe_2.12 -> `org.mockito:mockito-core:3.12.4` -> `org.objenesis:objenesis:3.2:test`
Therefore, for the `common/unsafe` module, we can see from the dependency tree that `objenesis` is the `compile` scope and the version change from 2.6 to 3.2 because this pr upgrade mockito from 3.4.x to 3.12.x.
In order to test compatibility, I did the following tests
1. Manually clone a kryo repository: `git clone git@github.com:EsotericSoftware/kryo.git kryo`
2. Switch to `kryo-parent-4.0.2 ` tag: `git checkout -b kryo-parent-4.0.2 kryo-parent-4.0.2 `
3. Manually change `objenesis` version from `2.5.1` to `3.2`
```
diff --git a/pom.xml b/pom.xml
index 3d808a7..94d0aa3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -71,7 +71,7 @@
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
- <version>2.5.1</version>
+ <version>3.2</version>
</dependency>
```
4. Execute `mvn clean test` and all UTs passed.
```
Results :
Tests run: 220, Failures: 0, Errors: 0, Skipped: 1
```
So I think this pr upgrade `objenesis` is safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000023625
**[Test build #146498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146498/testReport)** for PR 34991 at commit [`13c661a`](https://github.com/apache/spark/commit/13c661a0a4096feeb70ca80405eabe7a480d41ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000038350
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000118840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146503/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000118840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146503/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34991: [WIP][SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000060765
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000023625
**[Test build #146498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146498/testReport)** for PR 34991 at commit [`13c661a`](https://github.com/apache/spark/commit/13c661a0a4096feeb70ca80405eabe7a480d41ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000110579
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50979/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r774297692
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
Let me find out why objenesis upgraded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000079371
**[Test build #146498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146498/testReport)** for PR 34991 at commit [`13c661a`](https://github.com/apache/spark/commit/13c661a0a4096feeb70ca80405eabe7a480d41ee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000115777
**[Test build #146503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146503/testReport)** for PR 34991 at commit [`36679c0`](https://github.com/apache/spark/commit/36679c0774074076fff2be31b843a2ae72a875f9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r774403274
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
Let me investigate again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34991:
URL: https://github.com/apache/spark/pull/34991#discussion_r774306663
##########
File path: dev/deps/spark-deps-hadoop-2-hive-2.3
##########
@@ -201,7 +201,7 @@ metrics-json/4.2.2//metrics-json-4.2.2.jar
metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
minlog/1.3.0//minlog-1.3.0.jar
netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
-objenesis/2.6//objenesis-2.6.jar
+objenesis/3.2//objenesis-3.2.jar
Review comment:
objenesis is the cascade dependency of mockito, it's scope is considered as `compile` in `unsafe` module:
```
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ spark-unsafe_2.12 ---
[INFO] org.apache.spark:spark-unsafe_2.12:jar:3.3.0-SNAPSHOT
[INFO] +- org.apache.spark:spark-tags_2.12:jar:3.3.0-SNAPSHOT:compile
[INFO] | \- org.scala-lang:scala-library:jar:2.12.15:compile
[INFO] +- org.apache.spark:spark-tags_2.12:test-jar:tests:3.3.0-SNAPSHOT:test
[INFO] +- com.twitter:chill_2.12:jar:0.10.0:compile
[INFO] | +- com.twitter:chill-java:jar:0.10.0:compile
[INFO] | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] | \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] +- com.google.guava:guava:jar:14.0.1:provided
[INFO] +- org.slf4j:slf4j-api:jar:1.7.30:provided
[INFO] +- org.mockito:mockito-core:jar:3.12.4:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.11.13:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.11.13:test
[INFO] | \- org.objenesis:objenesis:jar:3.2:compile
```
I haven't found the root cause yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003232569
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000054903
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000084709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146498/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1000084709
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146498/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003248767
> Although we don't know still why objenesis is upgraded here, I agree with @srowen 's decision.
Sorry, I've been a little busy recently. I'll continue to investigate this when I have time @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003248960
Never mind, @LuciferYang ~ It's okay to ignore that for now. ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003301172
@dongjoon-hyun @srowen
I found that `objenesis ` as `compile` scope dependency comes from `kryo-shaded`.
Take `common/unsafe` module as an example, the dependency chain is as follows:
- spark-unsafe_2.12 -> `com.twitter:chill_2.12:0.10.0` -> `com.esotericsoftware:kryo-shaded:4.0.2` -> `org.objenesis:objenesis:2.5.1:compile`
- spark-unsafe_2.12 -> `org.mockito:mockito-core:3.12.4` -> `org.objenesis:objenesis:3.2:test`
Therefore, for the `common/unsafe` module, we can see from the dependency tree that `objenesis` is the `compile` scope and the version change from 2.6 to 3.2 because this pr upgrade mockito from 3.4.x to 3.12.x.
In order to test compatibility, I did the following tests
1. Manually clone a kryo repository: `git clone git@github.com:EsotericSoftware/kryo.git kryo`
2. Switch to `kryo-parent-4.0.2 ` tag: `git checkout -b kryo-parent-4.0.2 kryo-parent-4.0.2 `
3. Manually change `objenesis` version from `2.5.1` to `3.2`
```
diff --git a/pom.xml b/pom.xml
index 3d808a7..94d0aa3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -71,7 +71,7 @@
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
- <version>2.5.1</version>
+ <version>3.2</version>
</dependency>
```
4. Execute `mvn clean test` and all UTs passed.
```
Results :
Tests run: 220, Failures: 0, Errors: 0, Skipped: 1
```
So I think this pr upgrade `objenesis` is safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #34991: [SPARK-37719][BUILD] Remove the `-add-exports` compilation option introduced by SPARK-37070
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #34991:
URL: https://github.com/apache/spark/pull/34991#issuecomment-1003301172
@dongjoon-hyun @srowen
I found that `objenesis ` as `compile` scope dependency comes from `kryo`.
Take `common/unsafe` module as an example, the dependency chain is as follows:
- spark-unsafe_2.12 -> `com.twitter:chill_2.12:0.10.0` -> `com.esotericsoftware:kryo-shaded:4.0.2` -> `org.objenesis:objenesis:2.5.1:compile`
- spark-unsafe_2.12 -> `org.mockito:mockito-core:3.12.4` -> `org.objenesis:objenesis:3.2:test`
Therefore, for the `common/unsafe` module, we can see from the dependency tree that `objenesis` is the `compile` scope and the version change from 2.6 to 3.2 because this pr upgrade mockito from 3.4.x to 3.12.x.
In order to test compatibility, I did the following tests
1. Manually clone a kryo repository: `git clone git@github.com:EsotericSoftware/kryo.git kryo`
2. Switch to `kryo-parent-4.0.2 ` tag: `git checkout -b kryo-parent-4.0.2 kryo-parent-4.0.2 `
3. Manually change `objenesis` version from `2.5.1` to `3.2`
```
diff --git a/pom.xml b/pom.xml
index 3d808a7..94d0aa3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -71,7 +71,7 @@
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
- <version>2.5.1</version>
+ <version>3.2</version>
</dependency>
```
4. Execute `mvn clean test` and all UTs passed.
```
Results :
Tests run: 220, Failures: 0, Errors: 0, Skipped: 1
```
So I think this pr upgrade `objenesis` is safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org