You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/09/01 07:30:48 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
LuciferYang opened a new pull request, #42766:
URL: https://github.com/apache/spark/pull/42766
### What changes were proposed in this pull request?
This pr aims to set `shadeTestJar` of `core` module to `false` to skip shade `spark-core**-tests.jar` process.
### Why are the changes needed?
Before SPARK-41244, the core module used the `maven-shade-plugin` configuration in the `parent pom.xml` for shading, where the `shadeTestJar` configuration item was set to the default `false` in the `maven-shade-plugin` in `parent pom.xml`.
In SPARK-41244 | SPARK-41244 https://github.com/apache/spark/pull/38779, the core module started using a separate `maven-shade-plugin` configuration and set the `shadeTestJar` configuration item to `true`.
And when `shadeTestJar` is true, `maven-shade-plugin` always try to find(sometime is downloading) some non-existent jars:
```
[INFO] --- maven-shade-plugin:3.5.0:shade (default) @ spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-plus/9.4.51.v20230217/jetty-plus-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-plus/9.4.51.v20230217/jetty-plus-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-plus:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-security/9.4.51.v20230217/jetty-security-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.51.v20230217/jetty-security-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-security:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-util/9.4.51.v20230217/jetty-util-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.51.v20230217/jetty-util-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-util:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-io/9.4.51.v20230217/jetty-io-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.51.v20230217/jetty-io-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-io:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-continuation/9.4.51.v20230217/jetty-continuation-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-continuation/9.4.51.v20230217/jetty-continuation-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-continuation:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.51.v20230217 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-proxy/9.4.51.v20230217/jetty-proxy-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-proxy/9.4.51.v20230217/jetty-proxy-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-proxy:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-client/9.4.51.v20230217/jetty-client-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-client/9.4.51.v20230217/jetty-client-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-client:jar:9.4.51.v20230217:compile
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.51.v20230217 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-servlets/9.4.51.v20230217/jetty-servlets-9.4.51.v20230217-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlets/9.4.51.v20230217/jetty-servlets-9.4.51.v20230217-tests.jar
[WARNING] Could not get tests for org.eclipse.jetty:jetty-servlets:jar:9.4.51.v20230217:compile
[INFO] Including com.google.protobuf:protobuf-java:jar:3.23.4 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4-tests.jar
[WARNING] Could not get tests for com.google.protobuf:protobuf-java:jar:3.23.4:compile
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0-tests.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0-tests.jar
[WARNING] Could not get tests for org.spark-project.spark:unused:jar:1.0.0:compile
```
Under poor network conditions, the time for Maven compilation will increase.
On the other hand, setting shadeTestJar back to false has not resulted in any Maven test failures, so this pr reset `shadeTestJar` to `false`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Action
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1704458471
cc @gengliangwang FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1704566844
@gengliangwang Do you still remember why changed `shadeTestJar` to false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1705863504
> @gengliangwang Do you still remember why changed shadeTestJar to false?
I guess it was a mistake...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1702306212
Wait test result https://github.com/LuciferYang/spark/runs/16407130667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1705873204
@LuciferYang Got it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1707613904
@gengliangwang After further investigation, there are three modules that depend on `spark-streaming_2.12:test-jar`: `mllib`, `streaming-kafka-0-10` and `kinesis-asl`. As I lack the necessary credentials to access AWS, I'm unable to verify all Unit Tests of the `kinesis-asl` module, so I've chosen to retain the configuration of `<shadeTestJar>true</shadeTestJar>` in the streaming module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
URL: https://github.com/apache/spark/pull/42766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1705871748
@gengliangwang The `connector` module has already been set to false, and the `core` module is changed to false in the current PR. The streaming module is a historical legacy and it seems to be intentionally retained(https://github.com/apache/spark/commit/37a5e272f898e946c09c2e7de5d1bda6f27a8f39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1705864189
BTW, there were multiple places setting `shadeTestJar` as true. @LuciferYang could you update the unnecessary ones as well?
```
$ git grep shadeTestJar
connector/protobuf/pom.xml: <shadeTestJar>false</shadeTestJar>
core/pom.xml: <shadeTestJar>true</shadeTestJar>
streaming/pom.xml: <shadeTestJar>true</shadeTestJar>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42766: [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42766:
URL: https://github.com/apache/spark/pull/42766#issuecomment-1707607819
Merged into master. Thanks @gengliangwang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org