You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/06 13:32:53 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
HyukjinKwon opened a new pull request, #38132:
URL: https://github.com/apache/spark/pull/38132
### What changes were proposed in this pull request?
This PR makes the following dependencies shaded:
```
com.google.android:annotations
com.google.api.grpc:proto-google-common-proto
io.perfmark:perfmark-api
org.codehaus.mojo:animal-sniffer-annotations
com.google.errorprone:error_prone_annotations
com.google.j2objc:j2objc-annotations
```
Before https://github.com/apache/spark/pull/38109, it worked because related dependences pulled together but now we don't as Spark Connect would be a single jar. This issue has existed from the very first place.
### Why are the changes needed?
Otherwise, the tests fails if you build Spark Connect with Maven. SBT does not have the issue because it does the assemply with all dependencies.
### Does this PR introduce _any_ user-facing change?
No, the codes are not released out yet.
### How was this patch tested?
Manually tested via Maven:
```bash
./build/mvn clean package
./python/run-tests --testnames 'pyspark.sql.tests.test_connect_basic'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270058932
cc @amaliujia too FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270601270
question:
how do you decide which package needs a shading?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270871393
Probably there should be a better way I believe but I am going to merge it first to fix the broken test cases with Maven.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38132:
URL: https://github.com/apache/spark/pull/38132#discussion_r989909422
##########
connector/connect/pom.xml:
##########
@@ -271,6 +282,31 @@
<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.connect.grpc</shadedPattern>
</relocation>
+
+ <relocation>
+ <pattern>com.google.android</pattern>
Review Comment:
@HyukjinKwon Sorry for the late reply
The relocation rule may be incorrect
I think it should be
```
<pattern>android.annotation</pattern>
<shadedPattern>${spark.shade.packageName}.connect.android.annotation</shadedPattern>
```
and unzip the `assembly` jar, the contents are as follows:
```
ls
META-INF android com google grpc org spark
```
Not all contents are placed in `${spark.shade.packageName}.connect`, which may require further check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270089451
Build: https://github.com/HyukjinKwon/spark/actions/runs/3197581888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
URL: https://github.com/apache/spark/pull/38132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38132:
URL: https://github.com/apache/spark/pull/38132#discussion_r989958977
##########
connector/connect/pom.xml:
##########
@@ -271,6 +282,31 @@
<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.connect.grpc</shadedPattern>
</relocation>
+
+ <relocation>
+ <pattern>com.google.android</pattern>
Review Comment:
OK~ Maybe do this tomorrow, a little tired today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270057502
cc @LuciferYang FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270870416
Oh, these were jars missing, andI manually added them (see also https://github.com/apache/spark/pull/38109#issuecomment-1268357290)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38132:
URL: https://github.com/apache/spark/pull/38132#issuecomment-1270871867
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38132: [SPARK-40677][CONNECT][BUILD] Shade more dependency to be able to run separately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38132:
URL: https://github.com/apache/spark/pull/38132#discussion_r989938384
##########
connector/connect/pom.xml:
##########
@@ -271,6 +282,31 @@
<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.connect.grpc</shadedPattern>
</relocation>
+
+ <relocation>
+ <pattern>com.google.android</pattern>
Review Comment:
Hey sure, please go ahead with a followup. I just rushed to merge to fix up the broken tests with Maven :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org