You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hasnain-db (via GitHub)" <gi...@apache.org> on 2023/10/09 05:15:42 UTC
[PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
hasnain-db opened a new pull request, #43289:
URL: https://github.com/apache/spark/pull/43289
### What changes were proposed in this pull request?
Exclude a `LICENSE` file in favor of an existing `LICENSE.txt` with the same contents to fix the build for network-yarn
### Why are the changes needed?
Making a distribution build using the following command currently fails:
```
./dev/make-distribution.sh --tgz -Phive -Phive-thriftserver -Pyarn
```
Failures look like
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
[ERROR] java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
[ERROR] around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
```
After much trial and error (I added a debugging section for the next person who runs into this) I realized the problem is that inside the jar, the META-inf folder looks like this:
```
META-INF/LICENSE
META-INF/LICENSE.txt
META-INF/license/
META-INF/license/LICENSE.aix-netbsd.txt
META-INF/license/LICENSE.boringssl.txt
META-INF/license/LICENSE.mvn-wrapper.txt
META-INF/license/LICENSE.tomcat-native.txt
```
Since the filesystem on macosx is case insensitive, this causes issues. Unzipping this fails, as we unzip the file first, then do not create the directory, and fail to unzip the files after that.
This PR proposes to fix it by removing the `LICENSE` file and keeping the `license` folder, which lets the unzipping proceed. The contents of the `LICENSE` file and`LICENSE.txt` are the same so that should be fine I hope license wise - we are still showing the same license in the output jar file.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI
I reran
```
./dev/make-distribution.sh --tgz -Phive -Phive-thriftserver -Pyarn
```
and it succeeded
### Was this patch authored or co-authored using generative AI tooling?
No
### Appendix: debugging
I was able to debug the contents of the jar using `jar tf $jar` and print out the files using `unzip -q -c $jar $path`.
To make iterations go faster I went to `/Users/hasnain.lakhani/spark/common/network-yarn/target/antrun` and edited the ant file by hand then ran `ant -f build-main.xml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1756633448
> @LuciferYang it failed again :/ is it possible to merge this with that test failing?
I've checked, the failure should be unrelated to this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350364095
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
@srowen I'll try the initial attempt of this PR then - we just remove a single file called LICENSE which is the apache2 license and is identical to a LICENSE.txt that is still going to be in the final jar. If that works (testing now), would it be ok? IMO it's preferable as then we don't need to update the license-binary dir and it's also future proof in case the contents of the directory change. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1755442652
just retriggered. I've done it 3-4 times already though so I'm not sure this will help. will post back when it's done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350440904
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/LICENSE</exclude>
Review Comment:
Understood, thank you for your explanation @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350261258
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
If there's no way around that, we probably need to copy those licenses into the Spark license-binary dir. We should not drop them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1756630873
Merged into master for Spark 4.0, thanks @hasnain-db @srowen @dongjoon-hyun @wangyum @cfmcgrady
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350031986
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
cc @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350363495
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
Since this is a MacOS only issue and we generally don't use MacOS for official version releases, if we accept not packaging `META-INF/license/*` on MacOS, then we might be able to workaround by adding a profile that is activated by default on MacOS.
```
<profile>
<id>mac</id>
<activation>
<os>
<family>mac</family>
</os>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
...
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/license/*</exclude>
</excludes>
</filter>
</filters>
...
</plugin>
</plugins>
</build>
</profile>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350258223
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
[ERROR] java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
[ERROR] around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
```
A MacOS only issue: https://github.com/apache/spark/pull/43164#issuecomment-1751949491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752467234
we may consider adding these renaming rules to the SBT build.
```
$ ./build/sbt -Pyarn network-yarn/assembly
$ jar tf common/network-yarn/target/scala-2.13/spark-network-yarn-4.0.0-SNAPSHOT-hadoop3.3.6.jar | grep libnetty
META-INF/native/libnetty_transport_native_epoll_aarch_64.so
META-INF/native/libnetty_transport_native_epoll_x86_64.so
META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350365295
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
> If there's no way around that, we probably need to copy those licenses into the Spark license-binary dir. We should not drop them.
If it's not acceptable to not package it on MacOS, then it might be necessary to fix it using this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1755735928
failed again, retrying once more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1754434466
Yes, `re-trigger` will help. I saw the same situation before after upgrading the recent Oracle image, but didn't check the root cause yet, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350398908
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
If it's Apache licensed then no action needed, this is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1754433043
> @LuciferYang the docker integration tests are consistently failing with
>
> ```
> [info] *** 2 SUITES ABORTED ***
> [error] Error during tests:
> [error] org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite
> [error] org.apache.spark.sql.jdbc.OracleIntegrationSuite
> ```
>
> I don't know if they are flaky but it seems to be unrelated to this PR
https://github.com/apache/spark/pull/43123#issuecomment-1752118564
friendly ping @dongjoon-hyun, it seems like this is an unstable case? I noticed that you have also encountered similar failures.
@hasnain-db Can you manually re-trigger `Run Docker integration tests`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752388058
> This depends on whether the YarnShuffleService needs the tcnative-boringssl related functions. If so, it indeed needs to be shaded, and libnetty_tcnative_* should likely be renamed similarly; if not, these jars can be directly excluded from the shade.
If it's not provided, the functionality will still work, it'll just be slower (falling back to the JDK implementation). I think for now for simplicity's sake (especially since I don't have a great setup for testing yarn beyond unit tests) I think it makes sense to exclude from the shading, that seems simpler. Will try that out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752380733
> Another question is, are we sure that network-yarn needs to shade the netty-tcnative-boringssl-static related jar packages? If indeed necessary, should we rename libnetty_tcnative_* in a similar way to libnetty_transport*?
I am not sure if it *needs* to shade them. I'll admit I'm not the most familiar with this aspect of things, happy to try it without. I will say that even with the shading it does seem to work fine (tests in https://github.com/apache/spark/pull/42685 pass) so aside from this issue I am not sure about the harm of keeping it shaded for consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752480682
> libnetty_transport_native_kqueue_x86_64
Yes, this needs to be fixed. However, inconsistencies between `sbt assembly` and `maven shaded` already existed before adding `boringssl `, and there are likely more. The main goal of this pr is to fix `make-distribution.sh`, so I personally prefer to address the differences between `sbt assembly` and `maven shaded` in a series of separate PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1755870502
@LuciferYang it failed again :/ is it possible to merge this with that test failing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1349878165
##########
common/network-yarn/pom.xml:
##########
@@ -174,7 +174,18 @@
<configuration>
<target>
<echo message="Shade netty native libraries to ${spark.shade.native.packageName}" />
- <unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/" />
+ <!--
+ We have a LICENSE file with apache licenses and license directory for netty-tcnative licenses. This breaks
+ packaging on OSX where filesystems are case insensitive. So, create the directory, which results
+ in the LICENSE file not being written, letting the netty licenses be copied over.
+ LICENSE.txt also contains the apache license so this should be OK
+ -->
+ <mkdir dir="${project.build.directory}/exploded/META-INF/LICENSE" />
+ <unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/">
+ <patternset>
+ <exclude name="META-INF/LICENSE"/>
+ </patternset>
+ </unzip>
Review Comment:
trying that out now, will report back!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752406371
@hasnain-db I suggest following @LuciferYang's suggestion to follow the existing rule to rename (the prefix must exactly match the relocated package name) the newly introduced native library, it's actually a standard way of creating artifacts including shaded Netty classes and native libraries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752358901
cc: @wangyum @LuciferYang please let me know if this fix works for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752422252
I've updated to rename the libraries + use the exclusion, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350432918
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/LICENSE</exclude>
Review Comment:
```
[WARNING] jackson-annotations-2.15.2.jar, jackson-core-2.15.2.jar, jackson-databind-2.15.2.jar, jackson-module-scala_2.13-2.15.2.jar, log4j-1.2-api-2.20.0.jar, log4j-api-2.20.0.jar, log4j-core-2.20.0.jar, log4j-slf4j2-impl-2.20.0.jar, spark-common-utils_2.13-4.0.0-SNAPSHOT.jar, spark-network-common_2.13-4.0.0-SNAPSHOT.jar, spark-network-shuffle_2.13-4.0.0-SNAPSHOT.jar, spark-network-yarn_2.13-4.0.0-SNAPSHOT.jar define 1 overlapping resource:
[WARNING] - META-INF/LICENSE
```
From the compile log, `META-INF/LICENSE` might come from multiple jars.
Since this `exclude` is indiscriminate, in this case, all `META-INF/LICENSE` would be excluded. If it includes a non-Apache LICENSE, it would also be excluded. Would this be a problem @srowen ?
Of course, even without the `exclude`, there would still be issues currently because multiple files with the same name would overwrite each other, and it seems difficult to determine which `META-INF/LICENSE` would be retained in the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350437693
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/LICENSE</exclude>
Review Comment:
It looks like we include the license if all of those already in the LICENSE file (or no action is required) so it seems OK here. While it's nice to retain the license buried in the artifact, we should be addressing all licenses in the top level LICENSE and license/ dir, rather than rely on this anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1754435400
Thank you for your confirmation @dongjoon-hyun ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #43289: [SPARK-45464][CORE] Fix network-yarn distribution build
URL: https://github.com/apache/spark/pull/43289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1756710477
np - thanks for merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350258223
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
[ERROR] java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
[ERROR] around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (unpack) on project spark-network-yarn_2.13: An Ant BuildException has occured: Error while expanding /Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar
java.nio.file.FileSystemException: /Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/META-INF/license/LICENSE.aix-netbsd.txt: Not a directory
around Ant part ...<unzip src="/Users/hasnain.lakhani/spark/common/network-yarn/target/scala-2.13/spark-4.0.0-SNAPSHOT-yarn-shuffle.jar" dest="/Users/hasnain.lakhani/spark/common/network-yarn/target/exploded/" />... @ 5:198 in /Users/hasnain.lakhani/spark/common/network-yarn/target/antrun/build-main.xml
```
A MacOS only issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1349871927
##########
common/network-yarn/pom.xml:
##########
@@ -174,7 +174,18 @@
<configuration>
<target>
<echo message="Shade netty native libraries to ${spark.shade.native.packageName}" />
- <unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/" />
+ <!--
+ We have a LICENSE file with apache licenses and license directory for netty-tcnative licenses. This breaks
+ packaging on OSX where filesystems are case insensitive. So, create the directory, which results
+ in the LICENSE file not being written, letting the netty licenses be copied over.
+ LICENSE.txt also contains the apache license so this should be OK
+ -->
+ <mkdir dir="${project.build.directory}/exploded/META-INF/LICENSE" />
+ <unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/">
+ <patternset>
+ <exclude name="META-INF/LICENSE"/>
+ </patternset>
+ </unzip>
Review Comment:
https://github.com/apache/spark/blob/ffc6994f46534dd955a387cea51afb9a67cb781e/common/network-yarn/pom.xml#L106-L114
How about adding `<exclude>META-INF/license/*</exclude>` directly to the above `excludes` list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350256419
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
Why is this excluded? We should in general retain license files in artifacts, and I don't imagine it would generally cause any problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350401556
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/license/*</exclude>
Review Comment:
@srowen it's apache licensed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "hasnain-db (via GitHub)" <gi...@apache.org>.
hasnain-db commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1754299341
@LuciferYang the docker integration tests are consistently failing with
```
[info] *** 2 SUITES ABORTED ***
[error] Error during tests:
[error] org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite
[error] org.apache.spark.sql.jdbc.OracleIntegrationSuite
```
I don't know if they are flaky but it seems to be unrelated to this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43289:
URL: https://github.com/apache/spark/pull/43289#issuecomment-1752385817
This depends on whether the `YarnShuffleService` needs the `tcnative-boringssl` related functions. If so, it indeed needs to be shaded, and `libnetty_tcnative_*` should likely be renamed similarly; if not, these jars can be directly excluded from the shade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45464][CORE] Fix network-yarn distribution build [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43289:
URL: https://github.com/apache/spark/pull/43289#discussion_r1350432918
##########
common/network-yarn/pom.xml:
##########
@@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/LICENSE</exclude>
Review Comment:
```
[WARNING] jackson-annotations-2.15.2.jar, jackson-core-2.15.2.jar, jackson-databind-2.15.2.jar, jackson-module-scala_2.13-2.15.2.jar, log4j-1.2-api-2.20.0.jar, log4j-api-2.20.0.jar, log4j-core-2.20.0.jar, log4j-slf4j2-impl-2.20.0.jar, spark-common-utils_2.13-4.0.0-SNAPSHOT.jar, spark-network-common_2.13-4.0.0-SNAPSHOT.jar, spark-network-shuffle_2.13-4.0.0-SNAPSHOT.jar, spark-network-yarn_2.13-4.0.0-SNAPSHOT.jar define 1 overlapping resource:
[WARNING] - META-INF/LICENSE
```
From the compile log, `META-INF/LICENSE` might come from multiple jars.
Since this `exclude` is indiscriminate, in this case, all `META-INF/LICENSE` would be excluded. If it includes a non-Apache LICENSE, it would also be excluded. Would this be a problem @srowen ?
Of course, even without the exclude, there would still be issues currently because multiple files with the same name would overwrite each other, and it seems difficult to determine which `META-INF/LICENSE` would be retained in the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org