You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/15 03:37:21 UTC
[GitHub] [spark] LuciferYang edited a comment on pull request #34881: [SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71
LuciferYang edited a comment on pull request #34881:
URL: https://github.com/apache/spark/pull/34881#issuecomment-994261211
> May I ask you to run a maven build as well? Just to be in the safe side... I am not 100% sure but as you changed the exclusion in `SparkBuild.scala` but not in a pom.xml we might find something here.
> And agree, I think you need Maven exclusions to match SBT
@attilapiros @srowen
In SPARK-35134 (https://github.com/apache/spark/pull/32230), there is a netty version conflict issue during sbt test process, the phenomenon that triggers the bug is that even if we use `netty-4.1.51-all.jar`, there will be `copying deps/jars/netty-handler-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars` and `copying deps/jars/netty-transport-native-epoll-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars` behavior during sbt testing.
So I add 2 new `ExclusionRule` in SPARK-35134 to fix this sbt test bug.
```
ExclusionRule("io.netty", "netty-handler"),
ExclusionRule("io.netty", "netty-transport-native-epoll")
```
We didn't find this problem during Maven testing, so I didn't make corresponding change for Maven In SPARK-35134.
Then after Netty-4.1.69 ([Netty#11732 netty-all should not re-package jars](https://github.com/netty/netty/pull/11732)) , netty-all replaces a fat jar with separate modules, keeping `ExclusionRule("io.netty", "netty-handler")` and `ExclusionRule("io.netty", "netty-transport-native-epoll")` will result in the absence of `netty-transport-native-epoll-4.1.71.Final.jar` and `netty-handler-4.1.71.Final.jar` in the during sbt test because `netty-all-4.1.71.jar` no longer contains them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org