You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/06/28 18:05:31 UTC
[spark] branch master updated: [SPARK-44128][BUILD] Upgrade netty to 4.1.93
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d2054fc358d [SPARK-44128][BUILD] Upgrade netty to 4.1.93
d2054fc358d is described below
commit d2054fc358d4727d13e3fb7cb26c35b68f6a3fb8
Author: panbingkun <pb...@gmail.com>
AuthorDate: Wed Jun 28 11:05:20 2023 -0700
[SPARK-44128][BUILD] Upgrade netty to 4.1.93
### What changes were proposed in this pull request?
This pr aims to upgrade netty from 4.1.92 to 4.1.93.
### Why are the changes needed?
1.v4.1.92 VS v4.1.93
https://github.com/netty/netty/compare/netty-4.1.92.Final...netty-4.1.93.Final
2.The new version brings some bug fix, eg:
- Reset byte buffer in loop for AbstractDiskHttpData.setContent ([#13320](https://github.com/netty/netty/pull/13320))
- OpenSSL MAX_CERTIFICATE_LIST_BYTES option supported ([#13365](https://github.com/netty/netty/pull/13365))
- Adapt to DirectByteBuffer constructor in Java 21 ([#13366](https://github.com/netty/netty/pull/13366))
- HTTP/2 encoder: allow HEADER_TABLE_SIZE greater than Integer.MAX_VALUE ([#13368](https://github.com/netty/netty/pull/13368))
- Upgrade to latest netty-tcnative to fix memory leak ([#13375](https://github.com/netty/netty/pull/13375))
- H2/H2C server stream channels deactivated while write still in progress ([#13388](https://github.com/netty/netty/pull/13388))
- Channel#bytesBefore(un)writable off by 1 ([#13389](https://github.com/netty/netty/pull/13389))
- HTTP/2 should forward shutdown user events to active streams ([#13394](https://github.com/netty/netty/pull/13394))
- Respect the number of bytes read per datagram when using recvmmsg ([#13399](https://github.com/netty/netty/pull/13399))
3.The release notes as follows:
- https://netty.io/news/2023/05/25/4-1-93-Final.html
4.Why not upgrade to `4-1-94-Final` version?
Because the return value of the 'threadCache()' (from `PoolThreadCache` to `PoolArenasCache`) method of the netty Inner class used in the 'arrow memory netty' version '12.0.1' has changed and belongs to break change, let's wait for the upgrade of the 'arrow memory netty' before upgrading to the '4-1-94-Final' version.
The reference is as follows:
https://github.com/apache/arrow/blob/6af660f48472b8b45a5e01b7136b9b040b185eb1/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java#L164
https://github.com/netty/netty/blob/da1a448d5bc4f36cc1744db93fcaf64e198db2bd/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L732-L736
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA.
Closes #41681 from panbingkun/upgrade_netty.
Authored-by: panbingkun <pb...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 36 +++++++++++++++++------------------
pom.xml | 6 +++++-
2 files changed, 23 insertions(+), 19 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 153f5b57ce5..8630040abad 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -183,24 +183,24 @@ metrics-jmx/4.2.18//metrics-jmx-4.2.18.jar
metrics-json/4.2.18//metrics-json-4.2.18.jar
metrics-jvm/4.2.18//metrics-jvm-4.2.18.jar
minlog/1.3.0//minlog-1.3.0.jar
-netty-all/4.1.92.Final//netty-all-4.1.92.Final.jar
-netty-buffer/4.1.92.Final//netty-buffer-4.1.92.Final.jar
-netty-codec-http/4.1.92.Final//netty-codec-http-4.1.92.Final.jar
-netty-codec-http2/4.1.92.Final//netty-codec-http2-4.1.92.Final.jar
-netty-codec-socks/4.1.92.Final//netty-codec-socks-4.1.92.Final.jar
-netty-codec/4.1.92.Final//netty-codec-4.1.92.Final.jar
-netty-common/4.1.92.Final//netty-common-4.1.92.Final.jar
-netty-handler-proxy/4.1.92.Final//netty-handler-proxy-4.1.92.Final.jar
-netty-handler/4.1.92.Final//netty-handler-4.1.92.Final.jar
-netty-resolver/4.1.92.Final//netty-resolver-4.1.92.Final.jar
-netty-transport-classes-epoll/4.1.92.Final//netty-transport-classes-epoll-4.1.92.Final.jar
-netty-transport-classes-kqueue/4.1.92.Final//netty-transport-classes-kqueue-4.1.92.Final.jar
-netty-transport-native-epoll/4.1.92.Final/linux-aarch_64/netty-transport-native-epoll-4.1.92.Final-linux-aarch_64.jar
-netty-transport-native-epoll/4.1.92.Final/linux-x86_64/netty-transport-native-epoll-4.1.92.Final-linux-x86_64.jar
-netty-transport-native-kqueue/4.1.92.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.92.Final-osx-aarch_64.jar
-netty-transport-native-kqueue/4.1.92.Final/osx-x86_64/netty-transport-native-kqueue-4.1.92.Final-osx-x86_64.jar
-netty-transport-native-unix-common/4.1.92.Final//netty-transport-native-unix-common-4.1.92.Final.jar
-netty-transport/4.1.92.Final//netty-transport-4.1.92.Final.jar
+netty-all/4.1.93.Final//netty-all-4.1.93.Final.jar
+netty-buffer/4.1.93.Final//netty-buffer-4.1.93.Final.jar
+netty-codec-http/4.1.93.Final//netty-codec-http-4.1.93.Final.jar
+netty-codec-http2/4.1.93.Final//netty-codec-http2-4.1.93.Final.jar
+netty-codec-socks/4.1.93.Final//netty-codec-socks-4.1.93.Final.jar
+netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
+netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
+netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
+netty-resolver/4.1.93.Final//netty-resolver-4.1.93.Final.jar
+netty-transport-classes-epoll/4.1.93.Final//netty-transport-classes-epoll-4.1.93.Final.jar
+netty-transport-classes-kqueue/4.1.93.Final//netty-transport-classes-kqueue-4.1.93.Final.jar
+netty-transport-native-epoll/4.1.93.Final/linux-aarch_64/netty-transport-native-epoll-4.1.93.Final-linux-aarch_64.jar
+netty-transport-native-epoll/4.1.93.Final/linux-x86_64/netty-transport-native-epoll-4.1.93.Final-linux-x86_64.jar
+netty-transport-native-kqueue/4.1.93.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.93.Final-osx-aarch_64.jar
+netty-transport-native-kqueue/4.1.93.Final/osx-x86_64/netty-transport-native-kqueue-4.1.93.Final-osx-x86_64.jar
+netty-transport-native-unix-common/4.1.93.Final//netty-transport-native-unix-common-4.1.93.Final.jar
+netty-transport/4.1.93.Final//netty-transport-4.1.93.Final.jar
objenesis/3.3//objenesis-3.3.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.15.0//okio-1.15.0.jar
diff --git a/pom.xml b/pom.xml
index 5c682792b34..c8dbd0ab2d8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -212,7 +212,11 @@
<commons-cli.version>1.5.0</commons-cli.version>
<bouncycastle.version>1.60</bouncycastle.version>
<tink.version>1.9.0</tink.version>
- <netty.version>4.1.92.Final</netty.version>
+ <!--
+ Please don't upgrade the version to 4.1.94.Final,
+ it needs to wait for `arrow-memory-netty` to be greater than 12.0.1.
+ -->
+ <netty.version>4.1.93.Final</netty.version>
<!--
If you are changing Arrow version specification, please check
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org