[GitHub] [iceberg] yanivkrol commented on issue #3405: Spark 3.2 benchmark failed with netty.buffer issue

["yanivkrol (via GitHub)" <gi...@apache.org>]

yanivkrol commented on issue #3405:
URL: https://github.com/apache/iceberg/issues/3405#issuecomment-1587372823

   > Hi @Rajashekhar-meesala, is the error message exactly the same? If it is, you need to check all your dependencies including the transitive dependencies to make sure there is no old version(< 4.1.63.Final) of netty buffer get into your Spark runtime.
   
   Hi, I'm getting the same error. It seems that I get version `4.1.45.Final` as part of spark's dependencies:
   ```
   Your branch is up to date with 'origin/master'.
   ❯ mvn dependency:tree | grep netty
   [INFO] |  |  \- io.netty:netty:jar:3.10.5.Final:compile
   ❯ mvn dependency:tree | grep netty
   [INFO] |  |  +- io.netty:netty-all:jar:4.1.45.Final:provided
   [INFO] |  |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.45.Final:provided
   [INFO] |  |  |  +- io.netty:netty-common:jar:4.1.45.Final:provided
   [INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.45.Final:provided
   [INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.45.Final:provided
   [INFO] |  |  |  |  \- io.netty:netty-resolver:jar:4.1.45.Final:provided
   [INFO] |  |  |  \- io.netty:netty-transport-native-unix-common:jar:4.1.45.Final:provided
   [INFO] |  |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final:provided
   [INFO] |  |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.87.Final:provided
   [INFO] |  |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final:provided
   [INFO] |  |  |  \- io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final:provided
   [INFO] |  |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.45.Final:provided
   [INFO] |  |  \- org.apache.arrow:arrow-memory-netty:jar:11.0.0:provided
   [INFO] |  |  \- io.netty:netty:jar:3.10.5.Final:compile
   ```
   Using:
   ```
   <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-sql_2.13</artifactId>
       <version>3.4.0</version>
       <scope>provided</scope>
   </dependency>
   ```
   
   I'm on MacOS with M1 chip


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org