You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mrmadira (via GitHub)" <gi...@apache.org> on 2023/07/17 08:26:41 UTC
[GitHub] [arrow] mrmadira opened a new issue, #36713: Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)
mrmadira opened a new issue, #36713:
URL: https://github.com/apache/arrow/issues/36713
### Describe the enhancement requested
Latest arrow-memory-netty jar fails to work with latest netty-handler.jar with the following exception.
```
java.lang.NoSuchMethodError: io/netty/buffer/PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolThreadCache; (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33) called from class io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33).
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:315)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:310)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:298)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)
at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:725)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.loadBatch(ArrowConverters.scala:189)
at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.nextBatch(ArrowConverters.scala:165)
at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.<init>(ArrowConverters.scala:144)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.fromBatchIterator(ArrowConverters.scala:143)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.$anonfun$toDataFrame$1(ArrowConverters.scala:203)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:839)
```
We believe this change in netty-handler is where the integration breaks.: https://github.com/netty/netty/commit/9412424091eb89f9d55372e38fedbcfcc285dd94
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on issue #36713: [Java] Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on issue #36713:
URL: https://github.com/apache/arrow/issues/36713#issuecomment-1648126227
Hey @mrmadira , a new Arrow v13 release is in process right now that will release arrow-memory-netty v13.0.0, which will be pinned to netty v4.1.94.final.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on issue #36713: [Java] Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on issue #36713:
URL: https://github.com/apache/arrow/issues/36713#issuecomment-1648134237
Here's the PR that upgrades netty in Arrow: https://github.com/apache/arrow/pull/36211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org