You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Midhunpottammal (via GitHub)" <gi...@apache.org> on 2024/02/29 08:27:43 UTC

[I] Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available [arrow]

Midhunpottammal opened a new issue, #40287:
URL: https://github.com/apache/arrow/issues/40287

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I am trying to integrate Apache Arrow with Apache Spark in a PySpark application, but I am encountering an issue related to sun.misc.Unsafe or java.nio.DirectByteBuffer during the execution.
   
   ```
   import os
   import pandas as pd
   from pyspark.sql import SparkSession
   
   extra_java_options = os.getenv("SPARK_EXECUTOR_EXTRA_JAVA_OPTIONS", "")
   
   spark = SparkSession.builder \
       .appName("ArrowPySparkExample") \
       .getOrCreate()
   
   spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true")
   spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
   pdf = pd.DataFrame(["midhun"])
   df = spark.createDataFrame(pdf)
   result_pdf = df.select("*").toPandas()
   ```
   Error Message:
   `
   in stage 0.0 (TID 11) (192.168.140.22 executor driver): java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available at org.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:174) at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229) ...`
   
   Environment:
   
   Apache Spark version: 3.4 Apache Arrow version: 1.5 Java version: jdk 21
   
   ### Component(s)
   
   Java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available [arrow]

Posted by "Midhunpottammal (via GitHub)" <gi...@apache.org>.
Midhunpottammal commented on issue #40287:
URL: https://github.com/apache/arrow/issues/40287#issuecomment-1985078546

   @danepitkin spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true") is not required for this 
   
   i am able to solve this , more details 
   [Arrow Issue](https://github.com/apache/spark/pull/41805#issuecomment-1982513050)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available [arrow]

Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on issue #40287:
URL: https://github.com/apache/arrow/issues/40287#issuecomment-1985797292

   Ah that makes sense. Spark 3.X does not support Java 21, but Spark 4.0 will. Thanks for resolving!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available [arrow]

Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on issue #40287:
URL: https://github.com/apache/arrow/issues/40287#issuecomment-1984560733

   From my understanding, this config is not something that can be set at runtime:
   ```
   spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true")
   ```
   
   Can you try configuring it before starting spark/pyspark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available [arrow]

Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin closed issue #40287: Apache Arrow with Apache Spark - UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer not available
URL: https://github.com/apache/arrow/issues/40287


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org