[GitHub] [spark] HyukjinKwon commented on a change in pull request #31551: Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #31551:
URL: https://github.com/apache/spark/pull/31551#discussion_r575619704



##########
File path: docs/index.md
##########
@@ -53,6 +53,11 @@ uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scal
 For Python 3.9, Arrow optimization and pandas UDFs might not work due to the supported Python versions in Apache Arrow. Please refer to the latest [Python Compatibility](https://arrow.apache.org/docs/python/install.html#python-compatibility) page.
 For Java 11, `-Dio.netty.tryReflectionSetAccessible=true` is required additionally for Apache Arrow library. This prevents `java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available` when Apache Arrow uses Netty internally.
 
+```python
+# PySpark
+SparkSession.builder.config('spark.driver.extraJavaOptions', '-Dio.netty.tryReflectionSetAccessible=true').getOrCreate()

Review comment:
       @Nozziel, I think we should better just mention both configurations `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` because there are many other ways to set the configurations, for example, Spark submit script, Spark configuration file, etc. In addition, Arrow exposes a set of developer APIs that are not specific to PySpark side.
   
   Also, it would be great if we file a JIRA, and fix the PR title accordingly. See also http://spark.apache.org/contributing.html




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org