You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Berg Lloyd-Haig (Jira)" <ji...@apache.org> on 2021/08/18 13:45:00 UTC
[jira] [Commented] (SPARK-34095) Spark 3.0.1 is giving WARN while
running job
[ https://issues.apache.org/jira/browse/SPARK-34095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401067#comment-17401067 ]
Berg Lloyd-Haig commented on SPARK-34095:
-----------------------------------------
[~smurarka] I am getting a similar warning to this, which I am unable to hide in Jupyter Lab running openjdk-11-jre-headless/now 11.0.11+9-0ubuntu2~20.04.
I've set the log4j.logger.org.apache.spark.api.python.PythonGatewayServer setting to ERROR to try to hide this, however this has not worked for me (despite setting the Spark context logger to ERROR by default).
This is somewhat annoying for users trying to launch a Spark session in a notebook :)
Here is the output:
{code:java}
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.1.2
/_/Using Python version 3.9.6 (default, Jul 11 2021 03:39:48)
Spark context Web UI available at http://4d0ddf19f818:4040
Spark context available as 'sc' (master = local[*], app id = local-1629293803625).
SparkSession available as 'spark'.
{code}
> Spark 3.0.1 is giving WARN while running job
> --------------------------------------------
>
> Key: SPARK-34095
> URL: https://issues.apache.org/jira/browse/SPARK-34095
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 3.0.1
> Reporter: Sachit Murarka
> Priority: Minor
>
> I am running Spark 3.0.1 with Java11. Getting the following WARN.
>
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform ([file:/opt/spark/jars/spark-unsafe_2.12-3.0.1.jar|file://opt/spark/jars/spark-unsafe_2.12-3.0.1.jar]) to constructor java.nio.DirectByteBuffer(long,int)
> WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
> WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
> WARNING: All illegal access operations will be denied in a future release
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org