You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Hsu, Philip" <ph...@imperial.ac.uk> on 2021/06/24 07:08:14 UTC

Issue with Running Spark in Jupyter Notebook

Hi there,

My name is Philip, a master’s student at Imperial College London. I’m trying to use Spark to complete my course work assignment. I ran the following code:

from pyspark import SparkContext
sc = SparkContext.getOrCreate()

and got the following error message:

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class org.sparkproject.jetty.http.MimeTypes
        at org.sparkproject.jetty.server.handler.gzip.GzipHandler.<init>(GzipHandler.java:190)
        at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:485)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:147)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:147)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:147)
        at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:486)
        at org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:486)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:486)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.base/java.lang.Thread.run(Thread.java:829)

While in my Macbook’s terminal, it’s showing following error messages:


WARNING: An illegal reflective access operation has occurred

pyspark_mongodb_nb  | WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)

pyspark_mongodb_nb  | WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform

pyspark_mongodb_nb  | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

pyspark_mongodb_nb  | WARNING: All illegal access operations will be denied in a future release

pyspark_mongodb_nb  | 21/06/24 06:57:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

pyspark_mongodb_nb  | Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

pyspark_mongodb_nb  | Setting default log level to "WARN".

pyspark_mongodb_nb  | To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

pyspark_mongodb_nb  | 21/06/24 06:57:20 WARN MacAddressUtil: Failed to find a usable hardware address from the network interfaces; using random bytes: bd:af:a7:b4:a2:46:2a:28

I’m wondering if you could help me resolve the issues I have with my laptop. I have a 2020 MacBook Pro with a M1 chip. Thank you so much in advance.

Best,

Philip Hsu

Re: Issue with Running Spark in Jupyter Notebook

Posted by Artemis User <ar...@dtechspace.com>.
Looks like you didn't set up your environment properly.  I assume you 
are running this from a standalone python program instead of from the 
pyspark shell.  I would first run your code from the pyspark shell, then 
follow the spark python installation guide to set up your python 
environment properly.  Please note these are extra steps in addition to 
Spark installation.

-- ND

On 6/24/21 3:08 AM, Hsu, Philip wrote:
>
> Hi there,
>
> My name is Philip, a master’s student at Imperial College London. I’m 
> trying to use Spark to complete my course work assignment. I ran the 
> following code:
>
> from pyspark import SparkContext
>
> sc = SparkContext.getOrCreate()
>
> and got the following error message:
>
> Py4JJavaError: An error occurred while calling 
> None.org.apache.spark.api.java.JavaSparkContext.
>
> : java.lang.NoClassDefFoundError: Could not initialize class 
> org.sparkproject.jetty.http.MimeTypes
>
>         at 
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.<init>(GzipHandler.java:190)
>
>         at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:485)
>
>         at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:147)
>
>         at 
> org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:147)
>
>         at 
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>
>         at 
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>
>         at 
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>
>         at org.apache.spark.ui.WebUI.bind(WebUI.scala:147)
>
>         at 
> org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:486)
>
>         at 
> org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:486)
>
>         at scala.Option.foreach(Option.scala:407)
>
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:486)
>
>         at 
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
>
>         at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>
>         at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>         at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>         at 
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
>
>         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
>
>         at 
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
>         at py4j.Gateway.invoke(Gateway.java:238)
>
>         at 
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
>
>         at 
> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
>
>         at py4j.GatewayConnection.run(GatewayConnection.java:238)
>
>         at java.base/java.lang.Thread.run(Thread.java:829)
>
> While in my Macbook’s terminal, it’s showing following error messages:
>
> WARNING: An illegal reflective access operation has occurred
>
> pyspark_mongodb_nb|WARNING: Illegal reflective access by 
> org.apache.spark.unsafe.Platform 
> (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) 
> to constructor java.nio.DirectByteBuffer(long,int)
>
> pyspark_mongodb_nb|WARNING: Please consider reporting this to the 
> maintainers of org.apache.spark.unsafe.Platform
>
> pyspark_mongodb_nb|WARNING: Use --illegal-access=warn to enable 
> warnings of further illegal reflective access operations
>
> pyspark_mongodb_nb|WARNING: All illegal access operations will be 
> denied in a future release
>
> pyspark_mongodb_nb|21/06/24 06:57:17 WARN NativeCodeLoader: Unable to 
> load native-hadoop library for your platform... using builtin-java 
> classes where applicable
>
> pyspark_mongodb_nb|Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
>
> pyspark_mongodb_nb|Setting default log level to "WARN".
>
> pyspark_mongodb_nb|To adjust logging level use 
> sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>
> pyspark_mongodb_nb|21/06/24 06:57:20 WARN MacAddressUtil: Failed to 
> find a usable hardware address from the network interfaces; using 
> random bytes: bd:af:a7:b4:a2:46:2a:28
>
> I’m wondering if you could help me resolve the issues I have with my 
> laptop. I have a 2020 MacBook Pro with a M1 chip. Thank you so much in 
> advance.
>
> Best,
>
> Philip Hsu
>