You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by "Sylvain Gibier (Jira)" <ji...@apache.org> on 2021/04/24 10:49:00 UTC
[jira] [Created] (ZEPPELIN-5337) Zeppelin 0.9 / Kubernetes / Spark
interpreter / Context - scope per note
Sylvain Gibier created ZEPPELIN-5337:
----------------------------------------
Summary: Zeppelin 0.9 / Kubernetes / Spark interpreter / Context - scope per note
Key: ZEPPELIN-5337
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5337
Project: Zeppelin
Issue Type: Bug
Components: interpreter-setting, Interpreters, Kubernetes, spark
Affects Versions: 0.9.0
Reporter: Sylvain Gibier
Hi,
we have an issue with our current deployment of zeppelin on k8s, and more precisely with spark interpreter.
For reference - the spark context is: scala 2.12.10 / spark 2.4.7
We have a weird behaviour, running the spark interpreter in per note, scoped
To reproduce currently - we restart the spark interpreter in scoped per note, and create two notebooks (A & B) with the same following code:
{quote}%spark
import spark.implicits._
List(1, 2, 3).toDS.map(_ + 1).show{quote}
1- we run notebook A successfully
2 - we run notebook B - it fails with class cast exception
{quote}org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 24.0 failed 4 times, most recent failure: Lost task 0.3 in stage 24.0 (TID 161, 10.11.18.133, executor 2): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287) at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293){quote}
Anyone having a working zeppelin deployment with k8s / spark 2.4 - scala 2.12 ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)