[jira] [Created] (ZEPPELIN-3591) Some values of "args" property in interpreter settings for Spark ruin UDF execution

["Denis Efarov (JIRA)" <ji...@apache.org>]

Denis Efarov created ZEPPELIN-3591:
--------------------------------------

             Summary: Some values of "args" property in interpreter settings for Spark ruin UDF execution
                 Key: ZEPPELIN-3591
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3591
             Project: Zeppelin
          Issue Type: Bug
          Components: zeppelin-interpreter
    Affects Versions: 0.7.2
         Environment: CentOS Linux 7.3.1611

Java 1.8.0_60

Scala 2.11.8

Spark 2.1.1

Hadoop 2.6.0

Zeppelin 0.7.2

 

 
            Reporter: Denis Efarov


In "args" interpreter configuration property, any value which starts with "-" (minus) sign prevents correct UDF execution in Spark running on YARN. Text after "-" doesn't matter, it fails anyway. All the other properties do not affect this.

Steps to reproduce: 
 * On the interpreter settings page, find Spark interpreter
 * For "args" property, put any value starting with "-", for example "-test" 
 * Make sure spark starts on yarn (master=yarn-client)
 * Save settings and restart the interpreter
 * In any notebook, write and execute the following code:
 ** %spark
val udfDemo = (i: Int) => i + 10;
sqlContext.udf.register("demoUdf", (i: Int) => i);
sqlContext.sql("select demoUdf(1) val").show

Stacktrace:

{{java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD}}{{at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2083)}}{{at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)}}{{at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1996)}}{{at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)}}{{at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)}}{{at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)}}{{at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)}}{{at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)}}{{at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)}}{{at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)}}{{at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)}}{{at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)}}{{at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)}}{{at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:80)}}{{at org.apache.spark.scheduler.Task.run(Task.scala:99)}}{{at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)}}{{at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)}}{{at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)}}{{at java.lang.Thread.run(Thread.java:744)}}

Making the same UDF declaration in, for example, %pyspark interpreter, helps, even if one executes it in %spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)