[jira] [Created] (TOREE-457) spark context seen corrupted after load KAfka libraries

["fxoSa (JIRA)" <ji...@apache.org>]

fxoSa created TOREE-457:
---------------------------

             Summary: spark context seen corrupted after load KAfka libraries
                 Key: TOREE-457
                 URL: https://issues.apache.org/jira/browse/TOREE-457
             Project: TOREE
          Issue Type: Bug
          Components: Kernel
            Reporter: fxoSa
            Priority: Minor


I am trying to set up a jupyter notebook (apache-toree Scala) to access kafka logs from spark a streaming.
First I add dependencies using AddDeps:
 
{code:java}
%AddDeps org.apache.spark spark-streaming-kafka-0-10_2.11 2.2.0. 
Marking org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0 for download Preparing to fetch from:
-> file:/tmp/toree_add_deps8235567186565695423/
-> https://repo1.maven.org/maven2
-> New file at /tmp/toree_add_deps8235567186565695423/https/repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka-0-10_2.11/2.2.0/spark-streaming-kafka-0-10_2.11-2.2.0.jar
{code}
After that I am able to import successfully part of necesary libraries:

{code:java}

import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka010._
{code}


However code fails when I try to create streaming context:


{code:java}
val ssc = new StreamingContext(sc, Seconds(2))

    Name: Compile Error
Message: <console>:38: error: overloaded method constructor StreamingContext with alternatives:
  (path: String,sparkContext: org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.spark.streaming.StreamingContext <and>
  (path: String,hadoopConf: org.apache.hadoop.conf.Configuration)org.apache.spark.streaming.StreamingContext <and>
  (conf: org.apache.spark.SparkConf,batchDuration: org.apache.spark.streaming.Duration)org.apache.spark.streaming.StreamingContext <and>
  (sparkContext: org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext,batchDuration: org.apache.spark.streaming.Duration)org.apache.spark.streaming.StreamingContext
 cannot be applied to (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext, org.apache.spark.streaming.Duration)
       val ssc = new StreamingContext(sc, Seconds(2))
                 ^
StackTrace: 

{code}

I have try it, in a jupyter docker 
https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook
and in spark cluster set up in Google cloud platform with the same results
Thanks





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)