Checkpoint issue in spark streaming

[Sadaf <sa...@platalytics.com>]

Hi all.

I am writing a twitter connector using spark streaming. i have written the
following code to maintain checkpoint.

val
ssc=StreamingContext.getOrCreate("hdfs://192.168.23.109:9000/home/cloud9/twitterCheckpoint",()=>
{ managingContext() })
      
def managingContext():StreamingContext =
  {
      //making spark context
      val conf = new
SparkConf().setMaster("local[*]").setAppName("twitterConnector")
       val ssc = new StreamingContext(conf, Seconds(1))
       val sqlContext = new
org.apache.spark.sql.SQLContext(ssc.sparkContext)
       import sqlContext.implicits._ 
       
       //checkpointing  
     
/ssc.checkpoint("hdfs://192.168.23.109:9000/home/cloud9/twitterCheckpoint")
       ssc
  }
  

but it gives the following error:

java.lang.IllegalArgumentException: requirement failed: WindowedDStream has
been marked for checkpointing but the storage level has not been set to
enable persisting. Please use DStream.persist() to set the storage level to
use memory for better checkpointing performance.

I have also mentioned the storage level through the following code.
TwitterUtils.createStream(ssc,None,null,StorageLevel.MEMORY_AND_DISK_2)


Can anyone help me in this regard? 
Thanks :) 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Checkpoint-issue-in-spark-streaming-tp24031.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org