[jira] [Created] (SPARK-5146) saveAsHadoopFiles does not work with checkpointing

["Halit Olali (JIRA)" <ji...@apache.org>]

Halit Olali created SPARK-5146:
----------------------------------

             Summary: saveAsHadoopFiles does not work with checkpointing
                 Key: SPARK-5146
                 URL: https://issues.apache.org/jira/browse/SPARK-5146
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.2.0
            Reporter: Halit Olali
            Priority: Minor


Hello folks,
Consider the following simple app for word counting via network socket:
    val conf = new SparkConf().setAppName("Sample Application")
    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Seconds(5))
    ssc.checkpoint("target/checkpointDir")
    val lines = ssc.socketTextStream("localhost", 9999)
    val words = lines.flatMap(_.split(" "))
    val pairs = words.map(word => (word, 1))
    val wordCounts = pairs.reduceByKey(_ + _)
    wordCounts.saveAsHadoopFiles("target/prefix","suffix")
    //nc -lk 9999
    ssc.start()
    ssc.awaitTermination(60)

When this is packaged and executed on spark, following exception is thrown:
java.io.NotSerializableException: org.apache.hadoop.mapred.JobConf

JobConf usage inside saveAsHadoopFiles methods seems to be the cause.

Thanks




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org