Spark Streaming union expected behaviour?

[Shrikar archak <sh...@gmail.com>]

Hi All,

I was writing a simple Streaming job to get more understanding about Spark
streaming.
I am not understanding why the union behaviour in this particular case

*WORKS:*
    val lines = ssc.socketTextStream("localhost", 9999,
StorageLevel.MEMORY_AND_DISK_SER)
    val words = lines..flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    wordCounts.saveAsTextFiles("all")

This works as expected as well as the streams are stored as files


*DOESN'T WORK*
    val lines = ssc.socketTextStream("localhost", 9999,
StorageLevel.MEMORY_AND_DISK_SER)
    val lines1 = ssc.socketTextStream("localhost", 10000,
StorageLevel.MEMORY_AND_DISK_SER)
   * val words = lines.union(lines1).flatMap(_.split(" "))*


    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    wordCounts.saveAsTextFiles("all")

In the above case neither the messages are printed nor the files are saved.
Am I doing something wrong here?

Thanks,
Shrikar

Re: Spark Streaming union expected behaviour?

[Varad Joshi <vj...@pivotal.io>]

I am seeing the same exact behavior. Shrikar, did you get any response to
your post?

Varad



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-union-expected-behaviour-tp7206p14284.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org