Persist streams to text files

[gaganbm <ga...@gmail.com>]

Hi, 

I am trying to persist the DStreams to text files. When I use the inbuilt
API 'saveAsTextFiles' as : 

stream.saveAsTextFiles(resultDirectory) 

this creates a number of subdirectories, for each batch, and within each sub
directory, it creates bunch of text files for each RDD (I assume).

I am wondering if I can have single text files for each batch. Is there any
API for that ? Or else, a single output file for the entire stream ? 

I tried to manually write from each RDD stream to a text file as : 

stream.foreachRDD(rdd =>{ 
  rdd.foreach(element => { 
  fileWriter.write(element) 
  }) 
  }) 

where 'fileWriter' simply makes use of a Java BufferedWriter to write
strings to a file. However, this fails with exception : 

DStreamCheckpointData.writeObject used 
java.io.BufferedWriter 
java.io.NotSerializableException: java.io.BufferedWriter 
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) 
        at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) 
        ..... 

Any help on how to proceed with this ? 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Persist-streams-to-text-files-tp2986.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.