Spark Streaming: foreachRDD network output

[Jesper Lundgren <ko...@gmail.com>]

Hello all,

I have some questions regarding the foreachRDD output function in Spark
Streaming.

The programming guide (
http://spark.apache.org/docs/1.1.0/streaming-programming-guide.html)
describes how to output data using network connection on the worker nodes.

Are there some working examples on how to do this properly? (Most of the
guide just describes what to not do, instead of what to do).

Any suggestions on what is the best way to write tests for such code? To
make sure that connection objects are used properly etc.

How to handle network or other problems on worker node? Can I throw an
exception to force spark to try again with that data on another node? As an
example: a program that writes data to an sql database using foreachRDD.
One worker node might have connection issues to the database, so it has to
let another node finish the output operation.

Thanks!

-- Jesper Lundgren

Re: Spark Streaming: foreachRDD network output

[sivarani <wh...@gmail.com>]

Any one, any luck?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-foreachRDD-network-output-tp15205p18251.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org