Is there a way to create multiple streams in spark streaming?

[LinQili <li...@outlook.com>]

Hi all,I wonder if there is a way to create some child streaming while using spark streaming?For example, I create a netcat main stream, read data from a socket, then create 3 different child streams on the main stream,in stream1, we do fun1 on the input data then print result to screen;in stream2, we do fun2 on the input data then print result to screen;in stream3, we do fun3 on the input data then print result to screen.Is any one some hints?  		 	   		  

Re: Is there a way to create multiple streams in spark streaming?

[Gerard Maas <ge...@gmail.com>]

You can create as many functional derivates of your original stream by
using transformations. That's exactly the model that Spark Streaming offers.

In your example, that would become something like:

val stream = ssc.socketTextStream("localhost", 9999)
val stream1 = stream.map(fun1)
val stream2 = stream.map(fun2)
// you could also:
val stream3 = stream2.filter(predicate).flatMap(ffun3)

// Then you need some action to materialize the streams:
stream2.print
stream2.saveAsTextFiles(....)

-kr, Gerard.


On Tue, Oct 20, 2015 at 12:20 PM, LinQili <li...@outlook.com> wrote:

> Hi all,
> I wonder if there is a way to create some child streaming while using
> spark streaming?
> For example, I create a netcat main stream, read data from a socket, then
> create 3 different child streams on the main stream,
> in stream1, we do fun1 on the input data then print result to screen;
> in stream2, we do fun2 on the input data then print result to screen;
> in stream3, we do fun3 on the input data then print result to screen.
> Is any one some hints?
>