[Spark Streaming] DAG Execution Model Clarification

[Nipun Arora <ni...@gmail.com>]

Hi,

I would like some clarification on the execution model for spark streaming.

Broadly, I am trying to understand if output operations in a DAG are only
processed after all intermediate operations are finished for all parts of
the DAG.

Let me give an example:

I have a dstream -A , I do map operations on this dstream and create two
different dstreams -B and C such that

A ---> B -> (some operations) ---> kafka output 1
  \----> C---> ( some operations) --> kafka output 2

I want to understand will kafka output 1 and kafka output 2 wait for all
operations to finish on B and C before sending an output, or will they
simply send an output as soon as the ops in B and C are done.

What kind of synchronization guarantees are there?

Thanks
Nipun