Spark streaming concurrent job scheduling question

[Renyi Xiong <re...@gmail.com>]

Hi,

I am trying to run an I/O intensive RDD in parallel with CPU intensive RDD
within an application through a window like below:

var ssc = new StreamingContext(sc, 1min);
var ds1 = ...
var ds2 = ds1.Window(2min).ForeachRDD(...)
ds1.ForeachRDD(...)

I hope ds1 to start its job at 1min interval even if ds2's job not complete
yet.

but it is not the case when I run it - ds1's job won't start until ds2's
job completes.

I looked into document which mentions jobs within same SparkContext need to
be submitted in different thread in order to run in parallel.

is it true? then question becomes how should I submit the above 2 jobs in
different threads?

(I know there're concurrentJobs and receiver mode but both with some
particular issues)

Thanks a lot,
Renyi.