You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Renjie Liu <li...@gmail.com> on 2015/10/19 08:58:10 UTC

Guaranteed processing orders of each batch in Spark Streaming

Hi, all:
I've read source code and it seems that there is no guarantee that the
order of processing of each RDD is guaranteed since jobs are just submitted
to a thread pool. I  believe that this is quite important in streaming
since updates should be ordered.

Re: Guaranteed processing orders of each batch in Spark Streaming

Posted by Renjie Liu <li...@gmail.com>.

Hi, all:
I have given a detailed description of my proposal in this jira
<https://issues.apache.org/jira/browse/SPARK-11308>.

On Mon, Oct 19, 2015 at 2:58 PM Renjie Liu <li...@gmail.com> wrote:

> Hi, all:
> I've read source code and it seems that there is no guarantee that the
> order of processing of each RDD is guaranteed since jobs are just submitted
> to a thread pool. I  believe that this is quite important in streaming
> since updates should be ordered.
>
> --
Liu, Renjie
Software Engineer, MVAD

Re: Guaranteed processing orders of each batch in Spark Streaming

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

I guess the order is guaranteed unless you set
the spark.streaming.concurrentJobs to a higher number than 1.

Thanks
Best Regards

On Mon, Oct 19, 2015 at 12:28 PM, Renjie Liu <li...@gmail.com>
wrote:

> Hi, all:
> I've read source code and it seems that there is no guarantee that the
> order of processing of each RDD is guaranteed since jobs are just submitted
> to a thread pool. I  believe that this is quite important in streaming
> since updates should be ordered.
>
>