You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Georg Heiler <ge...@gmail.com> on 2018/03/08 20:11:11 UTC

Upgrades of streaming jobs

Hi

What is the state of spark structured streaming jobs and upgrades?

Can checkpoints of version 1 be read by version 2 of a job? Is downtime
required to upgrade the job?

Thanks

Re: Upgrades of streaming jobs

Posted by Tathagata Das <ta...@gmail.com>.

Yes, all checkpoints are forward compatible.

However, you do need to restart the query if you want to update the code of
the query. This downtime can be in less than a second (if you just restart
the query without stopping the application/Spark driver) or 10s of seconds
(if you have to stop the application and resubmit your application to the
cluster).

On Thu, Mar 8, 2018 at 12:11 PM, Georg Heiler <ge...@gmail.com>
wrote:

> Hi
>
> What is the state of spark structured streaming jobs and upgrades?
>
> Can checkpoints of version 1 be read by version 2 of a job? Is downtime
> required to upgrade the job?
>
> Thanks
>