You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Charles vinodh <mi...@gmail.com> on 2019/07/09 01:55:58 UTC

disable checkpointing in structured streaming

Hi ,

is it possible to disable checkpointing in structured streaming and have it
replaced by our own checkpointing implementation where the offsets are
saved in an external database?.. I looked up the docs and it seems this is
supported on spark DStream streaming but not in structured streaming for
some reason. Is that the case and if so why is this not possible with
structured streaming? is it possible to have this feature in the future?

Thanks,
Charles

Re: disable checkpointing in structured streaming

Posted by Charles vinodh <mi...@gmail.com>.

resurfacing this thread again..

On Mon, Jul 8, 2019 at 8:55 PM Charles vinodh <mi...@gmail.com> wrote:

> Hi ,
>
> is it possible to disable checkpointing in structured streaming and have
> it replaced by our own checkpointing implementation where the offsets are
> saved in an external database?.. I looked up the docs and it seems this is
> supported on spark DStream streaming but not in structured streaming for
> some reason. Is that the case and if so why is this not possible with
> structured streaming? is it possible to have this feature in the future?
>
> Thanks,
> Charles
>