Restart Spark Streaming after deployment

[KhajaAsmath Mohammed <md...@gmail.com>]

Hi,

I am new in the usage of spark streaming. I have developed one spark
streaming job which runs every 30 minutes with checkpointing directory.

I have to implement minor change, shall I kill the spark streaming job once
the batch is completed using yarn application -kill command and update the
jar file?

Question I have is, if I follow the above approach will spark streaming
picks up data from offset saved in checkpoint after restart?

is there any other better approaches you have. Thanks in advance for your
suggestions.

Thanks,
Asmath

Re: Restart Spark Streaming after deployment

[Jacek Laskowski <ja...@japila.pl>]

Hi,

You're right...killing the spark streaming job is the way to go. If a batch
was completed successfully, Spark Streaming will recover from the
controlled failure and start where it left off. I don't think there's other
way to do it.

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

On Wed, Nov 15, 2017 at 5:18 PM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:

> Hi,
>
> I am new in the usage of spark streaming. I have developed one spark
> streaming job which runs every 30 minutes with checkpointing directory.
>
> I have to implement minor change, shall I kill the spark streaming job
> once the batch is completed using yarn application -kill command and update
> the jar file?
>
> Question I have is, if I follow the above approach will spark streaming
> picks up data from offset saved in checkpoint after restart?
>
> is there any other better approaches you have. Thanks in advance for your
> suggestions.
>
> Thanks,
> Asmath
>