You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Terry Hoo <hu...@gmail.com> on 2018/03/12 04:54:29 UTC

The last successful batch before stop re-execute after restart the DStreams with checkpoint

Experts,

I see the last batch before stop (graceful shutdown) always re-execute
after restart the DStream from a checkpoint, is this a expected behavior?

I see a bug in JIRA: https://issues.apache.org/jira/browse/SPARK-20050,
whic reports duplicates on Kafka, I also see this with HDFS file.

Regards
- Terry