Message loss in streaming even with graceful shutdown

[Noorul Islam K M <no...@noorul.com>]

Hi all,

I have a streaming application with batch interval 10 seconds.

    val sparkConf = new SparkConf().setAppName("RMQWordCount")
      .set("spark.streaming.stopGracefullyOnShutdown", "true")
    val ssc = new StreamingContext(sparkConf, Seconds(10))

I also use reduceByKeyAndWindow() API for aggregation at window interval
of 5 minutes.

But when I send a SIGTERM to the streaming process at around 4th minute,
I don't see reduceByKeyAndWindow() action taking place. But the data is
already read for 4 minutes. I thought graceful shutdown would trigger
the action with received messages.

Am I missing something?

Thanks and regards
Noorul

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org