[jira] [Created] (SPARK-20052) Some InputDStream needs closing processing after all batches processed when graceful shutdown

["Sasaki Toru (JIRA)" <ji...@apache.org>]

Sasaki Toru created SPARK-20052:
-----------------------------------

             Summary: Some InputDStream needs closing processing after all batches processed when graceful shutdown
                 Key: SPARK-20052
                 URL: https://issues.apache.org/jira/browse/SPARK-20052
             Project: Spark
          Issue Type: Improvement
          Components: DStreams
    Affects Versions: 2.2.0
            Reporter: Sasaki Toru


Some class extend InputDStream needs closing processing after all batches processed when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to Kafka Broker)

InputDStream has method 'stop' to stop receiving data, but this method will be called before processing last batches generated for graceful shutdown.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org