You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sasaki Toru (JIRA)" <ji...@apache.org> on 2017/03/22 01:27:42 UTC
[jira] [Updated] (SPARK-20052) Some InputDStream needs closing
processing after all batches processed when graceful shutdown
[ https://issues.apache.org/jira/browse/SPARK-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sasaki Toru updated SPARK-20052:
--------------------------------
Description:
Some class extend InputDStream needs closing processing after processing all batches when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to Kafka Broker)
InputDStream has method 'stop' to stop receiving data, but this method will be called before processing last batches generated for graceful shutdown.
was:
Some class extend InputDStream needs closing processing after all batches processed when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to Kafka Broker)
InputDStream has method 'stop' to stop receiving data, but this method will be called before processing last batches generated for graceful shutdown.
> Some InputDStream needs closing processing after all batches processed when graceful shutdown
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-20052
> URL: https://issues.apache.org/jira/browse/SPARK-20052
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.2.0
> Reporter: Sasaki Toru
>
> Some class extend InputDStream needs closing processing after processing all batches when graceful shutdown enabled.
> (e.g. When using Kafka as data source, need to commit processed offsets to Kafka Broker)
> InputDStream has method 'stop' to stop receiving data, but this method will be called before processing last batches generated for graceful shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org