[jira] [Updated] (SPARK-16338) Streaming driver running on standalone cluster mode with supervise goes into bad state when application is killed from the UI

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-16338:
---------------------------------
    Labels: bulk-closed  (was: )

> Streaming driver running on standalone cluster mode with supervise goes into bad state when application is killed from the UI
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16338
>                 URL: https://issues.apache.org/jira/browse/SPARK-16338
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy, DStreams, Web UI
>    Affects Versions: 1.6.1
>            Reporter: Rohit Agarwal
>            Priority: Major
>              Labels: bulk-closed
>         Attachments: error
>
>
> We are going to start using Spark Streaming in production and I was testing various failure scenarios. I noticed one case where the spark streaming driver got into a bad state.
> Steps to reproduce:
> 1. Create a spark streaming application with Direct Kafka Streams and checkpointing enabled.
> 2. Deploy the application to a spark standalone cluster. With cluster mode and --supervise.
> 3. Let it run for sometime.
> 4. Kill the application (but not the driver) from the Spark Master UI.
> 5. The driver keeps on running but doesn't restart the application. What's worse is that it keeps updating the checkpoint every batch duration, so when you do restart the driver, it starts at a later point and you have lost data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org