You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rohit Agarwal (JIRA)" <ji...@apache.org> on 2016/06/30 22:29:10 UTC
[jira] [Created] (SPARK-16338) Streaming driver running on
standalone cluster mode with supervise goes into bad state when application
is killed from the UI
Rohit Agarwal created SPARK-16338:
-------------------------------------
Summary: Streaming driver running on standalone cluster mode with supervise goes into bad state when application is killed from the UI
Key: SPARK-16338
URL: https://issues.apache.org/jira/browse/SPARK-16338
Project: Spark
Issue Type: Bug
Components: Deploy, Streaming, Web UI
Affects Versions: 1.6.1
Reporter: Rohit Agarwal
We are going to start using Spark Streaming in production and I was testing various failure scenarios. I noticed one case where the spark streaming driver got into a bad state.
Steps to reproduce:
1. Create a spark streaming application with Direct Kafka Streams and checkpointing enabled.
2. Deploy the application to a spark standalone cluster. With cluster mode and --supervise.
3. Let it run for sometime.
4. Kill the application (but not the driver) from the Spark Master UI.
5. The driver keeps on running but doesn't restart the application. What's worse is that it keeps updating the checkpoint every batch duration, so when you do restart the driver, it starts at a later point and you have lost data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org