You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrei Dragomir (JIRA)" <ji...@apache.org> on 2015/09/03 10:19:46 UTC
[jira] [Commented] (SPARK-8630) Prevent from checkpointing
QueueInputDStream
[ https://issues.apache.org/jira/browse/SPARK-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728680#comment-14728680 ]
Andrei Dragomir commented on SPARK-8630:
----------------------------------------
Totally agree with [~asimjalis] comments; We have some integration tests set up that use QueueInputDStream, as well as a processing topology that has updateStateByKey (which implies checkpoints). This patch means that that test suite no longer runs. Is there a work around ? What is the preferred way to push test data into a spark job which has checkpointing ?
> Prevent from checkpointing QueueInputDStream
> --------------------------------------------
>
> Key: SPARK-8630
> URL: https://issues.apache.org/jira/browse/SPARK-8630
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
> Fix For: 1.4.1, 1.5.0
>
>
> It's better to prevent from checkpointing QueueInputDStream rather than failing the application when recovering `QueueInputDStream`, so that people can find the issue as soon as possible. See SPARK-8553 for example.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org