You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Bin Wang <wb...@gmail.com> on 2015/09/17 10:50:25 UTC
QueueStream doesn't support checkpoint makes it difficult to do unit test
I'm using spark streaming and use updateStateByKey, which forced to use
checkpoint. In my unit test, I create a queueStream to test. But in spark
1.5, QueueStream will throw an exception while use it with checkpoint, it
makes difficult to do unit test. Is there an option to disable this? Though
I know it will fail to recover from checkpoint but since it is a test I
don't care it.
I've found the git commit here
https://mail-archives.apache.org/mod_mbox/incubator-spark-commits/201506.mbox/%3C8efccd27016447fb8d1e0b3d9582ba78@git.apache.org%3E
Re: QueueStream doesn't support checkpoint makes it difficult to do
unit test
Posted by Bin Wang <wb...@gmail.com>.
Never mind. I've found a PR and it merged:
https://github.com/apache/spark/pull/8624/commits
Bin Wang <wb...@gmail.com>于2015年9月17日周四 下午4:50写道：
> I'm using spark streaming and use updateStateByKey, which forced to use
> checkpoint. In my unit test, I create a queueStream to test. But in spark
> 1.5, QueueStream will throw an exception while use it with checkpoint, it
> makes difficult to do unit test. Is there an option to disable this? Though
> I know it will fail to recover from checkpoint but since it is a test I
> don't care it.
>
> I've found the git commit here
> https://mail-archives.apache.org/mod_mbox/incubator-spark-commits/201506.mbox/%3C8efccd27016447fb8d1e0b3d9582ba78@git.apache.org%3E
>