You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2017/06/02 21:20:04 UTC
[jira] [Commented] (SPARK-20002) Add support for unions between
streaming and batch datasets
[ https://issues.apache.org/jira/browse/SPARK-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035441#comment-16035441 ]
Michael Armbrust commented on SPARK-20002:
------------------------------------------
I'm not sure that we will ever support this. The issue is that for batch datasets, we don't track what has been read. Thus its unclear what should happen when the query is restarted. Instead, I think you can always achieve the same result by just loading both datasets as a stream (even if you don't plan to change one of them). Would that work?
> Add support for unions between streaming and batch datasets
> -----------------------------------------------------------
>
> Key: SPARK-20002
> URL: https://issues.apache.org/jira/browse/SPARK-20002
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Structured Streaming
> Affects Versions: 2.0.2
> Reporter: Leon Pham
>
> Currently unions between streaming datasets and batch datasets are not supported.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org