You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/12/12 05:20:58 UTC

[jira] [Commented] (SPARK-18826) Make FileStream be able to start with most recent files

    [ https://issues.apache.org/jira/browse/SPARK-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741011#comment-15741011 ] 

Apache Spark commented on SPARK-18826:
--------------------------------------

User 'zsxwing' has created a pull request for this issue:
https://github.com/apache/spark/pull/16251

> Make FileStream be able to start with most recent files
> -------------------------------------------------------
>
>                 Key: SPARK-18826
>                 URL: https://issues.apache.org/jira/browse/SPARK-18826
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>
> When starting a stream with a lot of backfill and maxFilesPerTrigger, the user could often want to start with most recent files first. This would let you keep low latency for recent data and slowly backfill historical data.
> It's better to add an option to control this behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org