You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2016/12/12 05:17:58 UTC

[jira] [Created] (SPARK-18826) Make FileStream be able to start with most recent files

Shixiong Zhu created SPARK-18826:
------------------------------------

             Summary: Make FileStream be able to start with most recent files
                 Key: SPARK-18826
                 URL: https://issues.apache.org/jira/browse/SPARK-18826
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
            Reporter: Shixiong Zhu
            Assignee: Shixiong Zhu


When starting a stream with a lot of backfill and maxFilesPerTrigger, the user could often want to start with most recent files first. This would let you keep low latency for recent data and slowly backfill historical data.

It's better to add an option to control this behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org