[jira] [Created] (FLINK-31285) FileSource should support reading files in order

["Yaroslav Tkachenko (Jira)" <ji...@apache.org>]

Yaroslav Tkachenko created FLINK-31285:
------------------------------------------

             Summary: FileSource should support reading files in order
                 Key: FLINK-31285
                 URL: https://issues.apache.org/jira/browse/FLINK-31285
             Project: Flink
          Issue Type: New Feature
          Components: Connectors / FileSystem
    Affects Versions: 1.18.0
            Reporter: Yaroslav Tkachenko


Currently, Flink's *FileSource* uses *LocalityAwareSplitAssigner* as a default *FileSplitAssigner* and it doesn't guarantee any order. In many scenarios involving processing historical data, reading files in order can be a requirement, especially when using event-time processing. 

I believe a new FileSplitAssigner should be implemented that supports ordering. FileSourceBuilder should be extended to allow choosing a different FileSplitAssigner.

It's also clear that the files may not be read in _perfect_ order with parallelism > 1. However, in some cases, using parallelism of 1 might be fine.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)