File Source Limitations

[Kirti Dhar Upadhyay K via user <us...@flink.apache.org>]

Hi Community,

I am planning to use FileSource (with S3) in my application. Hence encountered with below limitations:
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/#current-limitations



  1.  Watermarking does not work very well for large backlogs of files. This is because watermarks eagerly advance within a file, and the next file might contain data later than the watermark.
Ques: Is there any ideal use case/settings/configurations where this problem does not come into picture? OR can be avoided?


  1.  For Unbounded File Sources, the enumerator currently remembers paths of all already processed files, which is a state that can, in some cases, grow rather large.
Ques: As a workaround of this problem, what if I configure a state backend (say RocksDBStateBackend) with some configured TTL, which shall automatically delete the older data. Is there any repercussions of this?


Regards,
Kirti Dhar