You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by William Wallace <th...@gmail.com> on 2025/07/15 10:25:06 UTC

FileSource enumerateSplits recursively call S3 list - high number of calls

We are using Flink File Connector to continuously scan s3 paths. We use
FileSource which uses NonSplittingRecursiveEnumerator to scan the s3 paths.
For each parent path, the enumerateSplits function will recursively call S3
list for each S3 “sub-directory”, which can result in a large number of
calls to S3 as the number of sub-directories grows. This can be excessively
slow versus calling the S3 ListObjectsV2 API directly on the parent path.


For example, if the parent path is /test, and there are 1000 subdirectories
under /test, this will result in 1000 calls to S3 versus 1 call to S3.


Could you let us know if this behavior is expected and in such case could
be optimized by reducing the high number of reads? This is a blocker for us.


Thank you.