You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Yaroslav Tkachenko (Jira)" <ji...@apache.org> on 2023/02/01 21:21:00 UTC

[jira] [Created] (FLINK-30873) FileSystem Table API connector doesn't discover new files when partitions are specified

Yaroslav Tkachenko created FLINK-30873:
------------------------------------------

             Summary: FileSystem Table API connector doesn't discover new files when partitions are specified
                 Key: FLINK-30873
                 URL: https://issues.apache.org/jira/browse/FLINK-30873
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem
    Affects Versions: 1.16.1
            Reporter: Yaroslav Tkachenko


When *source.monitor-interval* is configured I expect the source connector to discover new files. However, if the source table was created with partitions (using PARTITIONED BY), it only discovers new files in the partition folders that existed during the startup; it doesn't discover new partitions. 

I believe the problem is [here|https://github.com/apache/flink/blob/5f2d088a2713ced5c6ce072db92f4378f73bc739/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSource.java#L276-L286], only these file paths are monitored internally, no partition discovery is currently implemented. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)