[jira] [Created] (KAFKA-5046) Support file rotation in FileStreamSource Connector

["Konstantine Karantasis (JIRA)" <ji...@apache.org>]

Konstantine Karantasis created KAFKA-5046:
---------------------------------------------

             Summary: Support file rotation in FileStreamSource Connector
                 Key: KAFKA-5046
                 URL: https://issues.apache.org/jira/browse/KAFKA-5046
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
    Affects Versions: 0.10.2.0
            Reporter: Konstantine Karantasis
            Assignee: Konstantine Karantasis
            Priority: Minor


Currently when a source file is moved (for file rotation purposes, or between restarts of Kafka Connect) the FileStreamSource Connector can not detect the change, because it only uses the filename as key to its offset tracking. 

Nevertheless, file rotation can be detected easily by checking basic file attributes such as the {{fileKey}} in platforms that this attribute is supported (for instance file key includes the device id and the inode in unix based filesystems) and the file's creation time.

Such checks need to take place when the task starts and when no more records are read during a call to {{poll}}.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)