[jira] [Commented] (KAFKA-5046) Support file rotation in FileStreamSource Connector

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962203#comment-15962203 ] 

ASF GitHub Bot commented on KAFKA-5046:
---------------------------------------

GitHub user kkonstantine opened a pull request:

    https://github.com/apache/kafka/pull/2830

    KAFKA-5046: Support file rotation in FileStreamSource Connector

    - Detect file rotation/removal in FileStreamSource task.
    - Extend offset key to include creation time and fileKey attributes.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kkonstantine/kafka KAFKA-5046-Support-file-rotation-in-FileStreamSource-Connector

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/2830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2830
    
----
commit e9f062bc179c9308d683ea590ebcc90854d6c874
Author: Konstantine Karantasis <ko...@confluent.io>
Date:   2017-04-09T00:14:47Z

    KAFKA-5046: Support file rotation in FileStreamSource Connector
    
    - Detect file rotation/removal in FileStreamSource task.
    - Extend offset key to include creation time and fileKey attributes.

----


> Support file rotation in FileStreamSource Connector
> ---------------------------------------------------
>
>                 Key: KAFKA-5046
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5046
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.0
>            Reporter: Konstantine Karantasis
>            Assignee: Konstantine Karantasis
>            Priority: Minor
>
> Currently when a source file is moved (for file rotation purposes, or between restarts of Kafka Connect) the FileStreamSource Connector can not detect the change, because it only uses the filename as key to its offset tracking. 
> Nevertheless, file rotation can be detected easily by checking basic file attributes such as the {{fileKey}} in platforms that this attribute is supported (for instance file key includes the device id and the inode in unix based filesystems) and the file's creation time.
> Such checks need to take place when the task starts and when no more records are read during a call to {{poll}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)