[jira] [Commented] (FLINK-4329) Fix Streaming File Source Timestamps/Watermarks Handling

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415532#comment-15415532 ] 

ASF GitHub Bot commented on FLINK-4329:
---------------------------------------

GitHub user kl0u opened a pull request:

    https://github.com/apache/flink/pull/2350

    [FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kl0u/flink continuous_file_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2350
    
----
commit 6207a9f5da086d808331afe0e8caf0f03b3fabc5
Author: kl0u <kk...@gmail.com>
Date:   2016-08-09T12:11:45Z

    [FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling
    
    Now the ContinuousFileReaderOperator ignores the watermarks sent by
    the source function and emits its own watermarks in case we are
    opearating on Ingestion time. In addition, and for Ingestion time
    only, the reader also assigns the correct timestamps to the elements
    that it reads.

----


> Fix Streaming File Source Timestamps/Watermarks Handling
> --------------------------------------------------------
>
>                 Key: FLINK-4329
>                 URL: https://issues.apache.org/jira/browse/FLINK-4329
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 1.1.0
>            Reporter: Aljoscha Krettek
>            Assignee: Kostas Kloudas
>             Fix For: 1.1.1
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks, i.e. they are just passed through. This means that when the {{ContinuousFileMonitoringFunction}} closes and emits a {{Long.MAX_VALUE}} that watermark can "overtake" the records that are to be emitted in the {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness" setting in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion timestamps since it is not technically a source but looks like one to the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)