[jira] [Commented] (FLINK-3296) DataStream.write*() methods are not flushing properly

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126351#comment-15126351 ] 

ASF GitHub Bot commented on FLINK-3296:
---------------------------------------

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1563#discussion_r51428065
  
    --- Diff: docs/apis/streaming/index.md ---
    @@ -1768,6 +1768,11 @@ greater than 1, the output will also be prepended with the identifier of the tas
     </div>
     </div>
     
    +Note that the `write*()` methods on `DataStream` are mainly intended for debugging purposes.
    +They are not participating in Flink's checkpointing (no fault tolerance guarantees). The 
    --- End diff --
    
    May be worth adding that this means usually "at-least-once" , but may also mean data loss in cases where the output formats buffer data and do not immediately persist it.


> DataStream.write*() methods are not flushing properly
> -----------------------------------------------------
>
>                 Key: FLINK-3296
>                 URL: https://issues.apache.org/jira/browse/FLINK-3296
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>            Priority: Critical
>
> The DataStream.write() methods rely on the {{FileSinkFunctionByMillis}} class, which has a logic for flushing records, even though the underlying stream is never flushed. This is misleading for users as files are not written as they would expect it.
> The code was initial written with FileOutputFormats in mind, but the types were not set correctly. This PR opened the write() method to any output format: https://github.com/apache/flink/pull/706/files



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)