You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "hailong wang (Jira)" <ji...@apache.org> on 2020/09/26 13:38:00 UTC

[jira] [Created] (FLINK-19425) Correct the usage of BulkWriter#flush and BulkWriter#finish

hailong wang created FLINK-19425:
------------------------------------

             Summary: Correct the usage of BulkWriter#flush and BulkWriter#finish
                 Key: FLINK-19425
                 URL: https://issues.apache.org/jira/browse/FLINK-19425
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Common
    Affects Versions: 1.11.0
            Reporter: hailong wang
             Fix For: 1.12.0, 1.11.0


From the comments, BulkWriter#finish method should flush all buffer before close.

But some subclasses of it do not flush data. These classes are as follows:

1.AvroBulkWriter#finish

2.HadoopCompressionBulkWriter#finish

3.NoCompressionBulkWriter#finish

4.SequenceFileWriter#finish

We should invoke BulkWriter#flush in this finish methods.

On the other hand, We don't have to  invoke BulkWriter#flush in close method. For BulkWriter#finish will flush all data.

1. HadoopPathBasedPartFileWriter#closeForCommit

2. BulkPartWriter#closeForCommit

3. FileSystemTableSink#OutputFormat#close

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)