You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by twalthr <gi...@git.apache.org> on 2018/04/04 08:44:34 UTC

[GitHub] flink pull request #5811: [FLINK-9113] [connectors] Fix flushing behavior of...

GitHub user twalthr opened a pull request:

    https://github.com/apache/flink/pull/5811

    [FLINK-9113] [connectors] Fix flushing behavior of bucketing sink for local filesystems

    ## What is the purpose of the change
    
    This PR changes the flushing behavior for HDFS' local filesystem abstraction. See also FLINK-9113 for more details.
    
    
    ## Brief change log
    
    - Use `hsync` for local filesystems
    - Add method to disable the new behavior
    - Additional check for verifying correct valid length files
    
    
    ## Verifying this change
    
    This fix is difficult to verify as it requires a OS process that is killed before syncing. I added a dedicated local filesystem test.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): no
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: no
      - The runtime per-record code paths (performance sensitive): yes
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
      - The S3 file system connector: no
    
    ## Documentation
    
      - Does this pull request introduce a new feature? no
      - If yes, how is the feature documented? JavaDocs


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/twalthr/flink FLINK-9113

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5811.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5811
    
----
commit 543f206f0e9e8415468f5d1092553754a8869fc7
Author: Timo Walther <tw...@...>
Date:   2018-04-04T08:29:57Z

    [FLINK-9113] [connectors] Fix flushing behavior of bucketing sink for local filesystems

----


---

[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:

    https://github.com/apache/flink/pull/5811
  
    @kl0u do you have an idea why `BucketingSinkTest` is failing? It seems that the written data parts are always empty.


---

[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:

    https://github.com/apache/flink/pull/5811
  
    Thanks for looking into it @kl0u. I observed the same behavior during debugging. I will remove the check for now and open a follow up issue. If there is no better solution, we might need to close the writer for checkpoints on local filesystems for preventing data loss in cases where the OS/machine goes down.


---

[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:

    https://github.com/apache/flink/pull/5811
  
    I will close this PR for now. Because it could not solve the general issue mentioned in HADOOP-7844.


---

[GitHub] flink pull request #5811: [FLINK-9113] [connectors] Fix flushing behavior of...

Posted by twalthr <gi...@git.apache.org>.
Github user twalthr closed the pull request at:

    https://github.com/apache/flink/pull/5811


---

[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/5811
  
    Well it seems like for these tests, the `flush` is not actually flushing. The files are there, the `validPartLength` is correct (=6 as we just write `test1\n`) but the data is not actually on disk. If you call `close()` on the in-progress file when snapshotting, then the tests succeed and the data is there.
    
    I would recommend to just remove the check for now, and open a followup JIRA that contains the check that you will remove, and also points on the discussion about HDFS not flushing, and we see how to proceed.
    
    I thing that the fact that the end-to-end tests pass point to the direction that sth is wrong with the FS abstraction.


---

[GitHub] flink pull request #5811: [FLINK-9113] [connectors] Fix flushing behavior of...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5811#discussion_r179743870
  
    --- Diff: flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java ---
    @@ -366,6 +384,11 @@ public void initializeState(FunctionInitializationContext context) throws Except
     			throw new RuntimeException("Error while creating FileSystem when initializing the state of the BucketingSink.", e);
     		}
     
    +		// sync on flush for local file systems
    +		if (localSyncOnFlush && (fs instanceof LocalFileSystem) && (writerTemplate instanceof StreamWriterBase)) {
    --- End diff --
    
    Shouldn't `(writerTemplate instanceof StreamWriterBase)` check be converted here into `checState(writerTemplate instanceof StreamWriterBase)` inside the if branch, and the same check be extracted and validated whenever user calls:
    `setWriter(...)` or `setLocalSyncOnFlush(...)`? So that non `StreamWriterBase` and `localSyncOnFlush = true` would be invalid configuration? Otherwise users might experience `wtf` moments when flag is being ignored after changing their writer.


---