You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2017/05/02 10:09:41 UTC
[GitHub] flink pull request #3807: [FLINK-6427] Ensure file length is flushed in Stre...
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/3807
[FLINK-6427] Ensure file length is flushed in StreamWriterBase
StreamWriterBase is used with BucketingSink. With HDFS, it can happen that the
NameNode does not update the file length if we flush but don't properly close
the file. We now specify that we also want to flush the file length when
flushing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink bucketing-sink-flush-file-length
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3807.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3807
----
commit dbc3899d5dbda642fdce0f06e9fece43dbc9585f
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2017-05-02T09:31:40Z
[FLINK-6427] Ensure file length is flushed in StreamWriterBase
StreamWriterBase is used with BucketingSink. With HDFS, it can happen that the
NameNode does not update the file length if we flush but don't properly close
the file. We now specify that we also want to flush the file length when
flushing.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3807: [FLINK-6427] Ensure file length is flushed in StreamWrite...
Posted by juergenthomann <gi...@git.apache.org>.
Github user juergenthomann commented on the issue:
https://github.com/apache/flink/pull/3807
It seems that DFSOutputStream is considered a private class (see Annotation). HdfsDataOutputStream also provides an hsync with flags and is considered public. It is also the place where the SyncFlags are provided.
I will try it with my mentioned small change and see if that solves our problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3807: [FLINK-6427] Ensure file length is flushed in StreamWrite...
Posted by juergenthomann <gi...@git.apache.org>.
Github user juergenthomann commented on the issue:
https://github.com/apache/flink/pull/3807
@aljoscha I'll create a new pull request in some minutes. In my case the valid-length and the size reported by the namenode are exactly the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3807: [FLINK-6427] Ensure file length is flushed in Stre...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/3807
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3807: [FLINK-6427] Ensure file length is flushed in StreamWrite...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/3807
@juergenthomann Cool, if you want I can also close this PR and wait for you to open a fixed (and verified) change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---