You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/10/15 09:20:00 UTC
[jira] [Commented] (FLUME-3223) Flume HDFS Sink should retry close
prior to performing a recoverLease
[ https://issues.apache.org/jira/browse/FLUME-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649953#comment-16649953 ]
ASF subversion and git services commented on FLUME-3223:
--------------------------------------------------------
Commit 327a43dbefab57dc5d2da3ca9b7bb36ae4f743b7 in flume's branch refs/heads/trunk from [~emajor]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=327a43d ]
FLUME-3223 Flume HDFS Sink should retry close prior recover lease
This is based on @mcsanady 's original pull request #202
I took the test changes from him but reworked the new feature implementation
since it failed some unit tests.
Previously when a close failed we immediately did a recover lease.
This PR introduces a background retry mechanism. It uses the already
existing "hdfs.closeTries" parameter. Unfortunately it has infinite retries
by default, that seems a bit too long for me.
I also did a minimal code clean up. The most important is that
HDFSWriter writer in BucketWriter became final. This is essential for later use
in inner classes. Only some testing solutions made it not final.
I reworked those to use the constructor.
This closes #229
This closes #202
Reviewers: Peter Turcsanyi, Ferenc Szabo
(Endre Major via Ferenc Szabo)
> Flume HDFS Sink should retry close prior to performing a recoverLease
> ---------------------------------------------------------------------
>
> Key: FLUME-3223
> URL: https://issues.apache.org/jira/browse/FLUME-3223
> Project: Flume
> Issue Type: Improvement
> Reporter: Miklos Csanady
> Assignee: Endre Major
> Priority: Major
>
> Flume HDFS Sink will currently only attempt to close a file once, and if that fails, it will move to recoverLease. This appears to be causing some issues.
> The recommendation is to modify the close behavior to retry the close a configurable number of times with a configurable interval in between before attempting the recoverLease.
> There currently exists a {{closeTries}} and {{retryInterval}} property, however this only retries the rename operation, not the close operation as seen in the documentation:
> {quote}The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org