You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org> on 2012/02/21 22:52:50 UTC
[jira] [Commented] (FLUME-985) All HDFS Operations in HDFSEventSink
should have a timeout
[ https://issues.apache.org/jira/browse/FLUME-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212997#comment-13212997 ]
jiraposter@reviews.apache.org commented on FLUME-985:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3988/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
1) All HDFS actions are now done in async mode
2) If an HDFS append timesout, the file is closed and reopened.
3) Batching is now handled by BucketWriter which was always aware of the batch size.
This addresses bug FLUME-985.
https://issues.apache.org/jira/browse/FLUME-985
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
Diff: https://reviews.apache.org/r/3988/diff
Testing
-------
1) Unit tests were added for close/reopen scenario.
2) All unit tests pass
3) I manually verified this patch improved FlumeNG's behavior when the datanode it's writing to is restarted. In the past FlumeNG had to be restarted, now Flume moves on and starts writing to a new file.
Thanks,
Brock
> All HDFS Operations in HDFSEventSink should have a timeout
> ----------------------------------------------------------
>
> Key: FLUME-985
> URL: https://issues.apache.org/jira/browse/FLUME-985
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-985-0.patch
>
>
> In FLUME-871 appends were made asynchronous so we could time them out. All HDFS Operations should be done this same way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira