You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Prasad Mujumdar (Created) (JIRA)" <ji...@apache.org> on 2011/12/05 22:00:40 UTC
[jira] [Created] (FLUME-871) HDFS sink needs to handle blocked/hung
append operation
HDFS sink needs to handle blocked/hung append operation
-------------------------------------------------------
Key: FLUME-871
URL: https://issues.apache.org/jira/browse/FLUME-871
Project: Flume
Issue Type: Improvement
Components: Sinks+Sources
Affects Versions: NG alpha 2
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
Fix For: NG alpha 3
The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-871) HDFS sink needs to handle blocked/hung
append operation
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-871:
----------------------------------
Attachment: Flume-871.patch.1
> HDFS sink needs to handle blocked/hung append operation
> -------------------------------------------------------
>
> Key: FLUME-871
> URL: https://issues.apache.org/jira/browse/FLUME-871
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: NG alpha 3
>
> Attachments: Flume-871.patch.1
>
>
> The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-871) HDFS sink needs to handle
blocked/hung append operation
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164912#comment-13164912 ]
Hudson commented on FLUME-871:
------------------------------
Integrated in flume-728 #70 (See [https://builds.apache.org/job/flume-728/70/])
Flume-871. Background HDFS append with configurable wait timeout
> HDFS sink needs to handle blocked/hung append operation
> -------------------------------------------------------
>
> Key: FLUME-871
> URL: https://issues.apache.org/jira/browse/FLUME-871
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: NG alpha 3
>
> Attachments: Flume-871.patch.1
>
>
> The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-871) HDFS sink needs to handle
blocked/hung append operation
Posted by "Prasad Mujumdar (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar resolved FLUME-871.
-----------------------------------
Resolution: Fixed
Patch committed to flume-728
> HDFS sink needs to handle blocked/hung append operation
> -------------------------------------------------------
>
> Key: FLUME-871
> URL: https://issues.apache.org/jira/browse/FLUME-871
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: NG alpha 3
>
> Attachments: Flume-871.patch.1
>
>
> The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-871) HDFS sink needs to handle
blocked/hung append operation
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164821#comment-13164821 ]
jiraposter@reviews.apache.org commented on FLUME-871:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3019/
-----------------------------------------------------------
(Updated 2011-12-07 23:12:09.859775)
Review request for Arvind Prabhakar and Eric Sammer.
Changes
-------
Corrected the Bug#
Summary
-------
The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
The patch includes executing append in a separate thread and wait for the completion with timeout if specified.
This addresses bug FLUME-871.
https://issues.apache.org/jira/browse/FLUME-871
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 556aa3d
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadDataStream.java 233e8c1
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 3f7a735
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 93b0b69
Diff: https://reviews.apache.org/r/3019/diff
Testing
-------
HDFS tests. Added new test cases to test the hung/slow sink operations
Thanks,
Prasad
> HDFS sink needs to handle blocked/hung append operation
> -------------------------------------------------------
>
> Key: FLUME-871
> URL: https://issues.apache.org/jira/browse/FLUME-871
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: NG alpha 3
>
> Attachments: Flume-871.patch.1
>
>
> The HDFS sink currently waits indefinitely for the append to complete. We need to support a configurable timeout for the append so that the flow can continue when a given operation is hung.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira