You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "E. Sammer (JIRA)" <ji...@apache.org> on 2011/09/07 17:16:11 UTC
[jira] [Created] (FLUME-761) Implement HDFS Flume NG sink
Implement HDFS Flume NG sink
----------------------------
Key: FLUME-761
URL: https://issues.apache.org/jira/browse/FLUME-761
Project: Flume
Issue Type: Sub-task
Reporter: E. Sammer
Assignee: E. Sammer
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Attachment: FLUME-761.patch.4
HDFS sink with roll, batch and bucketing support.
compression part is not fully tested yet ..
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer reassigned FLUME-761:
-------------------------------
Assignee: Prasad Mujumdar (was: E. Sammer)
Assigning to Prasad.
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Attachment: FLUME-761.patch.2
First cut with tests
Limited functionality, no bucketing support yet.
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Attachment: FLUME-761.patch.5
Compression and parametric serializer interface
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4, FLUME-761.patch.5
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Attachment: FLUME-761.patch.4
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer updated FLUME-761:
----------------------------
Affects Version/s: NG
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Affects Versions: NG
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4, FLUME-761.patch.5
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Status: Patch Available (was: In Progress)
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on FLUME-761 started by Prasad Mujumdar.
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099057#comment-13099057 ]
E. Sammer commented on FLUME-761:
---------------------------------
Port the Flume HDFS sink functionality over to Flume NG.
The interesting features are file rotation, output bucketing, and support for append (flush).
A minimal implementation would support file rotation. Rotation should be configurable based on both time interval (specified in seconds) and size. Ideally, we do not create files unless there are events output (i.e. lazy file creation). It should be possible to specify rotation for time and size together, meaning rotate on whichever happens first.
Output bucketing is a feature support by Flume today that allows interpolation of event attributes in output paths. For instance, an output path of /logs/%{year}/%{month}/%{day}/ should become /logs/2011/01/01/ for an event with the atributes year=2011, month=01, day=01. This implies we must keep multiple writers open concurrently, each with separate bookkeeping on rotation time and output size.
Support for append should be orthogonal to file rotation. In other words we should still allow the user to specify a rotation policy (time and size) but we should call flush with a given frequency, probably specified in terms of the number of events. A fully durable configuration would flush after each event (i.e. flushInterval=1). We should only enable append support if the underlying HDFS install supports it. If the user specifies a flush policy and HDFS doesn't support append, we should warn, but continue.
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: E. Sammer
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer updated FLUME-761:
----------------------------
Resolution: Fixed
Fix Version/s: NG
Status: Resolved (was: Patch Available)
Committed to the flume-728 branch. Thanks Prasad!
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Affects Versions: NG
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Fix For: NG
>
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4, FLUME-761.patch.5
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer closed FLUME-761.
---------------------------
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Affects Versions: NG
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Fix For: NG
>
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4, FLUME-761.patch.5
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-761) Implement HDFS Flume NG sink
Posted by "E. Sammer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120534#comment-13120534 ]
E. Sammer commented on FLUME-761:
---------------------------------
Committed to flume-728 branch with modifications.
* Renamed classes to be camel case.
* Moved HDFS* related classes into o.a.flume.sink.hdfs.
* Added Hadoop as a dependency to the pom so we properly build / test.
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1, FLUME-761.patch.2, FLUME-761.patch.4, FLUME-761.patch.4, FLUME-761.patch.5
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-761) Implement HDFS Flume NG sink
Posted by "Prasad Mujumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-761:
----------------------------------
Attachment: FLUME-761.patch.1
First cut.
Limited functionality. not fully tested. Additional changes and tests will follow
> Implement HDFS Flume NG sink
> ----------------------------
>
> Key: FLUME-761
> URL: https://issues.apache.org/jira/browse/FLUME-761
> Project: Flume
> Issue Type: Sub-task
> Components: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
> Reporter: E. Sammer
> Assignee: Prasad Mujumdar
> Attachments: FLUME-761.patch.1
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira