You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Subramanyam Satyanarayana (Created) (JIRA)" <ji...@apache.org> on 2011/12/16 23:44:33 UTC
[jira] [Created] (FLUME-888) Port Hbase Sink from OG
Port Hbase Sink from OG
------------------------
Key: FLUME-888
URL: https://issues.apache.org/jira/browse/FLUME-888
Project: Flume
Issue Type: New Feature
Components: Sinks+Sources
Reporter: Subramanyam Satyanarayana
Fix For: NG alpha 3
The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-888) Port Hbase Sink from OG
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199357#comment-13199357 ]
jiraposter@reviews.apache.org commented on FLUME-888:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3741/
-----------------------------------------------------------
(Updated 2012-02-02 23:24:02.518399)
Review request for Flume and Arvind Prabhakar.
Summary
-------
HBase Sink, ported to Flume NG.
Configurable EventToPutDecoder in place.
This addresses bug FLUME-888.
https://issues.apache.org/jira/browse/FLUME-888
Diffs
-----
flume-ng-sinks/flume-hbase-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/DefaultEventToPutDecoder.java PRE-CREATION
flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/EventToPutDecoder.java PRE-CREATION
flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseEventSink.java PRE-CREATION
pom.xml 2bf32bb
Diff: https://reviews.apache.org/r/3741/diff
Testing
-------
Tested with the DefaultEventToPutDecoder in place, to map Event-s to Put-s.
Thanks,
Karthik
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-888) Port Hbase Sink from OG
Posted by "Karthik K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik K updated FLUME-888:
----------------------------
Attachment: FLUME-888.patch
HBaseSink , takes care of pushing the 'Put's from Events downstream.
Pluggable - EventToPutDecoder , in place (with a Default* implementation) to play around with Event formats to convert them to 'Put's.
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-888) Port Hbase Sink from OG
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-888:
-----------------------------------
Affects Version/s: v1.0.0
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-888) Port Hbase Sink from OG
Posted by "Karthik K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197504#comment-13197504 ]
Karthik K commented on FLUME-888:
---------------------------------
We have a similar need , where the decoding of 'Event's w.r.t Put data might be a bit more non-trivial than the DefaultEventToPutDecoder , but that interface is still pluggable for us to reuse most of the code.
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-888) Port Hbase Sink from OG
Posted by "Karthik K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203799#comment-13203799 ]
Karthik K commented on FLUME-888:
---------------------------------
Review check, please ?
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-888) Port Hbase Sink from OG
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-888:
-----------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
HBase sink has gone into the trunk.
Resolving this issue as a duplicate. Please yell if that's not ok. Thanks.
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.2.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-888) Port Hbase Sink from OG
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206644#comment-13206644 ]
jiraposter@reviews.apache.org commented on FLUME-888:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3741/#review5041
-----------------------------------------------------------
Changes look good. Some feedback follows:
* Please remove trailing whitespaces and tabs where present. These are highlighted in red on the review board diff.
* The change is missing tests. Since this is an important feature enhancement, it is necessary to have some basic testing in place to assert that it works and continues to work.
* The implementation of the decoder depends upon pre-existing headers in the event. While this will work for planned flows that need to end up in HBase, it will not work that well for adhoc flows that the user wishes to route to HBase. In such cases, the requirement to have these headers in place will come in the way of making use of this. It is therefore preferable to have an alternate implementation that uses static configuration to place the events in HBase as well.
flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/DefaultEventToPutDecoder.java
<https://reviews.apache.org/r/3741/#comment11071>
The default implementation uses a few hardcoded header names. There should be a mechanism to override these via configuration if necessary.
- Arvind
On 2012-02-02 23:24:02, Karthik K wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3741/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-02 23:24:02)
bq.
bq.
bq. Review request for Flume and Arvind Prabhakar.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. HBase Sink, ported to Flume NG.
bq.
bq. Configurable EventToPutDecoder in place.
bq.
bq.
bq. This addresses bug FLUME-888.
bq. https://issues.apache.org/jira/browse/FLUME-888
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hbase-sink/pom.xml PRE-CREATION
bq. flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/DefaultEventToPutDecoder.java PRE-CREATION
bq. flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/EventToPutDecoder.java PRE-CREATION
bq. flume-ng-sinks/flume-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseEventSink.java PRE-CREATION
bq. pom.xml 2bf32bb
bq.
bq. Diff: https://reviews.apache.org/r/3741/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tested with the DefaultEventToPutDecoder in place, to map Event-s to Put-s.
bq.
bq.
bq. Thanks,
bq.
bq. Karthik
bq.
bq.
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-888) Port Hbase Sink from OG
Posted by "Karthik K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik K updated FLUME-888:
----------------------------
Status: Patch Available (was: Open)
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Attachments: FLUME-888.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-888) Port Hbase Sink from OG
Posted by "Arvind Prabhakar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197131#comment-13197131 ]
Arvind Prabhakar commented on FLUME-888:
----------------------------------------
Hi Subramanyam,
Any updates on this issue yet?
Thanks,
Arvind
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: v1.1.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-888) Port Hbase Sink from OG
Posted by "Arvind Prabhakar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar reassigned FLUME-888:
--------------------------------------
Assignee: Subramanyam Satyanarayana
> Port Hbase Sink from OG
> ------------------------
>
> Key: FLUME-888
> URL: https://issues.apache.org/jira/browse/FLUME-888
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Subramanyam Satyanarayana
> Assignee: Subramanyam Satyanarayana
> Fix For: NG alpha 3
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Flume NG implementation does not have a hbase sink implemented. A good starting point would be to port the hbase sink from the Flume OG branch, This came out of the flume hackathlon at Cloudera on 12/16/11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira