You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Prasad Mujumdar (Created) (JIRA)" <ji...@apache.org> on 2012/02/28 20:19:46 UTC
[jira] [Created] (FLUME-1008) Implement Hive sink
Implement Hive sink
-------------------
Key: FLUME-1008
URL: https://issues.apache.org/jira/browse/FLUME-1008
Project: Flume
Issue Type: Improvement
Components: Sinks+Sources
Affects Versions: v1.1.0
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
The HDFS sink's directory structure and bucketing is mostly sufficient for Hive tables. The bucketing could result into new directories getting created. This requrires altering Hive table to add a new partition for the directory.
This hive sink is basically to handle new directories getting added to a target table. The functionality discussed in FLUME-990 will be tracked as part of that ticket.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1008) Implement Hive sink
Posted by "E. Sammer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218787#comment-13218787 ]
E. Sammer commented on FLUME-1008:
----------------------------------
In 0.9.x there was the notion of a Hive sink as well. Since there's so much repeated code between what the HDFS sink does and what the Hive sink would do - and because I can imagine this being common - why not just create events and observers and have orthogonal plugins that can listen and react to events. For instance, the HDFSSink may support generating the events OnFileOpen, OnFileClose, OnAppend, etc.
I wrote a general purpose library to handle things like this at one point (we don't have to use it - it's just an example of what I'm talking about): https://github.com/esammer/observer
> Implement Hive sink
> -------------------
>
> Key: FLUME-1008
> URL: https://issues.apache.org/jira/browse/FLUME-1008
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
>
> The HDFS sink's directory structure and bucketing is mostly sufficient for Hive tables. The bucketing could result into new directories getting created. This requrires altering Hive table to add a new partition for the directory.
> This hive sink is basically to handle new directories getting added to a target table. The functionality discussed in FLUME-990 will be tracked as part of that ticket.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1008) Implement Hive sink
Posted by "Alexander Lorenz-Alten (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224227#comment-13224227 ]
Alexander Lorenz-Alten commented on FLUME-1008:
-----------------------------------------------
FLUME-990 duplicate
> Implement Hive sink
> -------------------
>
> Key: FLUME-1008
> URL: https://issues.apache.org/jira/browse/FLUME-1008
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
>
> The HDFS sink's directory structure and bucketing is mostly sufficient for Hive tables. The bucketing could result into new directories getting created. This requrires altering Hive table to add a new partition for the directory.
> This hive sink is basically to handle new directories getting added to a target table. The functionality discussed in FLUME-990 will be tracked as part of that ticket.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira