You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rani Yaroshinski <ra...@gmail.com> on 2015/11/07 10:17:47 UTC
Hbase Sink
Hi,
It seems a little weird, but it looks like the most simple sink you would
like for hbase,
which allows you to set key, and columns generally is missing.
You need to write specific custom serializer for any implementation which
would make
a sense for any production system.
Anyone with pointers to rellevant custom implementations which allows
configuration for key
type and value, and also for column family data.
Regards,
Rani
Re: Hbase Sink
Posted by Gonzalo Herreros <gh...@gmail.com>.
Sent without finishing the message:
I started by extending RegexHbaseEventSerializer, so I didn't have to
implement the full HbaseEventSerializer
Override (and call parent) configure(Context context) to get your custom
configuration properties
In
public void initialize(Event event, byte[] columnFamily)
you get the event and extract the data
In
public List<Row> getActions() throws FlumeException
you generate the HBase Put actions.
Regards,
Gonzalo
On 10 November 2015 at 14:55, Gonzalo Herreros <gh...@gmail.com> wrote:
> I started by extending RegexHbaseEventSerializer, so I didn't have to
> implement the full HbaseEventSerializer
>
> Override (and call parent) configure(Context context) to get your custom
> configuration properties
>
> In
>
>
> public List<Row> getActions() throws FlumeException
>
> where you conver the
>
> On 7 November 2015 at 11:07, Rani Yaroshinski <ra...@gmail.com>
> wrote:
>
>> Any pointers to the code, as sample ?
>>
>> On Sat, Nov 7, 2015 at 12:45 PM, Gonzalo Herreros <gh...@gmail.com>
>> wrote:
>>
>>> I did a custom serializer that parses the event an json and the top
>>> level properties become columns inside a configurable column family.
>>> I also have a custom property to configure which fields make up the
>>> composite key (which I salt based on the number of regions).
>>>
>>> It shouldn't be too hard having a property that maps input fields into
>>> hbase cf:column names (kinda like Hive does when you map an HBase table)
>>>
>>> Regards,
>>> Gonzalo
>>>
>>> On 7 November 2015 at 09:17, Rani Yaroshinski <
>>> rani.yaroshinski@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems a little weird, but it looks like the most simple sink you
>>>> would like for hbase,
>>>> which allows you to set key, and columns generally is missing.
>>>> You need to write specific custom serializer for any implementation
>>>> which would make
>>>> a sense for any production system.
>>>> Anyone with pointers to rellevant custom implementations which allows
>>>> configuration for key
>>>> type and value, and also for column family data.
>>>>
>>>> Regards,
>>>> Rani
>>>>
>>>
>>>
>>
>
Re: Hbase Sink
Posted by Gonzalo Herreros <gh...@gmail.com>.
I started by extending RegexHbaseEventSerializer, so I didn't have to
implement the full HbaseEventSerializer
Override (and call parent) configure(Context context) to get your custom
configuration properties
In
public List<Row> getActions() throws FlumeException
where you conver the
On 7 November 2015 at 11:07, Rani Yaroshinski <ra...@gmail.com>
wrote:
> Any pointers to the code, as sample ?
>
> On Sat, Nov 7, 2015 at 12:45 PM, Gonzalo Herreros <gh...@gmail.com>
> wrote:
>
>> I did a custom serializer that parses the event an json and the top level
>> properties become columns inside a configurable column family.
>> I also have a custom property to configure which fields make up the
>> composite key (which I salt based on the number of regions).
>>
>> It shouldn't be too hard having a property that maps input fields into
>> hbase cf:column names (kinda like Hive does when you map an HBase table)
>>
>> Regards,
>> Gonzalo
>>
>> On 7 November 2015 at 09:17, Rani Yaroshinski <rani.yaroshinski@gmail.com
>> > wrote:
>>
>>> Hi,
>>>
>>> It seems a little weird, but it looks like the most simple sink you
>>> would like for hbase,
>>> which allows you to set key, and columns generally is missing.
>>> You need to write specific custom serializer for any implementation
>>> which would make
>>> a sense for any production system.
>>> Anyone with pointers to rellevant custom implementations which allows
>>> configuration for key
>>> type and value, and also for column family data.
>>>
>>> Regards,
>>> Rani
>>>
>>
>>
>
Re: Hbase Sink
Posted by Rani Yaroshinski <ra...@gmail.com>.
Any pointers to the code, as sample ?
On Sat, Nov 7, 2015 at 12:45 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> I did a custom serializer that parses the event an json and the top level
> properties become columns inside a configurable column family.
> I also have a custom property to configure which fields make up the
> composite key (which I salt based on the number of regions).
>
> It shouldn't be too hard having a property that maps input fields into
> hbase cf:column names (kinda like Hive does when you map an HBase table)
>
> Regards,
> Gonzalo
>
> On 7 November 2015 at 09:17, Rani Yaroshinski <ra...@gmail.com>
> wrote:
>
>> Hi,
>>
>> It seems a little weird, but it looks like the most simple sink you would
>> like for hbase,
>> which allows you to set key, and columns generally is missing.
>> You need to write specific custom serializer for any implementation which
>> would make
>> a sense for any production system.
>> Anyone with pointers to rellevant custom implementations which allows
>> configuration for key
>> type and value, and also for column family data.
>>
>> Regards,
>> Rani
>>
>
>
Re: Hbase Sink
Posted by Gonzalo Herreros <gh...@gmail.com>.
I did a custom serializer that parses the event an json and the top level
properties become columns inside a configurable column family.
I also have a custom property to configure which fields make up the
composite key (which I salt based on the number of regions).
It shouldn't be too hard having a property that maps input fields into
hbase cf:column names (kinda like Hive does when you map an HBase table)
Regards,
Gonzalo
On 7 November 2015 at 09:17, Rani Yaroshinski <ra...@gmail.com>
wrote:
> Hi,
>
> It seems a little weird, but it looks like the most simple sink you would
> like for hbase,
> which allows you to set key, and columns generally is missing.
> You need to write specific custom serializer for any implementation which
> would make
> a sense for any production system.
> Anyone with pointers to rellevant custom implementations which allows
> configuration for key
> type and value, and also for column family data.
>
> Regards,
> Rani
>