Customr Sink with bulk processing

[Mark Bramnik <ma...@gmail.com>]

Hi, I'm a newbie in a flume world, and I have a question about the custom
sink.

I would like to use flume process to gather log files and push them into
the database.
I know that I have to implement the sink because we're interested in very
specific database with a very specific table structure which is not
supported by flume.

So, I need to create a custom Sink which is well explained in flume
documentation. However I would like to process the events from the channel
in bulk (use bulk insert into db).

What is the best approach to implement such a code?
I'm not sure how to implement this logic given the fact that the event
taken from the channel can't be put back/we don't have something like
peek() in the channel.

Thanks a lot in advance and have a nice day :)

Mark

Re: Customr Sink with bulk processing

[Roshan Naik <ro...@hortonworks.com>]

A large batchSize on sink is not good enough ?
-roshan


On Thu, Jul 17, 2014 at 8:43 AM, Mark Bramnik <ma...@gmail.com>
wrote:

> Hi, I'm a newbie in a flume world, and I have a question about the custom
> sink.
>
> I would like to use flume process to gather log files and push them into
> the database.
> I know that I have to implement the sink because we're interested in very
> specific database with a very specific table structure which is not
> supported by flume.
>
> So, I need to create a custom Sink which is well explained in flume
> documentation. However I would like to process the events from the channel
> in bulk (use bulk insert into db).
>
> What is the best approach to implement such a code?
> I'm not sure how to implement this logic given the fact that the event
> taken from the channel can't be put back/we don't have something like
> peek() in the channel.
>
> Thanks a lot in advance and have a nice day :)
>
> Mark
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.