[jira] [Created] (FLUME-2256) Generic JDBC Sink

["Jeremy Karlson (JIRA)" <ji...@apache.org>]

Jeremy Karlson created FLUME-2256:
-------------------------------------

             Summary: Generic JDBC Sink
                 Key: FLUME-2256
                 URL: https://issues.apache.org/jira/browse/FLUME-2256
             Project: Flume
          Issue Type: New Feature
            Reporter: Jeremy Karlson
            Assignee: Jeremy Karlson


I've been working on a generic JDBC sink.  It needs a bit more testing, but I think it's ready for review and feedback.  I have not yet updated the Flume documentation, but I can / will if people are happy with this.

Since the config file is how you’d interact with it, here’s a working example from my source tree:

a.sinks.k.type=jdbc
a.sinks.k.channel=c
a.sinks.k.driver=com.mysql.jdbc.Driver
a.sinks.k.url=jdbc:mysql://localhost:8889/flume
a.sinks.k.user=username
a.sinks.k.password=password
a.sinks.k.batchSize=100
a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})

The interesting part is the SQL statement.  You can put anything you want in there - it will get converted to a prepared statement on execution.  The Ant-ish tokens get parsed and replaced with parameters at startup.

The tokens are three part.  For example, in:

${body:string(UTF-8)}

The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”).  The second part ("string") is a type identifier that converts into an appropriate JDBC parameter.  The third part (“UTF-8") is a configuration string for that type, if needed.  As for types, so far I’ve defined:

body: string (with optional charset encoding), bytearray
header: string, long, int, float, double, date (with mandatory date format and optional timezone)

Additionally, if none of those make you happy you can define you own parameter converters:

${custom:com.company.foo.MyConverter(optionaltextconfig)}



--
This message was sent by Atlassian JIRA
(v6.1#6144)