[jira] [Commented] (FLUME-2256) Generic JDBC Sink

["Lior Zeno (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349503#comment-15349503 ] 

Lior Zeno commented on FLUME-2256:
----------------------------------

I have one comment here, I don't think that the user should pass a SQL command (insert into), but should specify the target table and the headers to column mapping. The sink will use insert into when batch size is one, however, when the batch size is larger than one it may use COPY IN, for example.

[~karlj000], are you still available for this?

> Generic JDBC Sink
> -----------------
>
>                 Key: FLUME-2256
>                 URL: https://issues.apache.org/jira/browse/FLUME-2256
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: Jeremy Karlson
>            Assignee: Jeremy Karlson
>             Fix For: v1.8.0
>
>         Attachments: FLUME-2256.diff
>
>
> I've been working on a generic JDBC sink.  It needs a bit more testing, but I think it's ready for review and feedback.  I have not yet updated the Flume documentation, but I can / will if people are happy with this.
> Since the config file is how you’d interact with it, here’s a working example from my source tree:
> {code}
> a.sinks.k.type=jdbc
> a.sinks.k.channel=c
> a.sinks.k.driver=com.mysql.jdbc.Driver
> a.sinks.k.url=jdbc:mysql://localhost:8889/flume
> a.sinks.k.user=username
> a.sinks.k.password=password
> a.sinks.k.batchSize=100
> a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
> {code}
> The interesting part is the SQL statement.  You can put anything you want in there - it will get converted to a prepared statement on execution.  The Ant-ish tokens get parsed and replaced with parameters at startup.
> The tokens are three part.  For example, in:
> {code}
> ${body:string(UTF-8)}
> {code}
> The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”).  The second part ("string") is a type identifier that converts into an appropriate JDBC parameter.  The third part (“UTF-8") is a configuration string for that type, if needed.  As for types, so far I’ve defined:
> body: string (with optional charset encoding), bytearray
> header: string, long, int, float, double, date (with mandatory date format and optional timezone)
> Additionally, if none of those make you happy you can define you own parameter converters:
> {code}
> ${custom:com.company.foo.MyConverter(optionaltextconfig)}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)