You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jeremy Karlson (JIRA)" <ji...@apache.org> on 2013/11/28 08:07:36 UTC
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Karlson updated FLUME-2256:
----------------------------------
Description:
I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this.
Since the config file is how you’d interact with it, here’s a working example from my source tree:
{code}
a.sinks.k.type=jdbc
a.sinks.k.channel=c
a.sinks.k.driver=com.mysql.jdbc.Driver
a.sinks.k.url=jdbc:mysql://localhost:8889/flume
a.sinks.k.user=username
a.sinks.k.password=password
a.sinks.k.batchSize=100
a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
{code}
The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup.
The tokens are three part. For example, in:
{code}
${body:string(UTF-8)}
{code}
The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined:
body: string (with optional charset encoding), bytearray
header: string, long, int, float, double, date (with mandatory date format and optional timezone)
Additionally, if none of those make you happy you can define you own parameter converters:
{code}
${custom:com.company.foo.MyConverter(optionaltextconfig)}
{code}
was:
I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this.
Since the config file is how you’d interact with it, here’s a working example from my source tree:
a.sinks.k.type=jdbc
a.sinks.k.channel=c
a.sinks.k.driver=com.mysql.jdbc.Driver
a.sinks.k.url=jdbc:mysql://localhost:8889/flume
a.sinks.k.user=username
a.sinks.k.password=password
a.sinks.k.batchSize=100
a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup.
The tokens are three part. For example, in:
${body:string(UTF-8)}
The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined:
body: string (with optional charset encoding), bytearray
header: string, long, int, float, double, date (with mandatory date format and optional timezone)
Additionally, if none of those make you happy you can define you own parameter converters:
${custom:com.company.foo.MyConverter(optionaltextconfig)}
> Generic JDBC Sink
> -----------------
>
> Key: FLUME-2256
> URL: https://issues.apache.org/jira/browse/FLUME-2256
> Project: Flume
> Issue Type: New Feature
> Reporter: Jeremy Karlson
> Assignee: Jeremy Karlson
>
> I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this.
> Since the config file is how you’d interact with it, here’s a working example from my source tree:
> {code}
> a.sinks.k.type=jdbc
> a.sinks.k.channel=c
> a.sinks.k.driver=com.mysql.jdbc.Driver
> a.sinks.k.url=jdbc:mysql://localhost:8889/flume
> a.sinks.k.user=username
> a.sinks.k.password=password
> a.sinks.k.batchSize=100
> a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
> {code}
> The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup.
> The tokens are three part. For example, in:
> {code}
> ${body:string(UTF-8)}
> {code}
> The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined:
> body: string (with optional charset encoding), bytearray
> header: string, long, int, float, double, date (with mandatory date format and optional timezone)
> Additionally, if none of those make you happy you can define you own parameter converters:
> {code}
> ${custom:com.company.foo.MyConverter(optionaltextconfig)}
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)