You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Timothy Farkas <ti...@gmail.com> on 2016/02/26 01:23:31 UTC

JDBC Streaming Connector

Hi All,

I'm new to Flink and was looking for a JDBC stream sink connector, and
didn't see one in flink-streaming-connectors. Is there one somewhere else,
or is there one currently in development. If not could I pick up a ticket
to add it?

Thanks,
Tim

Re: JDBC Streaming Connector

Posted by Timothy Farkas <ti...@gmail.com>.

Thanks everyone. I'll take a look at JDBCOutputFormat, and FLINK-3332
<https://issues.apache.org/jira/browse/FLINK-3332> along with other related
tickets.

Tim

On Fri, Feb 26, 2016 at 1:55 AM, Chesnay Schepler <c....@web.de> wrote:

> I'm actually working on a JDBC streaming connector with transactions.
>
>
> On 26.02.2016 10:33, Stephan Ewen wrote:
>
>> Hi Timothy!
>>
>> I would first just write a stream data sink based on the JDBCOutputFormat,
>> and then let's look into making it exactly-once.
>>
>> For JDBC with support for transactions, the basic idea would be to start a
>> transaction, insert elements, commit when a checkpoint is confirmed, and
>> start a new transaction.
>>
>> On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>> Hi Timothy,
>>>
>>> you can use the `JDBCOutputFormat` in combination with the
>>> `DataStream.writeUsingOutputFormat` method. However, this won't give you
>>> exactly once guarantees since the output formats don't take part in the
>>> checkpointing mechanism.
>>>
>>> Currently, Chesnay is working on a generic framework for almost exactly
>>> once sinks. Maybe you can sync with him. You'll find the corresponding
>>> JIRA
>>> ticket here [1].
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-3332
>>>
>>> Cheers,
>>> Till
>>>
>>>
>>> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas <
>>> timothytiborfarkas@gmail.com> wrote:
>>>
>>> Hi All,
>>>>
>>>> I'm new to Flink and was looking for a JDBC stream sink connector, and
>>>> didn't see one in flink-streaming-connectors. Is there one somewhere
>>>>
>>> else,
>>>
>>>> or is there one currently in development. If not could I pick up a
>>>> ticket
>>>> to add it?
>>>>
>>>> Thanks,
>>>> Tim
>>>>
>>>>
>

Re: JDBC Streaming Connector

Posted by Chesnay Schepler <c....@web.de>.

I'm actually working on a JDBC streaming connector with transactions.

On 26.02.2016 10:33, Stephan Ewen wrote:
> Hi Timothy!
>
> I would first just write a stream data sink based on the JDBCOutputFormat,
> and then let's look into making it exactly-once.
>
> For JDBC with support for transactions, the basic idea would be to start a
> transaction, insert elements, commit when a checkpoint is confirmed, and
> start a new transaction.
>
> On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Timothy,
>>
>> you can use the `JDBCOutputFormat` in combination with the
>> `DataStream.writeUsingOutputFormat` method. However, this won't give you
>> exactly once guarantees since the output formats don't take part in the
>> checkpointing mechanism.
>>
>> Currently, Chesnay is working on a generic framework for almost exactly
>> once sinks. Maybe you can sync with him. You'll find the corresponding JIRA
>> ticket here [1].
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-3332
>>
>> Cheers,
>> Till
>>
>>
>> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas <
>> timothytiborfarkas@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I'm new to Flink and was looking for a JDBC stream sink connector, and
>>> didn't see one in flink-streaming-connectors. Is there one somewhere
>> else,
>>> or is there one currently in development. If not could I pick up a ticket
>>> to add it?
>>>
>>> Thanks,
>>> Tim
>>>

Re: JDBC Streaming Connector

Posted by Stephan Ewen <se...@apache.org>.

Hi Timothy!

I would first just write a stream data sink based on the JDBCOutputFormat,
and then let's look into making it exactly-once.

For JDBC with support for transactions, the basic idea would be to start a
transaction, insert elements, commit when a checkpoint is confirmed, and
start a new transaction.

On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <tr...@apache.org>
wrote:

> Hi Timothy,
>
> you can use the `JDBCOutputFormat` in combination with the
> `DataStream.writeUsingOutputFormat` method. However, this won't give you
> exactly once guarantees since the output formats don't take part in the
> checkpointing mechanism.
>
> Currently, Chesnay is working on a generic framework for almost exactly
> once sinks. Maybe you can sync with him. You'll find the corresponding JIRA
> ticket here [1].
>
> [1] https://issues.apache.org/jira/browse/FLINK-3332
>
> Cheers,
> Till
>
>
> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas <
> timothytiborfarkas@gmail.com> wrote:
>
> > Hi All,
> >
> > I'm new to Flink and was looking for a JDBC stream sink connector, and
> > didn't see one in flink-streaming-connectors. Is there one somewhere
> else,
> > or is there one currently in development. If not could I pick up a ticket
> > to add it?
> >
> > Thanks,
> > Tim
> >
>

Re: JDBC Streaming Connector

Posted by Till Rohrmann <tr...@apache.org>.

Hi Timothy,

you can use the `JDBCOutputFormat` in combination with the
`DataStream.writeUsingOutputFormat` method. However, this won't give you
exactly once guarantees since the output formats don't take part in the
checkpointing mechanism.

Currently, Chesnay is working on a generic framework for almost exactly
once sinks. Maybe you can sync with him. You'll find the corresponding JIRA
ticket here [1].

[1] https://issues.apache.org/jira/browse/FLINK-3332

Cheers,
Till

On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas <
timothytiborfarkas@gmail.com> wrote:

> Hi All,
>
> I'm new to Flink and was looking for a JDBC stream sink connector, and
> didn't see one in flink-streaming-connectors. Is there one somewhere else,
> or is there one currently in development. If not could I pick up a ticket
> to add it?
>
> Thanks,
> Tim
>