You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Jocean shi <jo...@gmail.com> on 2018/11/21 10:07:32 UTC

Can JDBCSinkFunction support exectly once?

Hi,
Can JDBCSinkFunction support exectly once? Is it that The JDBCSinkFunction
dont't implement CheckpointListener meaning JDBCSinkFunction dont't support
exectly once?

cheers

Jocean

Re: Can JDBCSinkFunction support exectly once?

Posted by Jocean shi <jo...@gmail.com>.

Hi,
Thanks for help.
I find that many sink dont't support stream exectly-once. I need use
exectly-once Hbase sink and Mysql sink in my work.
I try to contribute to Hbase Sink firstly

Best Regards,
Jocean

Fabian Hueske <fh...@gmail.com> 于2018年11月21日周三 下午6:34写道：

> Hi,
>
> JDBCSinkFunction is a simple wrapper around the JDBCOutputFormat (the
> DataSet / Batch API output interface).
> Dominik is right, that JDBCSinkFunction does not support exactly-once
> output.
>
> It is not strictly required that an exactly-once sink implements
> TwoPhaseCommitFunction.
> TPCF is a convenience interface that internally use the CheckpointListener
> interface. It makes it easier to implement exactly-once sinks but is not
> the only way to do it.
>
> Implementing an exactly-once JDBC sink by extending TwoPhaseCommitFunction
> would require to be able to recover (and commit) an open transaction after
> a task was restarted.
> Not sure if the JDBC interface supports this.
>
> Best, Fabian
>
> Am Mi., 21. Nov. 2018 um 11:27 Uhr schrieb Dominik Wosiński <
> wossyn@gmail.com>:
>
>> Hey,
>>
>> As far as I know, the function needs to implement the
>> *TwoPhaseCommitFunction* and not the *CheckpointListener.
>> JDBCSinkFunction *does not implement the two-phase commit, so currently
>> it does not support exactly once.
>>
>> Best Regards,
>> Dom.
>>
>> śr., 21 lis 2018 o 11:07 Jocean shi <jo...@gmail.com> napisał(a):
>>
>>> Hi,
>>> Can JDBCSinkFunction support exectly once? Is it that The
>>> JDBCSinkFunction dont't implement CheckpointListener meaning
>>> JDBCSinkFunction dont't support exectly once?
>>>
>>> cheers
>>>
>>> Jocean
>>>
>>

Re: Can JDBCSinkFunction support exectly once?

Posted by Fabian Hueske <fh...@gmail.com>.

Hi,

JDBCSinkFunction is a simple wrapper around the JDBCOutputFormat (the
DataSet / Batch API output interface).
Dominik is right, that JDBCSinkFunction does not support exactly-once
output.

It is not strictly required that an exactly-once sink implements
TwoPhaseCommitFunction.
TPCF is a convenience interface that internally use the CheckpointListener
interface. It makes it easier to implement exactly-once sinks but is not
the only way to do it.

Implementing an exactly-once JDBC sink by extending TwoPhaseCommitFunction
would require to be able to recover (and commit) an open transaction after
a task was restarted.
Not sure if the JDBC interface supports this.

Best, Fabian

Am Mi., 21. Nov. 2018 um 11:27 Uhr schrieb Dominik Wosiński <
wossyn@gmail.com>:

> Hey,
>
> As far as I know, the function needs to implement the
> *TwoPhaseCommitFunction* and not the *CheckpointListener.
> JDBCSinkFunction *does not implement the two-phase commit, so currently
> it does not support exactly once.
>
> Best Regards,
> Dom.
>
> śr., 21 lis 2018 o 11:07 Jocean shi <jo...@gmail.com> napisał(a):
>
>> Hi,
>> Can JDBCSinkFunction support exectly once? Is it that The
>> JDBCSinkFunction dont't implement CheckpointListener meaning
>> JDBCSinkFunction dont't support exectly once?
>>
>> cheers
>>
>> Jocean
>>
>

Re: Can JDBCSinkFunction support exectly once?

Posted by Dominik Wosiński <wo...@gmail.com>.

Hey,

As far as I know, the function needs to implement the
*TwoPhaseCommitFunction* and not the *CheckpointListener.
JDBCSinkFunction *does
not implement the two-phase commit, so currently it does not support
exactly once.

Best Regards,
Dom.

śr., 21 lis 2018 o 11:07 Jocean shi <jo...@gmail.com> napisał(a):

> Hi,
> Can JDBCSinkFunction support exectly once? Is it that The JDBCSinkFunction
> dont't implement CheckpointListener meaning JDBCSinkFunction dont't support
> exectly once?
>
> cheers
>
> Jocean
>