Flink Jdbc streaming source support in 1.7.1 or in future?

[Manjusha Vuyyuru <vm...@gmail.com>]

Hello,

Do flink 1.7.1 supports connection to relational database(mysql)?
I want to use mysql as my streaming source to read some configuration..

Thanks,
Manju

Re: Flink Jdbc streaming source support in 1.7.1 or in future?

[Fabian Hueske <fh...@gmail.com>]

I think this is very hard to build in a generic way.
The common approach here would be to get access to the changelog stream of
the table, writing it to a message queue / event log (like Kafka, Pulsar,
Kinesis, ...) and ingesting the changes from the event log into a Flink
application.

You can of course build a custom source that returns all added rows if you
have some meta data in the table, e.g., timestamps indicating when a row
was added.
However, this can easily become very complex, for example if you need to
handle deletes and updates.

Best, Fabian

Am Mi., 23. Jan. 2019 um 08:14 Uhr schrieb Manjusha Vuyyuru <
vmanjusha.cse@gmail.com>:

> But 'JDBCInputFormat' will exit once its done reading all data.I need
> something like which keeps polling to mysql and fetch if there are any
> updates or changes.
>
> Thanks,
> manju
>
> On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <do...@gmail.com> wrote:
>
>> Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
>> read data from a database.
>> u can have a try.
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>
>

Re: Flink Jdbc streaming source support in 1.7.1 or in future?

[Puneet Kinra <pu...@customercentria.com>]

Then common way is to read in the cdc .writing generic operator wont be
easy .

On Wed, Jan 23, 2019 at 12:45 PM Manjusha Vuyyuru <vm...@gmail.com>
wrote:

> But 'JDBCInputFormat' will exit once its done reading all data.I need
> something like which keeps polling to mysql and fetch if there are any
> updates or changes.
>
> Thanks,
> manju
>
> On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <do...@gmail.com> wrote:
>
>> Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
>> read data from a database.
>> u can have a try.
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>
>

-- 
*Cheers *

*Puneet Kinra*

*Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
<pu...@customercentria.com>*

*e-mail :puneet.kinra@customercentria.com
<pu...@customercentria.com>*

Re: Flink Jdbc streaming source support in 1.7.1 or in future?

[Manjusha Vuyyuru <vm...@gmail.com>]

But 'JDBCInputFormat' will exit once its done reading all data.I need
something like which keeps polling to mysql and fetch if there are any
updates or changes.

Thanks,
manju

On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <do...@gmail.com> wrote:

> Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
> read data from a database.
> u can have a try.
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: Flink Jdbc streaming source support in 1.7.1 or in future?

[Zhenghua Gao <do...@gmail.com>]

Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
read data from a database.
u can have a try.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/