You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Punit Tandel <pu...@ericsson.com> on 2017/02/16 00:59:59 UTC
Flink jdbc
Hi All
Does flink jdbc support writing the data into H2 Database?
Thanks
Punit
Re: Flink jdbc
Posted by Punit Tandel <pu...@ericsson.com>.
HI,
Yes i have written custom jdbc sink function based on the jdbcoutformat
for streaming and its working and writing records in postgres db or H2
in memory db. However trying to figure out how many times open method is
called and establishes database connection because for my integration
tests its calling open() method for couple of times which is
establishing connection couple of times and that is only for one single
event coming from kafka. So can you please tell if its a default
implementation that will establish connection multiple times ?
Or db connection will be taken from connection pool if i will run my app
on Cluster? would be great help
Thanks
On 02/22/2017 10:47 PM, Fabian Hueske wrote:
> Hi,
>
> I should also mention that the JdbcOutputFormat batches writes to the
> database. Since it is not integrated with the Flink's checkpointing
> mechanism, data might get lost in case of a failure.
> I would recommend to implement a JdbcSinkFunction based on the code of
> the JdbcOutputFormat.
> If you use the batch JdbcOutputFormat you might get duplicates or lose
> data.
>
> Best, Fabian
>
> 2017-02-16 15:39 GMT+01:00 Punit Tandel <punit.tandel@ericsson.com
> <ma...@ericsson.com>>:
>
> Thanks for the info, At the moment i used the flink-jdbc to write
> the streaming data coming from kafka which i can process and write
> those data in postgres or mysql database configured on cluster or
> sandbox, However when trying to write integration tests i am using
> in memory H2 database which some what acting strange as i can not
> see any error being thrown by write record method but at the same
> time nothing is written in database. So kinda a little hard to
> figure whats going wrong here.
>
> Thanks
>
>
> On 02/16/2017 02:02 PM, Fabian Hueske wrote:
>> The JdbcOutputFormat was originally meant for batch jobs.
>> It should be possible to use it for streaming jobs as well,
>> however, you should be aware that it is not integrated with Flink
>> checkpointing mechanism.
>> So, you might have duplicate data in case of failures.
>>
>> I also don't know if or how well it works with H2.
>>
>> Best, Fabian
>>
>> 2017-02-16 11:06 GMT+01:00 Punit Tandel
>> <punit.tandel@ericsson.com <ma...@ericsson.com>>:
>>
>> Yes i have been following the tutorials and reading from H2
>> and writing to H2 works fine, But problem here is data coming
>> from kafka and writing them to h2 engine does not seems to
>> work and cant see any error thrown while writing into in
>> memory H2 database, So couldnt say whats the error and why
>> those data are not inserted.
>>
>> Have been trying to find out cause and looking for logs while
>> flink processes the operations but couldnt find any error
>> being thrown at the time of writing data. Any where i can
>> check for logs ?
>>
>> Thanks
>>
>>
>> On 02/16/2017 01:10 AM, Ted Yu wrote:
>>> See the tutorial at the beginning of:
>>>
>>> flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java
>>>
>>> Looks like plugging in "org.h2.Driver" should do.
>>>
>>> On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel
>>> <punit.tandel@ericsson.com
>>> <ma...@ericsson.com>> wrote:
>>>
>>> Hi All
>>>
>>> Does flink jdbc support writing the data into H2 Database?
>>>
>>> Thanks
>>> Punit
>>>
>>>
>>
>>
>
>
Re: Flink jdbc
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
I should also mention that the JdbcOutputFormat batches writes to the
database. Since it is not integrated with the Flink's checkpointing
mechanism, data might get lost in case of a failure.
I would recommend to implement a JdbcSinkFunction based on the code of the
JdbcOutputFormat.
If you use the batch JdbcOutputFormat you might get duplicates or lose data.
Best, Fabian
2017-02-16 15:39 GMT+01:00 Punit Tandel <pu...@ericsson.com>:
> Thanks for the info, At the moment i used the flink-jdbc to write the
> streaming data coming from kafka which i can process and write those data
> in postgres or mysql database configured on cluster or sandbox, However
> when trying to write integration tests i am using in memory H2 database
> which some what acting strange as i can not see any error being thrown by
> write record method but at the same time nothing is written in database. So
> kinda a little hard to figure whats going wrong here.
>
> Thanks
>
> On 02/16/2017 02:02 PM, Fabian Hueske wrote:
>
> The JdbcOutputFormat was originally meant for batch jobs.
> It should be possible to use it for streaming jobs as well, however, you
> should be aware that it is not integrated with Flink checkpointing
> mechanism.
> So, you might have duplicate data in case of failures.
>
> I also don't know if or how well it works with H2.
>
> Best, Fabian
>
> 2017-02-16 11:06 GMT+01:00 Punit Tandel <pu...@ericsson.com>:
>
>> Yes i have been following the tutorials and reading from H2 and writing
>> to H2 works fine, But problem here is data coming from kafka and writing
>> them to h2 engine does not seems to work and cant see any error thrown
>> while writing into in memory H2 database, So couldnt say whats the error
>> and why those data are not inserted.
>>
>> Have been trying to find out cause and looking for logs while flink
>> processes the operations but couldnt find any error being thrown at the
>> time of writing data. Any where i can check for logs ?
>>
>> Thanks
>>
>> On 02/16/2017 01:10 AM, Ted Yu wrote:
>>
>> See the tutorial at the beginning of:
>>
>> flink-connectors/flink-jdbc/src/main/java/org/apache/flink/
>> api/java/io/jdbc/JDBCInputFormat.java
>>
>> Looks like plugging in "org.h2.Driver" should do.
>>
>> On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel <pu...@ericsson.com>
>> wrote:
>>
>>> Hi All
>>>
>>> Does flink jdbc support writing the data into H2 Database?
>>>
>>> Thanks
>>> Punit
>>>
>>>
>>
>>
>
>
Re: Flink jdbc
Posted by Punit Tandel <pu...@ericsson.com>.
Thanks for the info, At the moment i used the flink-jdbc to write the
streaming data coming from kafka which i can process and write those
data in postgres or mysql database configured on cluster or sandbox,
However when trying to write integration tests i am using in memory H2
database which some what acting strange as i can not see any error being
thrown by write record method but at the same time nothing is written in
database. So kinda a little hard to figure whats going wrong here.
Thanks
On 02/16/2017 02:02 PM, Fabian Hueske wrote:
> The JdbcOutputFormat was originally meant for batch jobs.
> It should be possible to use it for streaming jobs as well, however,
> you should be aware that it is not integrated with Flink checkpointing
> mechanism.
> So, you might have duplicate data in case of failures.
>
> I also don't know if or how well it works with H2.
>
> Best, Fabian
>
> 2017-02-16 11:06 GMT+01:00 Punit Tandel <punit.tandel@ericsson.com
> <ma...@ericsson.com>>:
>
> Yes i have been following the tutorials and reading from H2 and
> writing to H2 works fine, But problem here is data coming from
> kafka and writing them to h2 engine does not seems to work and
> cant see any error thrown while writing into in memory H2
> database, So couldnt say whats the error and why those data are
> not inserted.
>
> Have been trying to find out cause and looking for logs while
> flink processes the operations but couldnt find any error being
> thrown at the time of writing data. Any where i can check for logs ?
>
> Thanks
>
>
> On 02/16/2017 01:10 AM, Ted Yu wrote:
>> See the tutorial at the beginning of:
>>
>> flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java
>>
>> Looks like plugging in "org.h2.Driver" should do.
>>
>> On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel
>> <punit.tandel@ericsson.com <ma...@ericsson.com>> wrote:
>>
>> Hi All
>>
>> Does flink jdbc support writing the data into H2 Database?
>>
>> Thanks
>> Punit
>>
>>
>
>
Re: Flink jdbc
Posted by Fabian Hueske <fh...@gmail.com>.
The JdbcOutputFormat was originally meant for batch jobs.
It should be possible to use it for streaming jobs as well, however, you
should be aware that it is not integrated with Flink checkpointing
mechanism.
So, you might have duplicate data in case of failures.
I also don't know if or how well it works with H2.
Best, Fabian
2017-02-16 11:06 GMT+01:00 Punit Tandel <pu...@ericsson.com>:
> Yes i have been following the tutorials and reading from H2 and writing
> to H2 works fine, But problem here is data coming from kafka and writing
> them to h2 engine does not seems to work and cant see any error thrown
> while writing into in memory H2 database, So couldnt say whats the error
> and why those data are not inserted.
>
> Have been trying to find out cause and looking for logs while flink
> processes the operations but couldnt find any error being thrown at the
> time of writing data. Any where i can check for logs ?
>
> Thanks
>
> On 02/16/2017 01:10 AM, Ted Yu wrote:
>
> See the tutorial at the beginning of:
>
> flink-connectors/flink-jdbc/src/main/java/org/apache/
> flink/api/java/io/jdbc/JDBCInputFormat.java
>
> Looks like plugging in "org.h2.Driver" should do.
>
> On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel <pu...@ericsson.com>
> wrote:
>
>> Hi All
>>
>> Does flink jdbc support writing the data into H2 Database?
>>
>> Thanks
>> Punit
>>
>>
>
>
Re: Flink jdbc
Posted by Punit Tandel <pu...@ericsson.com>.
Yes i have been following the tutorials and reading from H2 and writing
to H2 works fine, But problem here is data coming from kafka and writing
them to h2 engine does not seems to work and cant see any error thrown
while writing into in memory H2 database, So couldnt say whats the error
and why those data are not inserted.
Have been trying to find out cause and looking for logs while flink
processes the operations but couldnt find any error being thrown at the
time of writing data. Any where i can check for logs ?
Thanks
On 02/16/2017 01:10 AM, Ted Yu wrote:
> See the tutorial at the beginning of:
>
> flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java
>
> Looks like plugging in "org.h2.Driver" should do.
>
> On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel
> <punit.tandel@ericsson.com <ma...@ericsson.com>> wrote:
>
> Hi All
>
> Does flink jdbc support writing the data into H2 Database?
>
> Thanks
> Punit
>
>
Re: Flink jdbc
Posted by Ted Yu <yu...@gmail.com>.
See the tutorial at the beginning of:
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java
Looks like plugging in "org.h2.Driver" should do.
On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel <pu...@ericsson.com>
wrote:
> Hi All
>
> Does flink jdbc support writing the data into H2 Database?
>
> Thanks
> Punit
>
>