[jira] [Comment Edited] (FLINK-30960) OutOfMemory error using jdbc sink

["Bo Shen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-30960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685948#comment-17685948 ] 

Bo Shen edited comment on FLINK-30960 at 2/8/23 3:27 PM:
---------------------------------------------------------

[~martijnvisser] OK, will keep that in mind.

By looking at the code in the main branch of [https://github.com/apache/flink-connector-jdbc] "JdbcOutputFormat.java", the code concerned are mostly the same as the version in v1.13.6

The same issue is reproduced using Flink 1.16. I have uploaded a full run log.


was (Author: JIRAUSER298909):
[~martijnvisser] OK, will keep that in mind.

By looking at the code in the main branch of [https://github.com/apache/flink-connector-jdbc] "JdbcOutputFormat.java", the code concerned are mostly the same on main branch in the JDBC connector repository.

The same issue is reproduced using Flink 1.16. I have uploaded a full run log.

> OutOfMemory error using jdbc sink
> ---------------------------------
>
>                 Key: FLINK-30960
>                 URL: https://issues.apache.org/jira/browse/FLINK-30960
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / JDBC
>    Affects Versions: 1.16.0
>         Environment: Using SQL-Client + TableAPI
> Processing Kafka messages to MySQL databases.
>            Reporter: Bo Shen
>            Priority: Major
>         Attachments: heap-memory-metrics.png, image-2023-02-08-16-58-27-765.png, taskmanager.log
>
>
> Here in production I have an environment consisting of a kafka source, to which third-party providers push a few thousand messages every hour, and I use flink jdbc sink to store those messages into a mysql database.
> Recently there are a few taskmanager process crashes. My investigation suggests that the cause boils down to the way exceptions are handled in jdbc batched mode.
>  
> When writing to JDBC failed in batched mode due to some error like DataTuncation, the exception is stored in field "flushException" waiting to be processed by the task main thread.
> This exception is only checked on the next call to "writeRecords". In my case, if the exception happens to occur when processing the last batch of records, and there are no further record comming from source for the next hour, the flushing thread simply repeatly throws a new RuntimeException wrapping the last "flushException" as cause. The new RuntimeException is stored as the new "flushException". 
> Hence, every time the RuntimeException is thrown, the exception hierachy gets bigger, and eventually before the exception is processed on the main thread, the jvm runs out of memory, which causes an unexpected process crash. 
> !image-2023-02-08-16-58-27-765.png!
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)