[jira] [Comment Edited] (FLINK-30960) OutOfMemory error using jdbc sink

["Bo Shen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-30960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781713#comment-17781713 ] 

Bo Shen edited comment on FLINK-30960 at 11/1/23 12:14 PM:
-----------------------------------------------------------

This issue still exists in the latest master branch of flink-jdbc-connector.

I would like to contribute a PR for a quick fix if this is OK:
 # Introduce a new FlushingException extends RuntimeException, and throw this new type instead in "checkFlushException".
 # In JdbcOutputFormat's scheduler thread, add a type check before replacing existing flushException. If newly caught exception is of "FlushingException", ignore it.

 


was (Author: JIRAUSER298909):
This issue still exists in the latest master branch of flink-jdbc-connector.

I would like to contribute a PR for a quick fix if this is OK:
 # Introduce a new FlushingException extends RuntimeException, and throw new type instead in "checkFlushException".
 # In JdbcOutputFormat's scheduler thread, add a type check before replacing existing flushException. If newly caught exception is of "FlushingException", ignore it.

 

> OutOfMemory error using jdbc sink
> ---------------------------------
>
>                 Key: FLINK-30960
>                 URL: https://issues.apache.org/jira/browse/FLINK-30960
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / JDBC
>    Affects Versions: 1.16.0
>         Environment: Using SQL-Client + TableAPI
> Processing Kafka messages to MySQL databases.
>            Reporter: Bo Shen
>            Priority: Major
>         Attachments: heap-memory-metrics.png, image-2023-02-08-16-58-27-765.png, taskmanager.log
>
>
> Here in production I have an environment consisting of a kafka source, to which third-party providers push a few thousand messages every hour, and I use flink jdbc sink to store those messages into a mysql database.
> Recently there are a few taskmanager process crashes. My investigation suggests that the cause boils down to the way exceptions are handled in jdbc batched mode.
>  
> When writing to JDBC failed in batched mode due to some error like DataTuncation, the exception is stored in field "flushException" waiting to be processed by the task main thread.
> This exception is only checked on the next call to "writeRecords". In my case, if the exception happens to occur when processing the last batch of records, and there are no further record comming from source for the next hour, the flushing thread simply repeatly throws a new RuntimeException wrapping the last "flushException" as cause. The new RuntimeException is stored as the new "flushException". 
> Hence, every time the RuntimeException is thrown, the exception hierachy gets bigger, and eventually before the exception is processed on the main thread, the jvm runs out of memory, which causes an unexpected process crash. 
> !image-2023-02-08-16-58-27-765.png!
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)