You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Paul Lin (Jira)" <ji...@apache.org> on 2022/05/12 09:55:00 UTC
[jira] [Updated] (FLINK-27585) JdbcOutpuFormat fails to flush data on checkpoint and causes data loss
[ https://issues.apache.org/jira/browse/FLINK-27585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Lin updated FLINK-27585:
-----------------------------
Description:
Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot.
However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost.
I'm upgrading it to 1.14 to see if it happens still.
was:
Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot.
However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost.
I'm upgrading it to 1.14 to see if it happens still.
> JdbcOutpuFormat fails to flush data on checkpoint and causes data loss
> ----------------------------------------------------------------------
>
> Key: FLINK-27585
> URL: https://issues.apache.org/jira/browse/FLINK-27585
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / JDBC
> Affects Versions: 1.12.3
> Reporter: Paul Lin
> Priority: Critical
>
> Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot.
> However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost.
> I'm upgrading it to 1.14 to see if it happens still.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)