You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Andriy Redko (Jira)" <ji...@apache.org> on 2023/08/17 20:16:00 UTC
[jira] [Commented] (FLINK-32891) Opensearch SQL connector crash job on upsert from multiple sources (409 version conflict)
[ https://issues.apache.org/jira/browse/FLINK-32891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755686#comment-17755686 ]
Andriy Redko commented on FLINK-32891:
--------------------------------------
Thanks [~thebranchnotmerged] , I think it could be useful enhancements, [~martijnvisser] if you have no objections, please assign it to me
> Opensearch SQL connector crash job on upsert from multiple sources (409 version conflict)
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-32891
> URL: https://issues.apache.org/jira/browse/FLINK-32891
> Project: Flink
> Issue Type: Bug
> Affects Versions: opensearch-1.0.1
> Reporter: Kobe Fitussi
> Priority: Critical
>
> Using Opensearch SQL Connector for flink , An attempt to perform an Upsert for the same document ID from multiple jobs at the same time has resulted in job crush with 409 version conflict error message, In our environment we cannot guarantee that messages will arrive separately.
> I suggest in RowOpensearchEmitter.processUpsert() , when an UpdateRequest object is being created it will also call an existing method of the UpdateRequest "retryOnConflict(int retries)" which is designed to remedy this issue where 'retries' will be set by OpensearchConnectorOptions class , default 0.
> suggested parameter name : 'sink.upsert.retry-on-conflict'
> Also, I do not see evidence of bulk-retries being Effective in this case
--
This message was sent by Atlassian Jira
(v8.20.10#820010)