[jira] [Updated] (NIFI-11449) add autocommit property to control commit in PutDatabaseRecord processor

["Abdelrahim Ahmad (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdelrahim Ahmad updated NIFI-11449:
------------------------------------
    Description: 
The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When using the processor with the Trino-JDBC-Driver to write to an Iceberg catalog, it disables the autocommit feature. This leads to errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".

To fix this issue, the autocommit feature needs to be added in the processor to be enabled/disabled.
enabling auto-commit in the Nifi PutDatabaseRecord processor is important for Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by allowing atomic writes to be performed in the underlying database. This will allow the process to be widely used with bigger range of databases.

_*Improving this process will allow Nifi to be the main tool to ingest data into these new Technologies. So we don't have to deal with another way to ingest data.*_

P.S.: using PutSQL is not a good option at all due to the sensitivity of these tables when dealing with small inserts.

Thanks and best regards
Abdelrahim Ahmad

  was:
The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When using the processor with the Trino-JDBC-Driver to write to an Iceberg catalog, it disables the autocommit feature. This leads to errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".

To fix this issue, the autocommit feature needs to be added in the processor to be enabled/disabled.
enabling auto-commit in the Nifi PutDatabaseRecord processor is important for Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by allowing atomic writes to be performed in the underlying database. This will allow the process to be widely used with bigger range of databases.

P.S.: using PutSQL is not a good option at all due to the sensitivity of these tables when dealing with small inserts.

Thanks and best regards
Abdelrahim Ahmad


> add autocommit property to control commit in PutDatabaseRecord processor
> ------------------------------------------------------------------------
>
>                 Key: NIFI-11449
>                 URL: https://issues.apache.org/jira/browse/NIFI-11449
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.21.0
>         Environment: Any Nifi Deployment
>            Reporter: Abdelrahim Ahmad
>            Priority: Blocker
>              Labels: Trino, autocommit, database, iceberg, putdatabaserecord
>
> The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When using the processor with the Trino-JDBC-Driver to write to an Iceberg catalog, it disables the autocommit feature. This leads to errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".
> To fix this issue, the autocommit feature needs to be added in the processor to be enabled/disabled.
> enabling auto-commit in the Nifi PutDatabaseRecord processor is important for Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by allowing atomic writes to be performed in the underlying database. This will allow the process to be widely used with bigger range of databases.
> _*Improving this process will allow Nifi to be the main tool to ingest data into these new Technologies. So we don't have to deal with another way to ingest data.*_
> P.S.: using PutSQL is not a good option at all due to the sensitivity of these tables when dealing with small inserts.
> Thanks and best regards
> Abdelrahim Ahmad



--
This message was sent by Atlassian Jira
(v8.20.10#820010)