[jira] [Created] (IMPALA-11331) Create Iceberg transactions earlier

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-11331:
------------------------------------------

             Summary: Create Iceberg transactions earlier
                 Key: IMPALA-11331
                 URL: https://issues.apache.org/jira/browse/IMPALA-11331
             Project: IMPALA
          Issue Type: Bug
          Components: Frontend
            Reporter: Zoltán Borók-Nagy


Currently we create Iceberg transactions via IcebergUtil.getIcebergTransaction() in CatalogOpExecutor.

This is problematic in some cases, especially for INSERT OVERWRITEs, because in this case we open the transaction too late, when the data files are already written, then we open the transaction and commit it. INSERT statements in the meantime get overwritten instead of failing the INSERT OVERWRITE operation.

This can be problematic when we try to use INSERT OVERWRITE for compacting a table. In that case we definitely don't want to loose INSERTed data.

Moving transaction open/close to the coordinator requires a lot of work, and the handling of self-events would become even more complicated.

Alternatively, we could initiate an open transaction from the Coordinator, i.e. asking CatalogD to open one, then at the end CatalogD would commit the opened transaction.

We also need to abort transactions of failed queries. We also need ways of aborting transactions of crashed Coordinators.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org