You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Aleksandr Efimov (Jira)" <ji...@apache.org> on 2026/08/12 14:49:00 UTC
[jira] [Comment Edited] (IMPALA-13832) Add DML support for Iceberg tables in Iceberg REST Catalog
[ https://issues.apache.org/jira/browse/IMPALA-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104093#comment-18104093 ]
Aleksandr Efimov edited comment on IMPALA-13832 at 8/12/26 2:48 PM:
--------------------------------------------------------------------
Hi Zoltan, Peter,
I spent some time on IMPALA-13832 and put together a small end-to-end PoC for {{INSERT INTO}} against an Iceberg REST catalog without CatalogD. It reuses the existing Iceberg sink, then commits the transaction from the coordinator-side Java Frontend through the REST-loaded table. A fresh query sees the new snapshot and row.
I kept the first cut to {{INSERT INTO}} and the simple standalone topology with exactly one REST provider. If CatalogD is deployed or several REST providers are configured, the write is rejected. {{{}INSERT OVERWRITE{}}}, {{{}DELETE{}}}, {{{}UPDATE{}}}, {{{}MERGE{}}}, and {{OPTIMIZE}} are also rejected during analysis. The focused tests cover a successful insert, cleanup after a known commit failure, and retaining the file when the commit state is unknown.
The runtime test is still small: one unpartitioned table, one impalad, one writer, one data file, and {{{}INSERT VALUES{}}}. I have not tested partitioned writes, multiple executors/files, zero-row inserts, or concurrent inserts yet.
There are two shortcuts in the PoC that I do not want to turn into a contract by accident. The finalizer is selected from the global topology, and the target table is reloaded by name through the single REST provider. The design note explains both and sketches a path with provider affinity, table UUID verification, and shared Iceberg commit/cleanup code.
Before I take this further, I would like to check two design choices and one scope question:
# My current preference is for the owning {{MetaProvider}} to reload and commit the table, with {{Frontend}} kept as the JNI entry point and the Iceberg transaction code moved to a shared helper. Does that fit the direction you want for {{MetaProvider}} / {{{}MultiMetaProvider{}}}, or should mutations stay outside the provider interface?
# For provider affinity, I propose using a non-empty, unique {{iceberg.rest-catalog.name}} and carrying it in the Iceberg finalization parameters. Should the first mergeable change support mixed/multiple providers, or keep the current single-provider restriction and add affinity in the next step?
# I plan to carry {{Table.uuid()}} from analysis and verify it before commit. The CatalogD path also resolves the target by name, so it appears to have the same drop/recreate gap. Would you prefer that protection in the same change or as a separate follow-up for the existing path?
The WIP is here: [Gerrit 24682|https://gerrit.cloudera.org/c/24682/]
I also attached [^impala-13832-standalone-rest-insert-design.md] with the code path, test coverage, known gaps, and the direction I would take after design agreement.
was (Author: JIRAUSER313444):
Hi Zoltan, Peter,
I spent some time on IMPALA-13832 and put together a small end-to-end PoC for {{INSERT INTO}} against an Iceberg REST catalog without CatalogD. It reuses the existing Iceberg sink, then commits the transaction from the coordinator-side Java Frontend through the REST-loaded table. A fresh query sees the new snapshot and row.
I kept the first cut to {{INSERT INTO}} and the simple standalone topology with exactly one REST provider. If CatalogD is deployed or several REST providers are configured, the write is rejected. {{INSERT OVERWRITE}}, {{DELETE}}, {{UPDATE}}, {{MERGE}}, and {{OPTIMIZE}} are also rejected during analysis. The focused tests cover a successful insert, cleanup after a known commit failure, and retaining the file when the commit state is unknown.
The runtime test is still small: one unpartitioned table, one impalad, one writer, one data file, and {{INSERT VALUES}}. I have not tested partitioned writes, multiple executors/files, zero-row inserts, or concurrent inserts yet.
There are two shortcuts in the PoC that I do not want to turn into a contract by accident. The finalizer is selected from the global topology, and the target table is reloaded by name through the single REST provider. The design note explains both and sketches a path with provider affinity, table UUID verification, and shared Iceberg commit/cleanup code.
Before I take this further, I would like to check two design choices and one scope question:
# My current preference is for the owning {{MetaProvider}} to reload and commit the table, with {{Frontend}} kept as the JNI entry point and the Iceberg transaction code moved to a shared helper. Does that fit the direction you want for {{MetaProvider}} / {{MultiMetaProvider}}, or should mutations stay outside the provider interface?
# For provider affinity, I propose using a non-empty, unique {{iceberg.rest-catalog.name}} and carrying it in the Iceberg finalization parameters. Should the first mergeable change support mixed/multiple providers, or keep the current single-provider restriction and add affinity in the next step?
# I plan to carry {{Table.uuid()}} from analysis and verify it before commit. The CatalogD path also resolves the target by name, so it appears to have the same drop/recreate gap. Would you prefer that protection in the same change or as a separate follow-up for the existing path?
The WIP is here: [Gerrit 24682|https://gerrit.cloudera.org/c/Impala-ASF/+/24682]
I also attached [a design note|^impala-13832-standalone-rest-insert-design.md] with the code path, test coverage, known gaps, and the direction I would take after design agreement.
> Add DML support for Iceberg tables in Iceberg REST Catalog
> ----------------------------------------------------------
>
> Key: IMPALA-13832
> URL: https://issues.apache.org/jira/browse/IMPALA-13832
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Attachments: impala-13832-standalone-rest-insert-design.md
>
>
> Add DML support for Iceberg tables in Iceberg REST Catalog
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org