You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Noémi Pap-Takács (Jira)" <ji...@apache.org> on 2026/10/17 07:56:00 UTC
[jira] [Updated] (IMPALA-15433) [DOCS] Document Iceberg V3 table support (row lineage, deletion vectors, DML)
[ https://issues.apache.org/jira/browse/IMPALA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noémi Pap-Takács updated IMPALA-15433:
--------------------------------------
Fix Version/s: Impala 5.0.0
> [DOCS] Document Iceberg V3 table support (row lineage, deletion vectors, DML)
> -----------------------------------------------------------------------------
>
> Key: IMPALA-15433
> URL: https://issues.apache.org/jira/browse/IMPALA-15433
> Project: IMPALA
> Issue Type: Documentation
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 5.0.0
>
>
> Impala 5.0 supports Iceberg format-version 3 tables, but docs/ never mentions V3: a grep for "V3", "row lineage", "deletion vector" and _row_id under docs/ finds nothing. Add a concept (suggested id iceberg_v3) in docs/topics/impala_iceberg.xml next to iceberg_v2, covering:
> - Creating V3 tables: CREATE TABLE ... STORED BY ICEBERG TBLPROPERTIES('format-version'='3'), upgrading an existing table with ALTER TABLE ... SET TBLPROPERTIES('format-version'='3'), and ALTER TABLE ... CONVERT TO ICEBERG TBLPROPERTIES('format-version'='3') (IMPALA-15071; accepted values are 1, 2 and 3). CREATE TABLE LIKE keeps the
> source table's format version (IMPALA-14836).
> - Row lineage (IMPALA-14592, IMPALA-14825): the virtual columns _row_id and _last_updated_sequence_number (BIGINT), which exist only on V3 tables, are not returned by SELECT *, need qualifying when several V3 tables are in scope, and lose to a real column of the same name. Underneath are the hidden columns _file_row_id and
> _file_last_updated_sequence_number plus the virtual column ICEBERG__FIRST__ROW__ID; the definitions are COALESCE(_file_row_id, ICEBERG__FIRST__ROW__ID + FILE__POSITION) and COALESCE(_file_last_updated_sequence_number, ICEBERG__DATA__SEQUENCE__NUMBER). The hidden columns can be selected but not altered, dropped, inserted into or reused
> as column names, and COMPUTE STATS skips them.
> - How the values change: INSERT assigns new row ids; UPDATE and MERGE keep _row_id and assign a new _last_updated_sequence_number; OPTIMIZE preserves both.
> - DML on V3 (IMPALA-14588 DELETE, IMPALA-14913 UPDATE, IMPALA-14937 MERGE): row-level changes write deletion vectors in Puffin files instead of Parquet position delete files, at most one per data file. They show up in the files metadata table as content=1, file_format='PUFFIN'. The Parquet rule still applies: write.delete.format.default
> must be PARQUET.
> - The upgrade restriction: on a V3 table that still has V2 position delete files, DELETE, UPDATE and MERGE fail with "... is not allowed on Iceberg format version 3 table '<t>' as it has existing version 2 position delete file(s) ... Run 'OPTIMIZE TABLE <t>'".
> Text elsewhere that belongs to the same work:
> - impala_iceberg.xml: iceberg_delete (~line 600) and iceberg_update (~line 699) say "against Iceberg V2 tables", and the iceberg_update limitation "Only writes position delete files" doesn't hold for V3.
> - impala_iceberg.xml iceberg_v2 (~line 258) still tells the reader to set 'format-version'='2' to get a V2 table, which has been the default since the Iceberg 1.5 upgrade.
> - impala_dml.xml:88: "For Iceberg V2 tables, Impala supports the DELETE ... UPDATE ... MERGE ..." — add V3.
> - impala_langref_unsupported.xml:314: "Impala does not provide virtual columns." Wrong since 4.2 (INPUT__FILE__NAME), and more so now. Consider adding a list of the Iceberg virtual columns: INPUT__FILE__NAME, FILE__POSITION, PARTITION__SPEC__ID, ICEBERG__PARTITION__SERIALIZED, ICEBERG__DATA__SEQUENCE__NUMBER, ICEBERG__FIRST__ROW__ID,
> plus _row_id and _last_updated_sequence_number on V3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org