You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Zoltán Borók-Nagy (Jira)" <ji...@apache.org> on 2026/09/30 16:54:00 UTC
[jira] [Created] (IMPALA-15462) Iceberg: one column of a type Impala cannot read (GEOMETRY, GEOGRAPHY, TIME, FIXED, TIMESTAMP_NS, UNKNOWN) makes the whole table unloadable
Zoltán Borók-Nagy created IMPALA-15462:
------------------------------------------
Summary: Iceberg: one column of a type Impala cannot read (GEOMETRY, GEOGRAPHY, TIME, FIXED, TIMESTAMP_NS, UNKNOWN) makes the whole table unloadable
Key: IMPALA-15462
URL: https://issues.apache.org/jira/browse/IMPALA-15462
Project: IMPALA
Issue Type: Improvement
Components: Catalog, Frontend
Reporter: Zoltán Borók-Nagy
h3. Problem
Impala converts the whole Iceberg schema when it loads a table. If any column has an Iceberg type that {{IcebergSchemaConverter.toImpalaType()}} does not map, the load fails. Nested struct fields count too. After that, every statement on the table fails, even one that never reads the column: {{SELECT count(*)}}, a SELECT of other columns, DESCRIBE, SHOW CREATE TABLE, SHOW TABLE/COLUMN STATS, SHOW FILES, DESCRIBE HISTORY, metadata tables, COMPUTE STATS and CREATE EXTERNAL TABLE. Only SHOW TABLES works. For HMS tables, INVALIDATE METADATA and DROP TABLE also succeed. REFRESH reports success only once, on a table that was loaded before the change (see below). After a failed load, REFRESH returns the cached load error without reloading, even after another engine has removed the column. Only INVALIDATE METADATA clears it. ALTER TABLE ... DROP COLUMN fails too, so the column cannot be removed from Impala. The error names the type but not the column.
On master the unmapped types are GEOMETRY, GEOGRAPHY, TIMESTAMP_NANO (timestamp_ns and timestamptz_ns, both run), UNKNOWN (all v3), and TIME and FIXED (valid since v1). I verified all six.
Why this matters:
* A schema change made by another engine breaks existing Impala queries. After DuckDB added one GEOMETRY column, every Impala query on the table failed (for HMS tables, after the next REFRESH).
* IMPALA-15162 will make GEOMETRY load, but not the other five types. Its patch (Gerrit 24536 PS19, code reading only) also throws in {{toImpalaType()}} for a geometry column whose CRS is not OGC:CRS84, so such tables will still fail to load.
h3. Steps to reproduce
DuckDB 1.5.5 (iceberg + spatial extensions), attached to an Iceberg REST catalog:
{code:sql}
CREATE TABLE ath.zz.geom (id BIGINT, name VARCHAR, g GEOMETRY) WITH ('format-version'='3');
INSERT INTO ath.zz.geom VALUES (1, 'a', ST_Point(1, 2)), (2, 'b', ST_Point(3, 4)), (3, 'c', NULL);
{code}
Impala, with the REST catalog configured through {{--catalog_config_dir}}:
{code:sql}
SELECT count(*) FROM zz.geom;
SELECT id FROM zz.geom;
DESCRIBE zz.geom;
{code}
HMS variant: publish the HadoopTables layout ({{metadata/v1.metadata.json}} + {{version-hint.text}}), then run {{CREATE EXTERNAL TABLE ... STORED AS ICEBERG LOCATION '...' TBLPROPERTIES('iceberg.catalog'='hadoop.tables')}}.
Schema-change variant: register and query a v3 table {{(id, name)}} (on a v2 table, adding a GEOMETRY column fails at commit; there, {{ADD COLUMN t TIME}} breaks the table the same way). In DuckDB, run {{ALTER TABLE ... ADD COLUMN g GEOMETRY}}. Republish the layout for the new metadata file, run REFRESH in Impala, then query again.
Other types: empty tables created through the REST API with {{time}} or {{fixed\[16\]}} (v2), or with {{timestamp_ns}}, {{timestamptz_ns}}, {{geography}}, {{unknown}} or {{struct<x: long, g: geometry>}} (v3).
h3. Expected
The table loads. Statements that do not reference the column work: count(*), other columns, predicates on other columns. DESCRIBE lists the column and marks its type as unsupported. A statement that references the column, including {{SELECT *}}, fails during analysis with a message that names the column and its Iceberg type. Writes to the table are rejected.
Impala already does this for HBase ({{FeHBaseTable.java:181-186}}: "We can load the metadata but won't be able to select from it."). HMS tables with a type Impala parses but does not support, such as DATETIME, also load ({{FeCatalogUtils.java:78-83}}, {{ScalarType.isSupported()}} :317-319; documented in {{AnalyzerTest.java:276-279}}). Verified on master 3d4d47dc30 with an HMS table {{(id BIGINT, d DATETIME)}}: count(*), SELECT id and DESCRIBE work. SELECT d fails in the analyzer ({{SlotRef.java:172-181}}) with {{Unsupported type 'DATETIME' in 'd'.}} (5d4b095998fac9e2:d226f1f800000000). INSERT fails ({{InsertStmt.java:502-508}}) with {{... because the column 'd' has an unsupported type 'DATETIME'.}} (b6480a608e8312f4:54d1e2b800000000). COMPUTE STATS skips the column ({{ComputeStatsStmt.java:967-971}}).
h3. Actual (master 3d4d47dc30, RELEASE, docker; REST catalog in hybrid mode)
REST catalog, {{SELECT count(*) FROM zz_jira_geometry.geom}} (query df4539d7adb14864:270377a100000000), abridged:
{noformat}
AnalysisException: Failed to load metadata for table: 'zz_jira_geometry.geom'
CAUSED BY: TableLoadingException: Could not load table zz_jira_geometry.geom from catalog
CAUSED BY: TException: Every MetaProvider failed with the following exceptions: Iceberg REST (http://rest:8181): ...
Caused by: java.lang.IllegalStateException: Error loading Iceberg table zz_jira_geometry.geom
Caused by: org.apache.impala.common.ImpalaRuntimeException: Iceberg type 'GEOMETRY' is not supported in Impala
at org.apache.impala.util.IcebergSchemaConverter.toImpalaType(IcebergSchemaConverter.java:130)
at ...IcebergSchemaConverter.convertToHiveSchema(IcebergSchemaConverter.java:142)
at ...IcebergMetaProvider.createStorageDescriptor(IcebergMetaProvider.java:227)
Catalogd (URI TODO): ... InconsistentMetadataFetchException: Fetching TABLE failed: DB_NOT_FOUND ...
{noformat}
* Same error: {{SELECT id}} a5499332860a7a52:6230cfd600000000, DESCRIBE d74978d157aefac9:26251e0400000000, SHOW CREATE TABLE e6466d4188b5f1c5:fd825dd400000000, {{geom.snapshots}} 164976bebaaca411:9e0a672f00000000; on a second copy SHOW TABLE STATS 5349cee4ce80dbb2, SHOW COLUMN STATS 3a400b6729412d5a, SHOW FILES ae446eac6ea07b53, DESCRIBE HISTORY ce44b9534dc0f87f.
* {{geo.trip_v3}}: count(*) e640bebbf68ed2d8:2a94a4fb00000000, the {{p1-xy}} query aa41c487e3416d7f:c9426ac200000000; {{geo.trip_v3_hilbert}} 0f40363e5763f70c:899514f200000000; {{geo.building_v3}} af4cbfe9648386b4:6e28afb700000000. The {{p1-xy}} query on {{geo.trip_bridge}} (same rows, WKB as {{binary}}) returns cnt=14, sum_key=459188959 (cd4392972c099afa:c241684800000000). Trino 483 and StarRocks 4.1.4 return the same answer on {{geo.trip_v3}}.
* Other types: TIME 6149bceac19da5da:89a4aef500000000, FIXED b946ceb98bef8118:8c7796df00000000, TIMESTAMP_NANO ac4678f901cd284d:4ffba16300000000 (timestamp_ns) and 6945d06cc6a0529f:1bd34f9500000000 (timestamptz_ns), GEOGRAPHY 4e486e1d79f40c99:1c7ef78200000000, UNKNOWN 7345ff425846346d:b749633000000000. The struct case (7c4eee4eaf32248e:823aada600000000) says only {{Iceberg type 'GEOMETRY'}}, with no column or field name.
* HMS, CREATE EXTERNAL TABLE (49482d742e20196c:e9ff080900000000): {{ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: CAUSED BY: TableLoadingException: Iceberg type 'GEOMETRY' is not supported in Impala}}. This is misleading because no HMS RPC failed. The error comes from {{IcebergCatalogOpExecutor.populateExternalTableCols()}} ({{IcebergCatalogOpExecutor.java:130-132}}) and is wrapped at {{CatalogOpExecutor.java:4407-4408}}.
* HMS, schema change: after the ADD COLUMN, count(*) still returned 3 from the cached metadata (d84da571554ad277:bbd56b7800000000); through the REST catalog the same table already failed (aa4cbd9eeee43854:896e11c200000000). REFRESH then reported success (c0466540b7caa985:bc25661400000000), because it invalidates the table on an IcebergTableLoadingException ({{CatalogOpExecutor.java:7547-7553}}). The next count(*) failed (974adf54cb4e39d8:fc02f34400000000; catalogd: {{IcebergTableLoadingException ... CAUSED BY: TableLoadingException: Iceberg type 'GEOMETRY' is not supported in Impala}}). In a rerun, a second REFRESH failed with the same load error (6e4756d5c2cdecbe:220fa5af00000000). After DuckDB inserted a row and dropped the column (layout republished), REFRESH still returned the cached error in 1 ms without reloading ({{CatalogOpExecutor.java:8418-8425}}; 794839ac448986ad:8569e91c00000000). INVALIDATE METADATA fixed it (2b4bb8afe84c3111:e5458c9c00000000; count(*) 60423e89b7fb4a96:a120357800000000 returned 4). On a second copy, ALTER TABLE ... DROP COLUMN g (6f4ebcf4ca1051fa) and COMPUTE STATS (5841545f1791c4ad) failed with the load error; INVALIDATE METADATA (524746c0a8aedee8) and DROP TABLE (c74d1db0872eca91) succeeded.
h3. Analysis (master 3d4d47dc30)
* {{IcebergSchemaConverter.toImpalaType()}} throws in its {{default:}} branch ({{IcebergSchemaConverter.java:129-131}}). {{convertToHiveSchema()}} (:142) and {{convertToImpalaSchema()}} (:159) call it for every column at load time: in catalogd from {{IcebergTable.loadSchema()}} ({{IcebergTable.java:783-790}}, via {{load()}} :533 -> {{loadTableMetadata()}} :558 -> {{loadSchemaFromIceberg()}} :770), and for REST tables from {{IcebergMetaProvider.loadTable()}} (:199, wrapped at :212-213) and {{LocalIcebergTable.java:109}}. Time travel ({{IcebergTimeTravelTable.java:124}}), metadata tables ({{IcebergMetadataTable.java:71}}), partition fields ({{FeIcebergTable.java:1237}}), equality-delete fields ({{IcebergEqualityDeleteTable.java:50}}), fields with default values ({{IcebergScanPlanner.java:1165}}), CTAS ({{IcebergCtasTarget.java:122}}) and CREATE EXTERNAL TABLE ({{IcebergCatalogOpExecutor.java:130}}) call it too.
* Impala 4.5.2 has the same {{default:}} throw ({{IcebergSchemaConverter.java:116-117}}), and there UUID is unmapped too. Verified on the apache/impala:4.5.2 docker images (HMS, hadoop.tables). CREATE EXTERNAL TABLE of a v2 table with a TIME, FIXED or UUID column fails with {{Error making 'createTable' RPC to Hive Metastore: CAUSED BY: TableLoadingException: Iceberg type 'TIME' is not supported in Impala}} (d64def7424a0718e:041fa42d00000000; FIXED 4f4c0d96def483cf:062d42a700000000, UUID 64425d6bcd397a29:faa16a8000000000). A registered v2 table that got a TIME column from DuckDB fails to load after REFRESH, with a 6-line error (534319ac164af4e4:160748ca00000000), and its DROP COLUMN fails too (754d967e597bd146:480bac8200000000). 4.5.2 cannot read v3 tables at all: its Iceberg 1.3.1 rejects them with {{IllegalArgumentException: Cannot read unsupported version 3}} (bc4a4a38b37f8093:d17eaa7e00000000).
* Suggested fix: when converting an existing Iceberg schema (table load, and CREATE EXTERNAL TABLE, which copies it at {{IcebergCatalogOpExecutor.java:130}}), but not user-given types in CREATE TABLE or ALTER TABLE, map an unmapped Iceberg type to a placeholder instead of throwing: {{Type.INVALID}}, as for HBase, or an unsupported ScalarType that keeps the Iceberg type string for DESCRIBE and SHOW CREATE TABLE. Give it an HMS type string such as {{binary}} (PS19 of IMPALA-15162 stores GEOMETRY as binary in HMS). Keep the column in both {{convertToHiveSchema()}} and {{convertToImpalaSchema()}}: {{LocalIcebergTable.validateColumns()}} ({{LocalIcebergTable.java:183-189}}) needs the same column count and names (code reading). The reverse conversion (used by CREATE, ALTER and CTAS) must reject the placeholder. Make the SlotRef error name the Iceberg type. Reject DML that writes data files. Until then, the load error should name the column (and struct path) and not claim an HMS RPC failure.
* Fix scope: use the placeholder also for time-travel schemas and metadata tables. Time travel to a snapshot written with the column fails even after the column is dropped (see Workaround). In Iceberg 1.11 the {{readable_metrics}} of {{files}} and {{entries}} has lower_bound/upper_bound fields of the column's own type, so {{IcebergMetadataTable.java:71}} would still throw (code reading). Partition fields of every spec, old ones included, are converted at load ({{FeIcebergTable.java:1222}}, :1237), so an identity partition on a TIME, FIXED or timestamp_ns column needs the placeholder too. Equality-delete fields with such sources can stay unsupported at scan time. For HiveCatalog tables, Iceberg 1.11's {{HiveSchemaUtil}} throws on every commit for TIMESTAMP_NANO, GEOMETRY, GEOGRAPHY and UNKNOWN (code reading; PS19 patches GEOMETRY and GEOGRAPHY), so writes and ALTER TABLE on such tables can still fail.
* Tests: Impala test data has no Iceberg table with these types (git grep; the IMPALA-14738 commit deferred the V3-type tables). They can be made with REST createTable, as in the repro.
h3. Workaround
None in Impala. ALTER TABLE ... DROP COLUMN fails with the same load error (6f4ebcf4ca1051fa:ae17ca1100000000). Dropping the column with another engine makes the table load again. After DuckDB {{ALTER TABLE ... DROP COLUMN g}}, the REST table loaded at once (97452099defe564c:78798b7c00000000). The HMS table needed INVALIDATE METADATA; REFRESH kept returning the cached load error (794839ac448986ad:8569e91c00000000). Time travel to a snapshot written with the column still fails with {{AnalysisException: Could not create iceberg schema. CAUSED BY: ImpalaRuntimeException: Iceberg type 'GEOMETRY' is not supported in Impala}} (e34ed00cf978fb01:71d3d1db00000000). Otherwise, keep a copy of the table without the unsupported column. The benchmark keeps a v2 "bridge" copy with WKB in {{binary}} plus x/y DOUBLE columns. Impala reads that copy, and all 36 bridge runs executed (34 ok, 2 timed out on {{sb8}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org