You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "slachiewicz (via GitHub)" <gi...@apache.org> on 2026/08/23 13:15:41 UTC
[I] Delta to Iceberg conversion loses the field ids of map and list children under column mapping [incubator-xtable]
slachiewicz opened a new issue, #911:
URL: https://github.com/apache/incubator-xtable/issues/911
### Search before asking
I searched the open and closed issues; nothing covers this.
### Describe the problem
Converting a Delta table that has column mapping enabled and contains a `map` or `array` column produces an Iceberg table that cannot be read, once the Delta writer is 3.x. Reading it fails in Iceberg's Parquet reader:
```
java.lang.IllegalArgumentException: [col-127c1280-..., key_value, key] required binary key (STRING) is not in the store: [...]
```
`ITConversionController#testColumnMappingEnabledDeltaToIceberg` covers this path and passes today, because the build is on Delta 2.4. It fails against Delta 3.3, which is what surfaced this.
### Root cause
Delta 3.x writes parquet field ids; Delta 2.4 did not. The difference is visible in the footer of a file written by the same code against each version, for a table with `delta.columnMapping.mode = name`:
```
delta-core 2.4.0 optional group col-0284b2db-... (MAP) { repeated group key_value { ... } }
delta-spark 3.3.3 optional group col-d660aa27-... (MAP) = 2 { repeated group key_value { ... } }
```
XTable's Delta to Iceberg path relies on Iceberg's name mapping, which `IcebergConversionTarget#createAndSetNameMapping` populates with Delta's physical names. Iceberg applies a name mapping only to files that carry no field ids at all. Delta 3.x's ids therefore switch Iceberg to id based resolution, and Delta assigns column mapping ids to struct fields only, never to a map key, a map value or a list element. Those nested fields end up with no id in the file and no name mapping to fall back on, so they cannot be resolved. A map key is required, so the read fails rather than returning nulls.
Delta's own answer to this is `delta.enableIcebergCompatV2`, which hooks `DeltaParquetWriteSupport` specifically to, in its words, "write nested field IDs for list and map types to the parquet schema". With that enabled the ids are in the file, and Delta records them per field under `delta.columnMapping.nested.ids`, keyed by the path the child takes in the file, for example `col-1234.key`.
### What this is not
Ruled out by experiment:
- Iceberg 1.10.2 in place of 1.9.2.
- Parquet 1.13.1 in place of 1.15.2.
- An id or name bug on the XTable side. The Delta schema json and the id to physical name map that `IcebergSchemaExtractor` builds are identical between Delta 2.4 and 3.3.
### Suggested fix
Two parts, of which this issue covers the first:
1. `DeltaSchemaExtractor` should read `delta.columnMapping.nested.ids` and put those ids on the converted map key, map value and list element fields. `IcebergSchemaExtractor` already prefers a field's own id over one it generates, so the Iceberg schema then matches the file.
2. Decide what to do for a source table that does not enable IcebergCompatV2. On Delta 3.x such a table is not convertible to a readable Iceberg table when it contains collections, which is worth documenting whatever the outcome.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
*This issue was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delta to Iceberg conversion loses the field ids of map and list children under column mapping [incubator-xtable]
Posted by "slachiewicz (via GitHub)" <gi...@apache.org>.
slachiewicz commented on issue #911:
URL: https://github.com/apache/incubator-xtable/issues/911#issuecomment-5386316599
On the second item, #908 now carries a characterisation test, `ITConversionController#testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg`. It builds a column mapped table that does not enable IcebergCompatV2, syncs it to Iceberg, asserts the sync reports SUCCESS, and asserts that reading the resulting Iceberg table throws.
That is today's behaviour rather than the desired one: a caller gets a table that only fails when someone reads a collection column. If the project would rather refuse the sync, or warn at sync time, the test is the place that changes.
*This comment was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delta to Iceberg conversion loses the field ids of map and list children under column mapping [incubator-xtable]
Posted by "slachiewicz (via GitHub)" <gi...@apache.org>.
slachiewicz commented on issue #911:
URL: https://github.com/apache/incubator-xtable/issues/911#issuecomment-5388141527
A specification detail that bears on the refuse-versus-warn question, and on how much of this #912 can close.
`delta.columnMapping.nested.ids` is defined in `PROTOCOL.md` under **"Writer Requirements for IcebergCompatV2"**, so it is written only when `delta.enableIcebergCompatV2` is set. Under plain column mapping — `mode=id` and `mode=name` alike — a list's `element` and a map's `key`/`value` have no identity mechanism at all, and those Parquet nodes carry no `field_id`.
Two things follow.
Writing `field_id` for every `StructField` is a general column-mapping requirement rather than a mode-gated one; the modes differ only in how a reader resolves them. That is consistent with the footer diff in the issue body: the 3.x file has ids on the struct fields, and Iceberg switches to id-based resolution as soon as any id is present.
The two cases in this issue therefore need different answers. Where IcebergCompatV2 is on, the ids exist and carrying them is a complete fix. Where it is off, there is nothing to carry — so any id appearing in the generated Iceberg schema for a collection node would have to be invented, which is the condition that produces the unreadable table. No amount of carrying closes that second case.
Worth noting for anyone searching: there is no `delta.columnMapping.nested.physicalNames` key. A collection node's physical path is its nearest ancestor `StructField`'s `physicalName` plus `.element`, `.key` or `.value`.
If it helps the decision on `testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg`: refusing has the advantage that the failure is otherwise silent at sync time and only surfaces when someone reads a collection column, which is exactly what that test pins.
*This comment was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org