You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sclee01 (via GitHub)" <gi...@apache.org> on 2025/05/27 12:14:19 UTC
[I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
sclee01 opened a new issue, #13164:
URL: https://github.com/apache/iceberg/issues/13164
### Apache Iceberg version
1.9.0 (latest release)
### Query engine
Snowflake
### Please describe the bug 🐞
### Description
When creating an Iceberg table using a manually defined schema in Java — where field IDs are explicitly assigned — we observed that nested struct field IDs are silently reassigned during table creation.
As a result, the schema stored in metadata.json no longer matches the original schema the user defined.
This mismatch can break compatibility with external engines that rely on strict interpretation of metadata.json, including Snowflake and Spark.
### Problem
Iceberg does not persist the original user-defined field IDs in nested structs, even when the input schema is fully specified.
Instead, it generates new field IDs internally, leading to structural differences between the intended schema and what gets stored.
We reproduced this behavior using both HadoopCatalog and GlueCatalog, and confirmed that the persisted metadata did not retain the original nested field IDs from the input schema.
### Reproduction
The issue can be reproduced using the following code snippet. ([reproduction_code_java.txt](https://github.com/user-attachments/files/20456395/reproduction_code_java.txt))
As a simplified example, consider the case where the schema is defined as follows:
Schema schema = new Schema(
Types.NestedField.optional(1, "c1", Types.StringType.get()),
Types.NestedField.optional(2, "c2", Types.StringType.get()),
Types.NestedField.optional(4, "c3",
Types.StructType.of(
Types.NestedField.optional(3, "pos", Types.StringType.get())
)
),
Types.NestedField.optional(5, "c4", Types.StringType.get())
);
System.out.println("BEFORE CREATE:");
schema.columns().forEach(f -> System.out.println(f.fieldId() + " -> " + f.name()));
if (!catalog.tableExists(tableIdentifier)) {
catalog.createTable(tableIdentifier, schema, PartitionSpec.unpartitioned());
}
Table table = catalog.loadTable(tableIdentifier);
Schema saved = table.schema();
System.out.println("AFTER CREATE:");
saved.columns().forEach(f -> System.out.println(f.fieldId() + " -> " + f.name()));
/*
Input schema (user-defined)
table {
1: c1: optional string
2: c2: optional string
4: c3: optional struct<3: pos: optional string>
5: c4: optional string
}
*/
/*
Stored schema (in metadata.json)
table {
1 : c1: optional string
2: c2: optional string
3: c3: optional struct<5: pos: optional string>
4: c4: optional string
}
*/
> Note: The field IDs in the schema above are intentionally not sequential. This reflects a common pattern in real-world ingestion pipelines where fields may be reordered, extended, or composed in nested json.
While assigning IDs in strict sequence might incidentally avoid this issue in simple cases, nested schemas make this error-prone and harder to control — so this should not be treated as a user mistake.
You can see even though the user provided pos as field ID 3, it was reassigned to 5 inside the struct.
### Why This Matters
After writing the Iceberg table using Parquet, the data and metadata were successfully stored in S3.
However, when querying the table in Snowflake, the following error occurs: ([reproduction_code_java_write_parquet.txt](https://github.com/user-attachments/files/20456586/reproduction_code_java_write_parquet.txt))
> Parquet file schema node type does not match table column. Parquet Node Type: 'GROUP', Expected Type: 'PRIMITIVE', Parquet Path: 'c3', Column Name: 'C4', Column Type: 'string', File Name: 'warehouse/mynamespace.db/table_name_000/ceff.parquet'
The same issue occurs in Apache Spark, where it cannot match the field ID from the metadata with the expected projection.
This causes failure when reading data that was otherwise successfully written.
I have also captured a screenshot of the Snowflake error to help validate the issue, if that would be helpful.
### Temporary Workaround
We manually modified the metadata.json to restore the original nested field ID, after which both Snowflake and Spark were able to read the table successfully.
### Suggestion
I understand that Iceberg may reassign field IDs to ensure internal consistency, but I believe there is a valid use case — especially in Java environments — for preserving explicitly assigned IDs, particularly when full schemas are user-defined and intended for external consumption.
I'm raising this issue based on discussion above, and I think it could be a great opportunity for me to try addressing the bug with some guidance from the community.
Thank you for your time.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "sclee01 (via GitHub)" <gi...@apache.org>.
sclee01 commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061350212
I think somewhere in the java lib. I tested it using rest polaris catalog and glue but they have same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "sclee01 (via GitHub)" <gi...@apache.org>.
sclee01 closed issue #13164: Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines
URL: https://github.com/apache/iceberg/issues/13164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "phil-schreiber (via GitHub)" <gi...@apache.org>.
phil-schreiber commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061491338
I just dove through the table creation and it seems the given IDs are sent to the catalog rest endpoint, but it returns with the IDs overwritten by sequential number. I think this happens on catalog side.
Also to be noted: when updating a schema by adding columns the object UpdateSchema.addColumn() functions do not have an ID parameter. So new columns cannot be assigned an ID client-side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "sclee01 (via GitHub)" <gi...@apache.org>.
sclee01 commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061627475
Great. I will take a look at the catalog side based on your point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "phil-schreiber (via GitHub)" <gi...@apache.org>.
phil-schreiber commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061256178
Still running into this issue in 1.9.1. Any work on this would be much appreciated. As @sclee01 mentioned for schema evolution to be viable it's crucial to be able to specify deterministic ids. Also the Java class API specifically requires IDs to be set, which wouldn't make any sense, if these IDs will never be utilized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "sclee01 (via GitHub)" <gi...@apache.org>.
sclee01 commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061268323
@phil-schreiber Thanks for your comment. I will work on this and raise the PR asap!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "sclee01 (via GitHub)" <gi...@apache.org>.
sclee01 commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3068346577
Hi @phil-schreiber , I revisited this issue and found the root cause.
Even if the user explicitly sets a schema, when writing a Parquet file, the problem does not occur as long as the data file is written using the schema generated (i.e., reassigned with fresh IDs) by the core libraries.
In other words, users should use the schema returned by the core libraries rather than manually constructed schemas.
Since it's not a bug in the core and doesn't cause actual issues when used properly, I'm closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Nested field IDs in user-defined schema are reassigned during table creation, causing query failures in external engines [iceberg]
Posted by "phil-schreiber (via GitHub)" <gi...@apache.org>.
phil-schreiber commented on issue #13164:
URL: https://github.com/apache/iceberg/issues/13164#issuecomment-3061312480
Awesome, I can confirm this is happening in tabluario rest catalog. Do you have already any pointer, where this is going wrong? Is it on catalog side or are the ids reset somewhere within the java lib?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org