You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "qinghui-xu (via GitHub)" <gi...@apache.org> on 2026/03/30 08:29:39 UTC

[I] Unable to read data when using nested field as identifier [iceberg]

qinghui-xu opened a new issue, #15826:
URL: https://github.com/apache/iceberg/issues/15826

   ### Apache Iceberg version
   
   1.10.1 (latest release)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   We have a table using some nested fields as identifier, when we read individual columns (`select *` works fine) from such table an error is thrown:
   ```
   java.lang.IllegalArgumentException: Cannot find required field for ID 20
   	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:190)
   	at org.apache.iceberg.data.DeleteFilter.fileProjection(DeleteFilter.java:304)
   	at org.apache.iceberg.data.DeleteFilter.<init>(DeleteFilter.java:100)
   	at org.apache.iceberg.spark.source.BaseReader$SparkDeleteFilter.<init>(BaseReader.java:205)
   	at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:116)
   	at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
   	at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:141)
   	at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:120)
   	at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:158)
   	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:63)
   	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:63)
   	at scala.Option.exists(Option.scala:376)
   	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
   	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:97)
   	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
   	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
   	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
   	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
   	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
   	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
   	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
   	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
   	at org.apache.spark.scheduler.Task.run(Task.scala:141)
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
   	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
   	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   ```
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Unable to read data when using nested field as identifier [iceberg]

Posted by "qinghui-xu (via GitHub)" <gi...@apache.org>.
qinghui-xu commented on issue #15826:
URL: https://github.com/apache/iceberg/issues/15826#issuecomment-4651347335

   And this new error is because the [field lookup](https://github.com/apache/iceberg/blob/48727c9072a017f582045c429cfe861bae0c0537/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L314) lays out all id fields plainly to the top level, regardless whether they are nested to some inner struct or not. This later triggers the above error when trying to read nested_id_field on the top level columns of eq delete files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Unable to read data when using nested field as identifier [iceberg]

Posted by "bharos (via GitHub)" <gi...@apache.org>.
bharos commented on issue #15826:
URL: https://github.com/apache/iceberg/issues/15826#issuecomment-4166145977

   The crash is in DeleteFilter.fileProjection() — tableSchema.asStruct().field(fieldId) only resolves top-level fields, so nested equality delete fields (like `address.zip`) return null and the Preconditions.checkArgument throws exception.
   
   There's an existing [TODO on DeleteFilter.java](https://github.com/apache/iceberg/blob/da0ad6a7cc2632594f04cb7872bc6f5fb158fd6b/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L305) acknowledging this: "support adding nested columns. this will currently fail when finding nested columns to add", so this is a known limitation.
   
   I think the fix would use tableSchema.findField(fieldId) to locate the nested field, then reconstruct the parent struct path when merging it into the projected schema ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Unable to read data when using nested field as identifier [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #15826:
URL: https://github.com/apache/iceberg/issues/15826#issuecomment-4162748469

   > We have a table using some nested fields as identifier
   
   @qinghui-xu could you share the table schema and proprties?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Unable to read data when using nested field as identifier [iceberg]

Posted by "qinghui-xu (via GitHub)" <gi...@apache.org>.
qinghui-xu commented on issue #15826:
URL: https://github.com/apache/iceberg/issues/15826#issuecomment-4651243137

   Hey guys,
   Just spent some time to check out the latest release to see if issue is resolved, especially with [the PR](https://github.com/apache/iceberg/pull/15268) that seems to do the recursive field lookup when building the delete file reader's schema.
   But it turns out that this is not enough: it some solves the previous reported error of `Cannot find required field for ID`, but it introduces a new bug / error: 
   ```
   java.lang.IllegalArgumentException: Missing required field: host_platform
   	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.defaultReader(BaseParquetReaders.java:269)
   	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.struct(BaseParquetReaders.java:252)
   	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.message(BaseParquetReaders.java:219)
   	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.message(BaseParquetReaders.java:207)
   	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:48)
   	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:67)
   	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:59)
   	at org.apache.iceberg.data.parquet.GenericParquetReaders.buildReader(GenericParquetReaders.java:53)
   	at org.apache.iceberg.data.BaseDeleteLoader.lambda$newParquetReaderFunc$4(BaseDeleteLoader.java:269)
   	at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:121)
   	at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:74)
   	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:94)
   	at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:205)
   	at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:204)
   	at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:205)
   	at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:204)
   	at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:196)
   	at org.apache.iceberg.relocated.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:238)
   	at org.apache.iceberg.data.BaseDeleteLoader.materialize(BaseDeleteLoader.java:143)
   	at org.apache.iceberg.data.BaseDeleteLoader.readEqDeletes(BaseDeleteLoader.java:131)
   	at org.apache.iceberg.data.BaseDeleteLoader.getOrReadEqDeletes(BaseDeleteLoader.java:123)
   	at org.apache.iceberg.data.BaseDeleteLoader.lambda$loadEqualityDeletes$0(BaseDeleteLoader.java:111)
   	at org.apache.iceberg.data.BaseDeleteLoader.lambda$execute$7(BaseDeleteLoader.java:283)
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
   	at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
   	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Unable to read data when using nested field as identifier [iceberg]

Posted by "qinghui-xu (via GitHub)" <gi...@apache.org>.
qinghui-xu commented on issue #15826:
URL: https://github.com/apache/iceberg/issues/15826#issuecomment-4171400840

   Hello folks,
   Thanks for your replies! Indeed after digging a little bit, I think the root cause here is the `StructType#field(int id)` only looks up on the first level fields' id, as @bharos suggested. This is somehow "mitigated" for Spark 4.1 with the [PR](https://github.com/apache/iceberg/pull/15268) because it introduces the [`FieldLookup`](https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java#L259) which uses the recursive field id lookup from `Schema#findField(int id)`. But problem still exists for older versions such as Spark 3.5.
   So now the question is that should we backport this change to other Spark versions, or should we make the `StructType#field(int id)` recursive? Personally I'd prefer the latter as it should make the code more robust and seems to address [the TODO comment on DeleteFilter.java](https://github.com/apache/iceberg/blob/da0ad6a7cc2632594f04cb7872bc6f5fb158fd6b/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L305). I can send a PR with this approach, what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org