You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "npennequin (via GitHub)" <gi...@apache.org> on 2025/10/02 15:21:18 UTC
[I] Exception when using DELETE FROM Spark query on table using copy-on-write [iceberg]
npennequin opened a new issue, #14239:
URL: https://github.com/apache/iceberg/issues/14239
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
We are using a Spark job to remove expired rows from an Iceberg table based on a unix timestamp column (stored as an integer).
The query is
DELETE FROM $table
WHERE status = 'Deleted'
AND deleted_date < $timestamp
This works correctly when the table has `'write.delete.mode' = 'merge-on-read'`.
However with the table in `copy-on-write` mode, the query fails with an `IllegalStateException` in the last stage of the deletion to rewrite data:
```
Caused by: java.lang.IllegalStateException: Not an instance of java.lang.CharSequence: 1234
at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:138)
at org.apache.iceberg.data.InternalRecordWrapper.get(InternalRecordWrapper.java:101)
at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:121)
at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:94)
at org.apache.iceberg.util.StructLikeWrapper.equals(StructLikeWrapper.java:91)
at java.base/java.util.HashMap.getNode(HashMap.java:568)
at java.base/java.util.HashMap.containsKey(HashMap.java:592)
at java.base/java.util.HashSet.contains(HashSet.java:204)
at org.apache.iceberg.util.StructLikeSet.contains(StructLikeSet.java:61)
at org.apache.iceberg.data.DeleteFilter.lambda$applyEqDeletes$0(DeleteFilter.java:209)
at org.apache.iceberg.deletes.Deletes$1.shouldKeep(Deletes.java:101)
at org.apache.iceberg.util.Filter$Iterator.shouldKeep(Filter.java:48)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:66)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49)
at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:135)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:120)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:158)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:63)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:63)
at scala.Option.exists(Option.scala:376)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:168)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Exception when using DELETE FROM Spark query on table using copy-on-write [iceberg]
Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on issue #14239:
URL: https://github.com/apache/iceberg/issues/14239#issuecomment-3953617992
Hi @qinghui-xu @npennequin , can you share the steps to reproduce the issue ? I am unable to repro it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Exception when using DELETE FROM Spark query on table using copy-on-write [iceberg]
Posted by "qinghui-xu (via GitHub)" <gi...@apache.org>.
qinghui-xu commented on issue #14239:
URL: https://github.com/apache/iceberg/issues/14239#issuecomment-4681071391
Hello folks,
This issue has the same root cause as https://github.com/apache/iceberg/issues/13873, and has been resolved with the 1.11.0 & 1.10.2 release. (Related [PR](https://github.com/apache/iceberg/pull/15514))
I think we can close this issue now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Exception when using DELETE FROM Spark query on table using copy-on-write [iceberg]
Posted by "qinghui-xu (via GitHub)" <gi...@apache.org>.
qinghui-xu commented on issue #14239:
URL: https://github.com/apache/iceberg/issues/14239#issuecomment-3370604560
It seems the issue not only happens on the data rewriting stage of COW `delete`, but also when `select` on a table being upserted. It's likely related to the eq deletion that need to be applied:
```
java.lang.IllegalStateException: Not an instance of java.lang.CharSequence: 176
at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:138)
at org.apache.iceberg.data.InternalRecordWrapper.get(InternalRecordWrapper.java:101)
at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:121)
at org.apache.iceberg.types.Comparators$StructLikeComparator.compare(Comparators.java:94)
at org.apache.iceberg.util.StructLikeWrapper.equals(StructLikeWrapper.java:91)
at java.base/java.util.HashMap.putVal(HashMap.java:631)
at java.base/java.util.HashMap.put(HashMap.java:608)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:102)
at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:32)
at org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:370)
at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:332)
at org.apache.iceberg.data.BaseDeleteLoader.loadEqualityDeletes(BaseDeleteLoader.java:115)
at org.apache.iceberg.data.DeleteFilter.applyEqDeletes(DeleteFilter.java:207)
at org.apache.iceberg.data.DeleteFilter.applyEqDeletes(DeleteFilter.java:224)
at org.apache.iceberg.data.DeleteFilter.filter(DeleteFilter.java:178)
at org.apache.iceberg.spark.source.RowDataReader.open(RowDataReader.java:99)
at org.apache.iceberg.spark.source.RowDataReader.open(RowDataReader.java:43)
at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:141)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:120)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:158)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:63)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:63)
at scala.Option.exists(Option.scala:376)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:97)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:893)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:893)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org