You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Tmonster (via GitHub)" <gi...@apache.org> on 2025/08/06 09:00:48 UTC
[I] What am I missing when writing Positional Delete Files? [iceberg]
Tmonster opened a new issue, #13751:
URL: https://github.com/apache/iceberg/issues/13751
### Query engine
DuckDB-Iceberg
### Question
Hi Iceberg team,
I am currently implementing merge-on-read Delete support in DuckDB-Iceberg. I have a simple test case working in DuckDB, but am encountering errors when reading the table again in Spark. The Table can be read with python-iceberg just fine.
Below are the errors I am encountering/what I have tried.
- When writing a simple DuckDB positional-deletes file. File is attached in the zip
```
25/08/06 10:18:04 ERROR BaseDeleteLoader: Failed to process GenericDeleteFile{content=position_deletes, file_path=s3://warehouse/default/test_delete_table/data/0aa20b80-434e-4f65-becc-64bcd7ff9795-deletes.parquet, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=5, file_size_in_bytes=963, column_sizes=null, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@53de5f70, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@53de5f70, key_metadata=null, split_offsets=null, equality_ids=null, sort_order_id=null, data_sequence_number=2, file_sequence_number=2, first_row_id=null, referenced_data_file=s3://warehouse/default/test_delete_table/data/01987e74-d022-7109-8be2-7aa0a313d9fa.parquet, content_offset=null, content_size_in_bytes=null}
java.lang.IllegalArgumentException: Missing required field: file_path
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.defaultReader(BaseParquetReaders.java:269)
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.struct(BaseParquetReaders.java:252)
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.message(BaseParquetReaders.java:219)
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.message(BaseParquetReaders.java:207)
at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:48)
```
This was using file `duckdb-deletes-no-required.parquet`
I thought this would be because DuckDB parquet files don't label the file_path as required, so I tried writing a parquet file with a required file_path field. I got the same error as above
```
25/08/06 10:03:07 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found
25/08/06 10:03:07 ERROR BaseDeleteLoader: Failed to process GenericDeleteFile{content=position_deletes, file_path=s3://warehouse/default/test_delete_table/data/353d5bc1-38d4-4fab-b9a6-7a665453e3e1-deletes.parquet, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=5, file_size_in_bytes=967, column_sizes=null, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@10b77050, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@10b77050 ....
java.lang.IllegalArgumentException: Missing required field: file_path
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.defaultReader(BaseParquetReaders.java:269)
at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.struct(BaseParquetReaders.java:252)
```
This was using file `duckdb-deletes-required-columns.parquet`.
I then tried writing the positional delete file contents using Spark and uploading the resulting file to the location where the original DuckDB delete file was placed. I thought at the very lease this should **not** error. However, this error is even more confusing.
```
25/08/06 10:10:25 ERROR BaseDeleteLoader: Failed to process GenericDeleteFile{content=position_deletes, file_path=s3://warehouse/default/test_delete_table/data/353d5bc1-38d4-4fab-b9a6-7a665453e3e1-deletes.parquet, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=5, file_size_in_bytes=967, column_sizes=null, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@10b77050, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@10b77050, key_metadata=null, split_offsets=null, equality_ids=null, sort_order_id=null, data_sequence_number=2, file_sequence_number=2, first_row_id=null, referenced_data_file=s3://warehouse/default/test_delete_table/data/01987e63-6f21-7e85-995f-d09c8e68524d.parquet, content_offset=null, content_size_in_bytes=null}
java.lang.RuntimeException: org.apache.iceberg.parquet.ParquetIO$ParquetInputFile@41172525 is not a Parquet file. Expected magic number at tail, but found [0, 0, 0, 25]
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:602)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:934)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:925)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:710)
at org.apache.iceberg.parquet.ReadConf.newReader(ReadConf.java:194)
at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:76)
at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:71)
at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:91)
```
All three delete files are zipped here.
[delete-files.zip](https://github.com/user-attachments/files/21614486/delete-files.zip)
Each one of the attached files can be read with a normal spark session.
Any advice as to what I might be missing would be appreciated. I find it odd that python-iceberg can read the position delete files fine, while Spark cannot. This leads me to believe Spark is assuming that some field/parquet schema element exists.
Thanks,
Tom Ebergen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Help needed when writing Positional Delete Files [iceberg]
Posted by "Tmonster (via GitHub)" <gi...@apache.org>.
Tmonster closed issue #13751: Help needed when writing Positional Delete Files
URL: https://github.com/apache/iceberg/issues/13751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Help needed when writing Positional Delete Files [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #13751:
URL: https://github.com/apache/iceberg/issues/13751#issuecomment-3182015185
@Tmonster What are your Spark, iceberg, PyIceberg versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Help needed when writing Positional Delete Files [iceberg]
Posted by "Tmonster (via GitHub)" <gi...@apache.org>.
Tmonster commented on issue #13751:
URL: https://github.com/apache/iceberg/issues/13751#issuecomment-3191220385
Managed to figure this out I was using the field IDs for `_file` and `_pos` and not `file` and `pos`. https://iceberg.apache.org/spec/#reserved-field-ids
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org