You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Tishj (via GitHub)" <gi...@apache.org> on 2026/08/05 17:53:26 UTC
[I] [Spark] `catalog.system.add_files` produces manifest file with incorrect Avro key-value metadata [iceberg]
Tishj opened a new issue, #17535:
URL: https://github.com/apache/iceberg/issues/17535
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Spark
```sql
CREATE OR REPLACE TABLE iceberg_catalog.test_add_files.fake_data_test (
id bigint,
name string,
category string,
value double,
count bigint,
active boolean,
description string
)
USING iceberg
TBLPROPERTIES (
'format-version'='2'
)
CALL iceberg_catalog.system.add_files(
table => 'test_add_files.fake_data_test',
source_table => 'parquet.`{FILES_DIR}`'
)
```
This produces the Avro key-value metadata:
```
┌───────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ key │ value │
│ varchar │ varchar │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ schema │ {"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"name","required":false,"type":"st │
│ │ ring"},{"id":3,"name":"category","required":false,"type":"string"},{"id":4,"name":"value","required":false,"type":"double"},{"id":5,"name":"co │
│ │ unt","required":false,"type":"long"},{"id":6,"name":"active","required":false,"type":"boolean"},{"id":7,"name":"description","required":false, │
│ │ "type":"string"}]} │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ avro.schema │ {"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"d │
│ │ efault":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Locati │
│ │ on URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{ │
│ │ "name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of re │
│ │ cords in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_ │
│ │ size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118… │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ avro.codec │ deflate │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ format-version │ 1 │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ partition-spec-id │ 0 │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ iceberg.schema │ {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false," │
│ │ type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type" │
│ │ :"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, o │
│ │ rc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required" │
│ │ :true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total fi │
│ │ le size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"ty… │
├───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ partition-spec │ [] │
└───────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
That `format_version` reports `1`, but the actual manifest file's schema suggests v2 (as does the version of the table)
Most notably, the `manifest_entry.data_file.snapshot_id` attribute is missing, which is fine for v2, but not for v1, where it's marked as `required`.
If a reader trusts the `format_version` obtained from the manifest file's Avro key-value metadata, the reader will error.
And the contents of the file (not relevant, but sharing for completeness)
```
┌────────┬─────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ status │ snapshot_id │ data_file │
│ int32 │ int64 │ struct(file_path varchar, file_format varchar, "partition" "null", record_count bigint, file_size_in_bytes bigint, block_size_in_bytes bigi │
│ │ │ nt, column_sizes map(integer, bigint), value_counts map(integer, bigint), null_value_counts map(integer, bigint), nan_value_counts map(inte │
│ │ │ ger, bigint), lower_bounds map(integer, blob), upper_bounds map(integer, blob), key_metadata blob, split_offsets bigint[], sort_order_id in │
│ │ │ teger) │
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ { │
│ │ │ 'file_path': 'file:/output/files/file_0.parquet', │
│ │ │ 'file_format': PARQUET, 'partition': null, 'record_count': 22402, 'file_size_in_bytes': 952597, 'block_size_in_bytes': 67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {}, 'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {}, 'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null, 'sort_order_id': 0 │
│ │ │ } │
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ { │
│ │ │ 'file_path': 'file:/output/files/file_1.parquet', │
│ │ │ 'file_format': PARQUET, 'partition': null, 'record_count': 33436, 'file_size_in_bytes': 1418456, 'block_size_in_bytes': 67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {}, 'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {}, 'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null, 'sort_order_id': 0 │
│ │ │ } │
├────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ NULL │ { │
│ │ │ 'file_path': 'file:/output/files/file_2.parquet', │
│ │ │ 'file_format': PARQUET, 'partition': null, 'record_count': 44162, 'file_size_in_bytes': 1875070, 'block_size_in_bytes': 67108864, │
│ │ │ 'column_sizes': {}, 'value_counts': {}, 'null_value_counts': {}, 'nan_value_counts': {}, 'lower_bounds': {}, 'upper_bounds': {}, │
│ │ │ 'key_metadata': null, 'split_offsets': null, 'sort_order_id': 0 │
│ │ │ } │
└────────┴─────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Not super relevant, but the parquet files were created with DuckDB:
```py
con = duckdb.connect()
con.execute("select setseed(0.42)")
# Generate synthetic data inside DuckDB
con.execute(f"""
CREATE OR REPLACE TABLE fake_data AS
SELECT
i AS id,
'item_' || i AS name,
CASE floor(random() * 3)
WHEN 0 THEN 'A'
WHEN 1 THEN 'B'
ELSE 'C'
END AS category,
100 + 15 * random() AS value,
CAST(floor(random() * 99) + 1 AS INTEGER) AS count,
random() < 0.5 AS active,
substr(md5(random()::VARCHAR), 1, 24) AS description
FROM range(1, {n_rows + 1}) t(i)
""")
# Get categories
categories = [row[0] for row in con.execute(
"SELECT DISTINCT category FROM fake_data"
).fetchall()]
# Write one parquet file per category
for i, category in enumerate(categories):
file_path = parquet_dir / f"file_{i}.parquet"
con.execute(f"""
COPY (
SELECT * FROM fake_data
WHERE category = '{category}'
)
TO '{file_path}'
(FORMAT PARQUET)
""")
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Spark] `catalog.system.add_files` produces manifest file with incorrect Avro key-value metadata [iceberg]
Posted by "Tishj (via GitHub)" <gi...@apache.org>.
Tishj commented on issue #17535:
URL: https://github.com/apache/iceberg/issues/17535#issuecomment-5195389338
Nvm, sorry for the noise.
The field is present, it's just `NULL`, which should be allowed (although weird for V1 since there was no inheritance)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Spark] `catalog.system.add_files` produces manifest file with incorrect Avro key-value metadata [iceberg]
Posted by "Tishj (via GitHub)" <gi...@apache.org>.
Tishj closed issue #17535: [Spark] `catalog.system.add_files` produces manifest file with incorrect Avro key-value metadata
URL: https://github.com/apache/iceberg/issues/17535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org