You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "CaptainEureka (via GitHub)" <gi...@apache.org> on 2025/06/02 11:18:05 UTC
[I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
CaptainEureka opened a new issue, #2057:
URL: https://github.com/apache/iceberg-python/issues/2057
### Apache Iceberg version
0.9.1 (latest release)
### Please describe the bug 🐞
When attempting to add Parquet files to an Iceberg table using `Table.add_files`, the operation fails if a column defined as `DecimalType` in the Iceberg schema is physically stored as `FIXED_LEN_BYTE_ARRAY` in the Parquet file, *even if* the decimal's precision would typically map to `INT32` or `INT64` according to Iceberg's preferred Parquet mapping.
I see in the Iceberg Spec that on-write the mapping is correct. However, the current behaviour seems to overly restrict the physical Parquet type for decimals during the file addition process. I believe this greatly limits the *kinds* of parquet files that can be "added" to an Iceberg table this way.
**Steps to Reproduce:**
1. Define an Iceberg table schema with a `DecimalType` column, for example, `Decimal(10, 2)`.
* Iceberg's preferred Parquet physical type for `Decimal(10, 2)` would be `INT64`.
2. Create a Parquet file where the corresponding column for this `Decimal(10, 2)` is physically stored as `FIXED_LEN_BYTE_ARRAY`. The data itself is valid for `Decimal(10, 2)`.
3. Attempt to add this Parquet file to the Iceberg table using `Table.add_files`.
**Behavior:**
The `Table.add_files` operation fails, with the following error:
```sh
ValueError: Unexpected physical type FIXED_LEN_BYTE_ARRAY for DecimalType(10, 2) expected INT32
```
indicating a mismatch between the expected physical type (e.g., `INT64`) and the actual physical type (`FIXED_LEN_BYTE_ARRAY`) found in the Parquet file for the decimal column.
**Expected Behavior:**
The `Table.add_files` operation should succeed and correctly read the decimal values from the `FIXED_LEN_BYTE_ARRAY` physical storage. The Iceberg reader/writer should be lenient with the physical storage format of decimals OR otherwise `Table.add_files` should document these limitations.
**Environment:**
* Python version: 3.12.9
* Parquet library and version: pyarrow 20.0.0
P.S. If this is just user error and I shouldn't be trying to do things this way I'd be happy to hear alternatives.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "basil-soma (via GitHub)" <gi...@apache.org>.
basil-soma commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-2932945040
I'm running into the same issue myself with table upserts since 0.9.1. Thanks for raising this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "CaptainEureka (via GitHub)" <gi...@apache.org>.
CaptainEureka commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-3034953108
@thijsheijden This is essentially what I was trying to do as well. Thanks for the example script!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-2942607000
I think this was fixed recently by #1839 which is not yet released
We have a nightly build running in https://test.pypi.org/project/pyiceberg/ which should include this change. Could you give that a try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "thijsheijden (via GitHub)" <gi...@apache.org>.
thijsheijden commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-3019511937
I am experiencing this bug as well. I am trying to insert the TPC-H dataset generated using DuckDB, written to Parquet files, into Iceberg using the `add_files` method. I have attached my Parquet generation code as well as my Iceberg creation code.
Generation of TPC-H Parquet files:
```
# Connect and load TPCH extension
con = duckdb.connect()
con.execute("INSTALL tpch; LOAD tpch")
# Define TPCH table names
tpch_tables = ['orders', 'lineitem']
# Optional: Set scale factor (controls number of rows)
scale = 1.0
con.execute(f"CALL dbgen(sf={scale})")
for table in tpch_tables:
print(f'Generating files for table {table}')
cur_offset = 0
file_idx = 0
batch_idx = 0
while True:
if file_idx % args.b == 0:
Path(f'{table}/batch_{batch_idx}').mkdir(exist_ok=True)
batch_idx += 1
print(f'Writing batch {batch_idx}')
query = f"SELECT * FROM '{table}' LIMIT 1000 OFFSET {cur_offset}"
df = con.execute(query).arrow() # Get as Arrow Table
# Write to a single Parquet file
pq.write_table(df, f'{table}/batch_{batch_idx - 1}/part{file_idx}.parquet')
if len(df) < 1000:
break
file_idx += 1
cur_offset += 1000
```
Insertion into Iceberg table:
```
warehouse_path = "/data"
catalog = load_catalog(
"pyiceberg",
**{
'type': 'sql',
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
},
)
catalog.create_namespace_if_not_exists("default")
# Load the batches of files to import
batches = os.listdir(args.file_dir)
first_file = os.path.join(args.file_dir, "batch_0", os.listdir(os.path.join(args.file_dir, "batch_0"))[0])
# Create table using schema of the first file
df = pq.read_table(first_file)
table = catalog.create_table_if_not_exists(
f"default.{args.table}",
schema=df.schema,
)
batch_idx = 1
for batch_dir in batches:
print(f"Adding batch {batch_idx}")
batch_dir = os.path.join(args.file_dir, batch_dir)
file_paths = os.listdir(batch_dir)
file_paths = [batch_dir + '/' + s for s in file_paths]
table.add_files(file_paths=file_paths, check_duplicate_files=False)
batch_idx += 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-2952600581
Can you paste a full script that can reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "CaptainEureka (via GitHub)" <gi...@apache.org>.
CaptainEureka commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-2948816026
> As of at least `0.10.0.dev20250605002910` this is resolved for me with the upsert. I have not replicated this with the original issue though. Thank you!
I tested this with build `0.10.0.dev20250606002903` and from `main` and it fails still on `Table.add_files`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-3775559308
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2057: `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64`
URL: https://github.com/apache/iceberg-python/issues/2057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "basil-soma (via GitHub)" <gi...@apache.org>.
basil-soma commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-2946325633
As of at least `0.10.0.dev20250605002910` this is resolved for me with the upsert. I have not checked with the original issue though. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `Table.add_files` fails for Parquet files with `DecimalType` columns stored as `FIXED_LEN_BYTE_ARRAY` when precision allows `INT32`/`INT64` [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2057:
URL: https://github.com/apache/iceberg-python/issues/2057#issuecomment-3712564625
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org