You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "vtk9 (via GitHub)" <gi...@apache.org> on 2024/04/29 22:51:13 UTC
[I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
vtk9 opened a new issue, #669:
URL: https://github.com/apache/iceberg-python/issues/669
### Apache Iceberg version
0.6.0 (latest release)
### Please describe the bug 🐞
Hello,
Is this a bug or is there something obvious I am misunderstanding/misusing. (I am relatively new to iceberg). Tested on MacOS M2 arm64
```python
from decimal import Decimal
from pyiceberg.catalog.sql import SqlCatalog
import pyarrow as pa
pylist = [{'decimal_col': Decimal('32768.')}]
arrow_schema = pa.schema(
[
pa.field('decimal_col', pa.decimal128(38, 0)),
],
)
arrow_table = pa.Table.from_pylist(pylist, schema=arrow_schema)
catalog = SqlCatalog(
'test_catalog',
**{
'type': "sql'",
'uri': 'sqlite:///pyiceberg.db',
},
)
namespace = 'test_ns'
table_name = 'test_table'
catalog.create_namespace(namespace=namespace)
new_table = catalog.create_table(
identifier=f'{namespace}.{table_name}',
schema=arrow_schema,
location='.',
)
new_table.append(arrow_table)
```
```
OverflowError: int too big to convert
```
- Note: `pylist = [{'decimal_col': Decimal('32767.')}]` works
- Switching `pa.field('decimal_col', pa.decimal128(38, 1)),` also works
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2085328760
I can confirm this on my end, let me see what's needed to fix this 👍 Thanks for reporting this @vtk9 and reproducing it on your end @bigluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2085395904
I think this is the main issue:
```
>>> (32768).bit_length()
16
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2084469240
For reference, this is the full stack trace:
```
Traceback (most recent call last):
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/test.py", line 31, in <module>
new_table.append(arrow_table)
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 1000, in append
for data_file in data_files:
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 2345, in _dataframe_to_data_files
yield from write_file(table, iter([WriteTask(write_uuid, next(counter), df)]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/io/pyarrow.py", line 1758, in write_file
fill_parquet_file_metadata(
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/io/pyarrow.py", line 1695, in fill_parquet_file_metadata
_min = agg.min_as_bytes()
^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/io/pyarrow.py", line 1355, in min_as_bytes
return self.serialize(
^^^^^^^^^^^^^^^
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/io/pyarrow.py", line 1337, in serialize
return to_bytes(self.primitive_type, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/03q8gn91mj95y5bqbcl90hyvmpqpz738-python3-3.11.7/lib/python3.11/functools.py", line 909, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/conversions.py", line 267, in _
return decimal_to_bytes(value)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/Desktop/pyiceberg-vlad-bug/venv/lib/python3.11/site-packages/pyiceberg/utils/decimal.py", line 81, in decimal_to_bytes
return unscaled_value.to_bytes(byte_length, byteorder="big", signed=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: int too big to convert
(venv)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2084497090
@Fokko `decimal_to_bytes`, when invoked without `byte_length`, uses `bytes_required` to get the required number of bytes.
```python
for v in ['32767', '32768', '32769', '32999']:
d = Decimal(v)
print(f'decimal_to_unscaled[{v}]', decimal_to_unscaled(d))
x = decimal_to_unscaled(d)
print(f'bytes_required[{v}]', bytes_required(d))
```
```
decimal_to_unscaled[32767] 32767
bytes_required[32767] 2
decimal_to_unscaled[32768] 32768
bytes_required[32768] 2
decimal_to_unscaled[32769] 32769
bytes_required[32769] 2
decimal_to_unscaled[32999] 32999
bytes_required[32999] 2
```
But it overflows because `decimal_to_bytes` creates a signed value:
```python
unscaled_value = decimal_to_unscaled(value)
if byte_length is None:
byte_length = bytes_required(unscaled_value)
return unscaled_value.to_bytes(byte_length, byteorder="big", signed=True)
```
2 bytes, as a signed int, can store values in the `-32,768` to `32,767` range.
Indeed the same code does not crash when using `Decimal ('32967')`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2088085023
Thanks @Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX closed issue #669: appending to a table with Decimal > 32767 results in `int too big to convert`
URL: https://github.com/apache/iceberg-python/issues/669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] appending to a table with Decimal > 32767 results in `int too big to convert` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #669:
URL: https://github.com/apache/iceberg-python/issues/669#issuecomment-2088044658
A fix has been merged, thanks for reporting this @bigluck and @vtk9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org