You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "simw (via GitHub)" <gi...@apache.org> on 2025/05/10 23:20:02 UTC
[I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
simw opened a new issue, #1986:
URL: https://github.com/apache/iceberg-python/issues/1986
### Apache Iceberg version
0.9.0 (latest release)
### Please describe the bug 🐞
Preamble: using a local sqlite db:
```python
from pyiceberg.catalog import load_catalog
warehouse_path = "data/warehouse"
catalog = load_catalog(
"default",
**{
'type': 'sql',
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
},
)
```
A pyiceberg UUID column works fine:
```python
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, UUIDType
schema = Schema(
NestedField(field_id=1, name="uuid", field_type=UUIDType(), required=False),
)
catalog.create_table("default.test2", schema=schema)
```
But a pyarrow UUID column gives an error:
```python
import pyarrow as pa
schema = pa.schema([pa.field("foo", pa.uuid(), nullable=True)])
catalog.create_table("default.test4", schema=schema)
```
The exception is:
```
File ~/Code/Projects/others/icebergs/.venv/lib/python3.13/site-packages/pyiceberg/io/pyarrow.py:1032, in _(obj, visitor)
1030 result = visit_pyarrow(field_type, visitor)
1031 except TypeError as e:
-> 1032 raise UnsupportedPyArrowTypeException(obj, f"Column '{obj.name}' has an unsupported type: {field_type}") from e
1033 visitor.after_field(obj)
1035 return visitor.field(obj, result)
UnsupportedPyArrowTypeException: Column 'foo' has an unsupported type: extension<arrow.uuid>
```
Related to https://github.com/simw/pydantic-to-pyarrow/issues/27
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1986: Error creating table from pyarrow schema with pa.uuid()
URL: https://github.com/apache/iceberg-python/issues/1986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "jim-ngoo (via GitHub)" <gi...@apache.org>.
jim-ngoo commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-2870545428
we have the [UUIDType](https://py.iceberg.apache.org/reference/pyiceberg/types/#pyiceberg.types.UUIDType) type already, I think what we missed is the `visit_pyarrow` decorator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-3071860837
Good catch, i think this was auto-closed when the PR was merged
I was able to verify that this issue still exists by changing `pa.binary(16)` to `pa.uuid()` in `test_uuid_partitioning`
```
poetry run pytest tests/integration/test_writes/test_writes.py::test_uuid_partitioning -m integration
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "Tishj (via GitHub)" <gi...@apache.org>.
Tishj commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-3069721325
I just tested, I don't think this is fixed?
I've build from source, creating version 0.10.0
I still can't use `pa.uuid()` to populate the table
I can use `pa.binary(16)` and that still doesn't add the UUIDType logical type to the parquet files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-3802436824
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #1986: Error creating table from pyarrow schema with pa.uuid()
URL: https://github.com/apache/iceberg-python/issues/1986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "simw (via GitHub)" <gi...@apache.org>.
simw commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-2910156608
This would be fixed by #2007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-2978273095
@simw Indeed, I think the PR is ready now 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1986:
URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-3736521966
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org