You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tjulioh (via GitHub)" <gi...@apache.org> on 2025/12/12 13:57:11 UTC
[I] Use of pyarrow_to_schema [iceberg-python]
tjulioh opened a new issue, #2825:
URL: https://github.com/apache/iceberg-python/issues/2825
### Question
Hello,
I have Iceberg tables and I’m loading them using load_table through the Glue catalog.
I want to compare them with Parquets using DuckDB `pqt_data = duckdb.sql(f"SELECT * FROM read_parquet({pqt}, union_by_name=true)").arrow()`, but after that I want to check if there are any different columns to perform schema evolution. At this point, I tried using `pqt_schema = pyarrow_to_schema(pqt_data.schema)` but the result is an error because it doesn’t have an id or name mapping `Parquet file does not have field-ids and the Iceberg table does not have 'schema.name-mapping.default' defined`. What is the reason and is there any solution for this? I just want to compare the types and names in a simple way.
Some code example:
```
pqt_data = duckdb.sql(f"SELECT * FROM read_parquet({pqt}, union_by_name=true)").arrow()
pqt_schema = pyarrow_to_schema(pqt_data.schema)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-3796946790
Comparing schemas, Arrow-to-Arrow and Iceberg-to-Iceberg, would be an interesting feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2825: Use of pyarrow_to_schema
URL: https://github.com/apache/iceberg-python/issues/2825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-5086194441
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-3807709399
@kevinjqliu huge nit here:
If we're encouraging people to use `_pyarrow_to_schema_without_ids` as-is, we should consider making it a public method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-3808089789
Thats a great point! I was just thinking in terms of hacking together something that works 😄
I think it would be helpful to have some public functions dealing with Arrow Schema and Iceberg Schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-3796944163
There are 2 types of schemas here, Iceberg schema and Arrow schema.
`pyarrow_to_schema` translates Arrow schema to Iceberg schema. This function requires the Arrow schema to have field ids
https://github.com/apache/iceberg-python/blob/a26fca71b14e5cb757e4d500ebf6bd207302e802/pyiceberg/io/pyarrow.py#L1119-L1138
You can try `_pyarrow_to_schema_without_ids`
as seen here
https://github.com/apache/iceberg-python/blob/a26fca71b14e5cb757e4d500ebf6bd207302e802/pyiceberg/io/pyarrow.py#L2719-L2734
Note, `_check_pyarrow_schema_compatible` and `_check_schema_compatible` has most of the logic for your use case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Use of pyarrow_to_schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2825:
URL: https://github.com/apache/iceberg-python/issues/2825#issuecomment-5247546830
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org