You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2025/07/15 02:32:59 UTC
[I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
kevinjqliu opened a new issue, #2209:
URL: https://github.com/apache/iceberg-python/issues/2209
### Feature Request / Improvement
I've seen in multiple issues and across multiple PRs where we depend on a specific version of pyarrow.
- Some features are only available after a certain pyarrow version.
- Some bugs might only be resolved after a certain pyarrow version.
The easiest solution is to bump the minimum pyarrow version we depend on. But since this library is often used in diverse python environments, users might want to use many different versions of pyarrow.
For example, we bumped pyarrow minimum version to `17.0.0` in #1621 in order to support filtering `pa.RecordBatch` directly. The previous solution still works, albeit not efficiently, for pyarrow versions lower than `17.0.0`.
I dont have a clear solution but maybe we can decorate certain functions to branch depending on the pyarrow version.
Would love to hear other's thoughts on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "dingo4dev (via GitHub)" <gi...@apache.org>.
dingo4dev commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3072643534
I've go thought some popular repos which also depends on pyarrow. Using if else condition to check version might be the simple and easy way to branch pyarrow version. The downside is the function may have masses code regarding to different version.
Using decorate dispatch is good for readability but I not sure is it worth to add these which will increase the maintenance process
> Dask
> https://github.com/dask/dask/blob/3f6051ae242f894af0f43fd9de7ad7131fb3e70d/dask/dataframe/_compat.py#L27-L28
> Pandas
> https://github.com/pandas-dev/pandas/blob/bc6ad140daf230c470fef92bec598831d4f94a16/pandas/compat/pyarrow.py#L17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3911167726
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3152226648
I would be in favor of keeping it simple, and be a bit of a frontrunner when it comes to bumping PyArrow. I see some projects that support multiple versions of Arrow, and do have if-else branches in the code based on the version, but this would also require testing against different versions which makes it very complex.
Right now, I think we depend too much on Arrow, and we should be able to delegate quite some stuff to Iceberg-Rust at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3829847301
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "gabeiglio (via GitHub)" <gi...@apache.org>.
gabeiglio commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3074406507
Pyarrow versioning is a pain-point internally since it prevents us from integrating to existing services that depend on a specific (old) pyarrow version. I would be very interested to hear solutions for this. I do like the idea of decorators for branching versions impl.
Another thing would be on agreeing how many pyarrow versions would PyIceberg be willing to support at a time, so we dont accumulate as much maintenance burden through the years.
Thanks for bringing this up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2209:
URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3104099526
Thanks for the examples @dingo4dev. i agree we're balancing features availability and project maintainability
> Another thing would be on agreeing how many pyarrow versions would PyIceberg be willing to support at a time, so we dont accumulate as much maintenance burden through the years.
we can have 2 separate conversations. one for minimal supported versions, another for how to support different versions from minimal to current
i also do want to split out the pyarrow specific code also so that we can possibly make it more modular
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2209: [discussion] dealing with multiple pyarrow versions
URL: https://github.com/apache/iceberg-python/issues/2209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org