You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Fokko Driesprong <fo...@apache.org> on 2023/08/15 14:21:04 UTC
PyIceberg 0.5.0 release
Hi everyone,
As mentioned in the latest Iceberg sync, I'd love to do another Python
release. I would like to reach out to the community to see if there is
anything we should include.
I know that it was promised that the next release would have to write
support, but so many features have already accumulated, and we want don't
want to rush write support, I would suggest doing a release in between to
get it out to the public.
A summary of what's on master:
- Add gzip metadata support <https://github.com/apache/iceberg/pull/7984>
- PyArrow HDFS support <https://github.com/apache/iceberg/pull/7997>
- Support serverless environments (AWS Lambda)
<https://github.com/apache/iceberg/pull/8061>
- Many fixes around Avro performance (PRs 1
<https://github.com/apache/iceberg/pull/8074>, 2
<https://github.com/apache/iceberg/pull/8075>, 3
<https://github.com/apache/iceberg/pull/8082>, 4
<https://github.com/apache/iceberg/pull/8084>)
- Remove the upper bound of PyParsing dependency
<https://github.com/apache/iceberg/pull/8116> (blocking a PR in Airflow
<https://github.com/apache/airflow/pull/32786>)
- Moving the reading of Avro to Cython
<https://github.com/apache/iceberg/pull/8134> (10x speed improvement(!))
- Support for the SQLCatalog
<https://github.com/apache/iceberg/pull/7921> (JDBC in Java)
- Fix support for UUID columns
<https://github.com/apache/iceberg/pull/8267>
- A lot of bugfixes!
What I think we should include (but please feel free to add anything to
this list):
- Support for adding columns
<https://github.com/apache/iceberg/pull/8174>
- Optimize concurrency <https://github.com/apache/iceberg/pull/8104>
(follow up on the Support servless environments)
- Bump Pydantic to v2 <https://github.com/apache/iceberg/pull/7782>
(this will unblock PyIceberg Polars integration
<https://github.com/pola-rs/polars/pull/10375>, and improve performance
of the JSON (de)serialization)
I've added those to the 0.5.0 milestone
<https://github.com/apache/iceberg/milestone/30>. Feel free to reach out if
you think anything is missing, otherwise we can proceed with the release.
Also, everyone is invited to review the open PR's :)
Cheers, Fokko