You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Fokko Driesprong <fo...@apache.org> on 2023/06/26 09:27:00 UTC
[VOTE] Release PyIceberg 0.4.0 RC1
Hi Everyone,
Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release is
packed with cool features:
- Support for converting Parquet schemas into Iceberg ones
<https://github.com/apache/iceberg/pull/6997>
- Support for reading data using FSSpec
<https://github.com/apache/iceberg/pull/6972>.
- Support fetching a limited number of rows
<https://github.com/apache/iceberg/pull/7163> to quickly peek into a
dataset.
- Reduced the number of calls
<https://github.com/apache/iceberg/pull/7588> to the object store with
PyArrow>=12.0.0.
- Speed up queries using the Iceberg metrics.
<https://github.com/apache/iceberg/pull/6714>
- Ability to do SQL style filters
<https://github.com/apache/iceberg/pull/6657>: row_filter='passengers >=
3'.|
- SigV4 support <https://github.com/apache/iceberg/pull/7519> for the
REST catalog.
- A complete makeover <https://github.com/apache/iceberg/pull/7108> of
the docs site.
- Support for positional deletes
<https://github.com/apache/iceberg/pull/6775>.
- Ability to set table properties
<https://github.com/apache/iceberg/pull/6323>.
- And many bugs have been fixed
<https://github.com/apache/iceberg/pulls?q=is%3Apr+label%3Apython+is%3Aclosed>
!
I propose that we release the following RC as the official PyIceberg 0.4.0
release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
* This corresponds to the tag: pyiceberg-0.4.0rc1
(c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
* https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
*
https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged on pypi:
https://pypi.org/project/pyiceberg/0.4.0rc1/
And can be installed using: pip3 install pyiceberg==0.4.0rc1
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as PyIceberg 0.4.0
[ ] +0
[ ] -1 Do not release this because...
Please consider this email a +1 from my side:
- Ran some basic table scans
- Including tables with positional deletes
- Checked to see if everything still works when PyArrow is not installed
- Set some table properties
Kind regards,
Fokko
Re: [VOTE] Release PyIceberg 0.4.0 RC1
Posted by Ryan Blue <bl...@tabular.io>.
+1
* Ran license validation with RAT
* Verified signature and checksum
* Built and tested with `make install` and `make test`
* Installed the RC and exercised the CLI with a REST catalog
On Mon, Jun 26, 2023 at 2:27 AM Fokko Driesprong <fo...@apache.org> wrote:
> Hi Everyone,
>
>
> Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
> is packed with cool features:
>
> - Support for converting Parquet schemas into Iceberg ones
> <https://github.com/apache/iceberg/pull/6997>
> - Support for reading data using FSSpec
> <https://github.com/apache/iceberg/pull/6972>.
> - Support fetching a limited number of rows
> <https://github.com/apache/iceberg/pull/7163> to quickly peek into a
> dataset.
> - Reduced the number of calls
> <https://github.com/apache/iceberg/pull/7588> to the object store with
> PyArrow>=12.0.0.
> - Speed up queries using the Iceberg metrics.
> <https://github.com/apache/iceberg/pull/6714>
> - Ability to do SQL style filters
> <https://github.com/apache/iceberg/pull/6657>: row_filter='passengers
> >= 3'.|
> - SigV4 support <https://github.com/apache/iceberg/pull/7519> for the
> REST catalog.
> - A complete makeover <https://github.com/apache/iceberg/pull/7108> of
> the docs site.
> - Support for positional deletes
> <https://github.com/apache/iceberg/pull/6775>.
> - Ability to set table properties
> <https://github.com/apache/iceberg/pull/6323>.
> - And many bugs have been fixed
> <https://github.com/apache/iceberg/pulls?q=is%3Apr+label%3Apython+is%3Aclosed>
> !
>
> I propose that we release the following RC as the official PyIceberg
> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> * This corresponds to the tag: pyiceberg-0.4.0rc1
> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>
> * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>
> *
> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> The release tarball, signature, and checksums are here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>
>
> You can find the KEYS file here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on pypi:
>
>
> https://pypi.org/project/pyiceberg/0.4.0rc1/
>
>
> And can be installed using: pip3 install pyiceberg==0.4.0rc1
>
>
> Please download, verify, and test.
>
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as PyIceberg 0.4.0
>
> [ ] +0
>
> [ ] -1 Do not release this because...
>
>
> Please consider this email a +1 from my side:
>
>
> - Ran some basic table scans
> - Including tables with positional deletes
> - Checked to see if everything still works when PyArrow is not
> installed
> - Set some table properties
>
> Kind regards,
>
> Fokko
>
--
Ryan Blue
Tabular
Re: [VOTE] Release PyIceberg 0.4.0 RC1
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thank you Dan, great catch!
I've pinned <https://github.com/apache/iceberg/pull/7927> the version of
pyparsing. It looks like it switched to non-greedy matching. The first
commit updates the tests, which exposes the issue, and the second commit
locks pyparsing to <3.1.0. It looks like the pyparsing library is rapidly
evolving, and the best thing for now is to just pin it to a range that
works.
Let's cancel this vote, and once the PR is merged, I'll cut the next RC.
Kind regards, Fokko
Op di 27 jun 2023 om 21:14 schreef Daniel Weeks <da...@gmail.com>:
> I ran into an issue with the row filtering:
>
> t.scan(row_filter="location_id > 1").to_pandas()
>
> File
> ~/workspace/apache/releases/pyiceberg/0.4.0-rc1/pyiceberg-0.4.0/pyiceberg/schema.py:183,
> in Schema.find_field(self, name_or_id, case_sensitive)
> 180 field_id = self._lazy_name_to_id_lower.get(name_or_id.lower())
> 182 if field_id is None:
> --> 183 raise ValueError(f"Could not find field with name
> {name_or_id}, case_sensitive={case_sensitive}")
> 185 return self._lazy_id_to_field[field_id]
>
> ValueError: Could not find field with name l, case_sensitive=True
>
> I shared this with Fokko.
>
> -Dan
>
>
> On Mon, Jun 26, 2023 at 9:58 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> +1 (non binding)
>>
>> Regards
>> JB
>>
>> On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org>
>> wrote:
>> >
>> > Hi Everyone,
>> >
>> >
>> > Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
>> is packed with cool features:
>> >
>> > Support for converting Parquet schemas into Iceberg ones
>> > Support for reading data using FSSpec.
>> > Support fetching a limited number of rows to quickly peek into a
>> dataset.
>> > Reduced the number of calls to the object store with PyArrow>=12.0.0.
>> > Speed up queries using the Iceberg metrics.
>> > Ability to do SQL style filters: row_filter='passengers >= 3'.|
>> > SigV4 support for the REST catalog.
>> > A complete makeover of the docs site.
>> > Support for positional deletes.
>> > Ability to set table properties.
>> > And many bugs have been fixed!
>> >
>> > I propose that we release the following RC as the official PyIceberg
>> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>> >
>> >
>> > * This corresponds to the tag: pyiceberg-0.4.0rc1
>> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>> >
>> > * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>> >
>> > *
>> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>> >
>> >
>> > The release tarball, signature, and checksums are here:
>> >
>> >
>> > * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>> >
>> >
>> > You can find the KEYS file here:
>> >
>> >
>> > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>> >
>> >
>> > Convenience binary artifacts are staged on pypi:
>> >
>> >
>> > https://pypi.org/project/pyiceberg/0.4.0rc1/
>> >
>> >
>> > And can be installed using: pip3 install pyiceberg==0.4.0rc1
>> >
>> >
>> > Please download, verify, and test.
>> >
>> >
>> > Please vote in the next 72 hours.
>> >
>> > [ ] +1 Release this as PyIceberg 0.4.0
>> >
>> > [ ] +0
>> >
>> > [ ] -1 Do not release this because...
>> >
>> >
>> > Please consider this email a +1 from my side:
>> >
>> > Ran some basic table scans
>> >
>> > Including tables with positional deletes
>> >
>> > Checked to see if everything still works when PyArrow is not installed
>> > Set some table properties
>> >
>> > Kind regards,
>> >
>> > Fokko
>>
>
Re: [VOTE] Release PyIceberg 0.4.0 RC1
Posted by Daniel Weeks <da...@gmail.com>.
I ran into an issue with the row filtering:
t.scan(row_filter="location_id > 1").to_pandas()
File
~/workspace/apache/releases/pyiceberg/0.4.0-rc1/pyiceberg-0.4.0/pyiceberg/schema.py:183,
in Schema.find_field(self, name_or_id, case_sensitive)
180 field_id = self._lazy_name_to_id_lower.get(name_or_id.lower())
182 if field_id is None:
--> 183 raise ValueError(f"Could not find field with name {name_or_id},
case_sensitive={case_sensitive}")
185 return self._lazy_id_to_field[field_id]
ValueError: Could not find field with name l, case_sensitive=True
I shared this with Fokko.
-Dan
On Mon, Jun 26, 2023 at 9:58 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> +1 (non binding)
>
> Regards
> JB
>
> On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org>
> wrote:
> >
> > Hi Everyone,
> >
> >
> > Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
> is packed with cool features:
> >
> > Support for converting Parquet schemas into Iceberg ones
> > Support for reading data using FSSpec.
> > Support fetching a limited number of rows to quickly peek into a dataset.
> > Reduced the number of calls to the object store with PyArrow>=12.0.0.
> > Speed up queries using the Iceberg metrics.
> > Ability to do SQL style filters: row_filter='passengers >= 3'.|
> > SigV4 support for the REST catalog.
> > A complete makeover of the docs site.
> > Support for positional deletes.
> > Ability to set table properties.
> > And many bugs have been fixed!
> >
> > I propose that we release the following RC as the official PyIceberg
> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
> >
> >
> > * This corresponds to the tag: pyiceberg-0.4.0rc1
> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
> >
> > * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
> >
> > *
> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
> >
> >
> > The release tarball, signature, and checksums are here:
> >
> >
> > * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
> >
> >
> > You can find the KEYS file here:
> >
> >
> > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
> >
> >
> > Convenience binary artifacts are staged on pypi:
> >
> >
> > https://pypi.org/project/pyiceberg/0.4.0rc1/
> >
> >
> > And can be installed using: pip3 install pyiceberg==0.4.0rc1
> >
> >
> > Please download, verify, and test.
> >
> >
> > Please vote in the next 72 hours.
> >
> > [ ] +1 Release this as PyIceberg 0.4.0
> >
> > [ ] +0
> >
> > [ ] -1 Do not release this because...
> >
> >
> > Please consider this email a +1 from my side:
> >
> > Ran some basic table scans
> >
> > Including tables with positional deletes
> >
> > Checked to see if everything still works when PyArrow is not installed
> > Set some table properties
> >
> > Kind regards,
> >
> > Fokko
>
Re: [VOTE] Release PyIceberg 0.4.0 RC1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (non binding)
Regards
JB
On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> Hi Everyone,
>
>
> Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release is packed with cool features:
>
> Support for converting Parquet schemas into Iceberg ones
> Support for reading data using FSSpec.
> Support fetching a limited number of rows to quickly peek into a dataset.
> Reduced the number of calls to the object store with PyArrow>=12.0.0.
> Speed up queries using the Iceberg metrics.
> Ability to do SQL style filters: row_filter='passengers >= 3'.|
> SigV4 support for the REST catalog.
> A complete makeover of the docs site.
> Support for positional deletes.
> Ability to set table properties.
> And many bugs have been fixed!
>
> I propose that we release the following RC as the official PyIceberg 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> * This corresponds to the tag: pyiceberg-0.4.0rc1 (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>
> * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>
> * https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> The release tarball, signature, and checksums are here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>
>
> You can find the KEYS file here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on pypi:
>
>
> https://pypi.org/project/pyiceberg/0.4.0rc1/
>
>
> And can be installed using: pip3 install pyiceberg==0.4.0rc1
>
>
> Please download, verify, and test.
>
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as PyIceberg 0.4.0
>
> [ ] +0
>
> [ ] -1 Do not release this because...
>
>
> Please consider this email a +1 from my side:
>
> Ran some basic table scans
>
> Including tables with positional deletes
>
> Checked to see if everything still works when PyArrow is not installed
> Set some table properties
>
> Kind regards,
>
> Fokko