You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Fokko Driesprong <fo...@apache.org> on 2023/06/26 09:27:00 UTC

[VOTE] Release PyIceberg 0.4.0 RC1

Hi Everyone,


Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release is
packed with cool features:

   - Support for converting Parquet schemas into Iceberg ones
   <https://github.com/apache/iceberg/pull/6997>
   - Support for reading data using FSSpec
   <https://github.com/apache/iceberg/pull/6972>.
   - Support fetching a limited number of rows
   <https://github.com/apache/iceberg/pull/7163> to quickly peek into a
   dataset.
   - Reduced the number of calls
   <https://github.com/apache/iceberg/pull/7588> to the object store with
   PyArrow>=12.0.0.
   - Speed up queries using the Iceberg metrics.
   <https://github.com/apache/iceberg/pull/6714>
   - Ability to do SQL style filters
   <https://github.com/apache/iceberg/pull/6657>: row_filter='passengers >=
   3'.|
   - SigV4 support <https://github.com/apache/iceberg/pull/7519> for the
   REST catalog.
   - A complete makeover <https://github.com/apache/iceberg/pull/7108> of
   the docs site.
   - Support for positional deletes
   <https://github.com/apache/iceberg/pull/6775>.
   - Ability to set table properties
   <https://github.com/apache/iceberg/pull/6323>.
   - And many bugs have been fixed
   <https://github.com/apache/iceberg/pulls?q=is%3Apr+label%3Apython+is%3Aclosed>
   !

 I propose that we release the following RC as the official PyIceberg 0.4.0
release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f


* This corresponds to the tag: pyiceberg-0.4.0rc1
(c3579a11b4bfa5387e313185e714c40a0ed1ccfe)

* https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1

*
https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f


The release tarball, signature, and checksums are here:


* https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/


You can find the KEYS file here:


* https://dist.apache.org/repos/dist/dev/iceberg/KEYS


Convenience binary artifacts are staged on pypi:


https://pypi.org/project/pyiceberg/0.4.0rc1/


And can be installed using: pip3 install pyiceberg==0.4.0rc1


Please download, verify, and test.


Please vote in the next 72 hours.

[ ] +1 Release this as PyIceberg 0.4.0

[ ] +0

[ ] -1 Do not release this because...


Please consider this email a +1 from my side:


   - Ran some basic table scans
      - Including tables with positional deletes
   - Checked to see if everything still works when PyArrow is not installed
   - Set some table properties

Kind regards,

Fokko

Re: [VOTE] Release PyIceberg 0.4.0 RC1

Posted by Ryan Blue <bl...@tabular.io>.
+1

* Ran license validation with RAT
* Verified signature and checksum
* Built and tested with `make install` and `make test`
* Installed the RC and exercised the CLI with a REST catalog

On Mon, Jun 26, 2023 at 2:27 AM Fokko Driesprong <fo...@apache.org> wrote:

> Hi Everyone,
>
>
> Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
> is packed with cool features:
>
>    - Support for converting Parquet schemas into Iceberg ones
>    <https://github.com/apache/iceberg/pull/6997>
>    - Support for reading data using FSSpec
>    <https://github.com/apache/iceberg/pull/6972>.
>    - Support fetching a limited number of rows
>    <https://github.com/apache/iceberg/pull/7163> to quickly peek into a
>    dataset.
>    - Reduced the number of calls
>    <https://github.com/apache/iceberg/pull/7588> to the object store with
>    PyArrow>=12.0.0.
>    - Speed up queries using the Iceberg metrics.
>    <https://github.com/apache/iceberg/pull/6714>
>    - Ability to do SQL style filters
>    <https://github.com/apache/iceberg/pull/6657>: row_filter='passengers
>    >= 3'.|
>    - SigV4 support <https://github.com/apache/iceberg/pull/7519> for the
>    REST catalog.
>    - A complete makeover <https://github.com/apache/iceberg/pull/7108> of
>    the docs site.
>    - Support for positional deletes
>    <https://github.com/apache/iceberg/pull/6775>.
>    - Ability to set table properties
>    <https://github.com/apache/iceberg/pull/6323>.
>    - And many bugs have been fixed
>    <https://github.com/apache/iceberg/pulls?q=is%3Apr+label%3Apython+is%3Aclosed>
>    !
>
>  I propose that we release the following RC as the official PyIceberg
> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> * This corresponds to the tag: pyiceberg-0.4.0rc1
> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>
> * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>
> *
> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> The release tarball, signature, and checksums are here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>
>
> You can find the KEYS file here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on pypi:
>
>
> https://pypi.org/project/pyiceberg/0.4.0rc1/
>
>
> And can be installed using: pip3 install pyiceberg==0.4.0rc1
>
>
> Please download, verify, and test.
>
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as PyIceberg 0.4.0
>
> [ ] +0
>
> [ ] -1 Do not release this because...
>
>
> Please consider this email a +1 from my side:
>
>
>    - Ran some basic table scans
>       - Including tables with positional deletes
>    - Checked to see if everything still works when PyArrow is not
>    installed
>    - Set some table properties
>
> Kind regards,
>
> Fokko
>


-- 
Ryan Blue
Tabular

Re: [VOTE] Release PyIceberg 0.4.0 RC1

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thank you Dan, great catch!

I've pinned <https://github.com/apache/iceberg/pull/7927> the version of
pyparsing. It looks like it switched to non-greedy matching. The first
commit updates the tests, which exposes the issue, and the second commit
locks pyparsing to <3.1.0. It looks like the pyparsing library is rapidly
evolving, and the best thing for now is to just pin it to a range that
works.

Let's cancel this vote, and once the PR is merged, I'll cut the next RC.

Kind regards, Fokko

Op di 27 jun 2023 om 21:14 schreef Daniel Weeks <da...@gmail.com>:

> I ran into an issue with the row filtering:
>
> t.scan(row_filter="location_id > 1").to_pandas()
>
> File
> ~/workspace/apache/releases/pyiceberg/0.4.0-rc1/pyiceberg-0.4.0/pyiceberg/schema.py:183,
> in Schema.find_field(self, name_or_id, case_sensitive)
>     180     field_id = self._lazy_name_to_id_lower.get(name_or_id.lower())
>     182 if field_id is None:
> --> 183     raise ValueError(f"Could not find field with name
> {name_or_id}, case_sensitive={case_sensitive}")
>     185 return self._lazy_id_to_field[field_id]
>
> ValueError: Could not find field with name l, case_sensitive=True
>
> I shared this with Fokko.
>
> -Dan
>
>
> On Mon, Jun 26, 2023 at 9:58 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> +1 (non binding)
>>
>> Regards
>> JB
>>
>> On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org>
>> wrote:
>> >
>> > Hi Everyone,
>> >
>> >
>> > Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
>> is packed with cool features:
>> >
>> > Support for converting Parquet schemas into Iceberg ones
>> > Support for reading data using FSSpec.
>> > Support fetching a limited number of rows to quickly peek into a
>> dataset.
>> > Reduced the number of calls to the object store with PyArrow>=12.0.0.
>> > Speed up queries using the Iceberg metrics.
>> > Ability to do SQL style filters: row_filter='passengers >= 3'.|
>> > SigV4 support for the REST catalog.
>> > A complete makeover of the docs site.
>> > Support for positional deletes.
>> > Ability to set table properties.
>> > And many bugs have been fixed!
>> >
>> >  I propose that we release the following RC as the official PyIceberg
>> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>> >
>> >
>> > * This corresponds to the tag: pyiceberg-0.4.0rc1
>> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>> >
>> > * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>> >
>> > *
>> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>> >
>> >
>> > The release tarball, signature, and checksums are here:
>> >
>> >
>> > * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>> >
>> >
>> > You can find the KEYS file here:
>> >
>> >
>> > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>> >
>> >
>> > Convenience binary artifacts are staged on pypi:
>> >
>> >
>> > https://pypi.org/project/pyiceberg/0.4.0rc1/
>> >
>> >
>> > And can be installed using: pip3 install pyiceberg==0.4.0rc1
>> >
>> >
>> > Please download, verify, and test.
>> >
>> >
>> > Please vote in the next 72 hours.
>> >
>> > [ ] +1 Release this as PyIceberg 0.4.0
>> >
>> > [ ] +0
>> >
>> > [ ] -1 Do not release this because...
>> >
>> >
>> > Please consider this email a +1 from my side:
>> >
>> > Ran some basic table scans
>> >
>> > Including tables with positional deletes
>> >
>> > Checked to see if everything still works when PyArrow is not installed
>> > Set some table properties
>> >
>> > Kind regards,
>> >
>> > Fokko
>>
>

Re: [VOTE] Release PyIceberg 0.4.0 RC1

Posted by Daniel Weeks <da...@gmail.com>.
I ran into an issue with the row filtering:

t.scan(row_filter="location_id > 1").to_pandas()

File
~/workspace/apache/releases/pyiceberg/0.4.0-rc1/pyiceberg-0.4.0/pyiceberg/schema.py:183,
in Schema.find_field(self, name_or_id, case_sensitive)
    180     field_id = self._lazy_name_to_id_lower.get(name_or_id.lower())
    182 if field_id is None:
--> 183     raise ValueError(f"Could not find field with name {name_or_id},
case_sensitive={case_sensitive}")
    185 return self._lazy_id_to_field[field_id]

ValueError: Could not find field with name l, case_sensitive=True

I shared this with Fokko.

-Dan


On Mon, Jun 26, 2023 at 9:58 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> +1 (non binding)
>
> Regards
> JB
>
> On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org>
> wrote:
> >
> > Hi Everyone,
> >
> >
> > Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release
> is packed with cool features:
> >
> > Support for converting Parquet schemas into Iceberg ones
> > Support for reading data using FSSpec.
> > Support fetching a limited number of rows to quickly peek into a dataset.
> > Reduced the number of calls to the object store with PyArrow>=12.0.0.
> > Speed up queries using the Iceberg metrics.
> > Ability to do SQL style filters: row_filter='passengers >= 3'.|
> > SigV4 support for the REST catalog.
> > A complete makeover of the docs site.
> > Support for positional deletes.
> > Ability to set table properties.
> > And many bugs have been fixed!
> >
> >  I propose that we release the following RC as the official PyIceberg
> 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
> >
> >
> > * This corresponds to the tag: pyiceberg-0.4.0rc1
> (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
> >
> > * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
> >
> > *
> https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
> >
> >
> > The release tarball, signature, and checksums are here:
> >
> >
> > * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
> >
> >
> > You can find the KEYS file here:
> >
> >
> > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
> >
> >
> > Convenience binary artifacts are staged on pypi:
> >
> >
> > https://pypi.org/project/pyiceberg/0.4.0rc1/
> >
> >
> > And can be installed using: pip3 install pyiceberg==0.4.0rc1
> >
> >
> > Please download, verify, and test.
> >
> >
> > Please vote in the next 72 hours.
> >
> > [ ] +1 Release this as PyIceberg 0.4.0
> >
> > [ ] +0
> >
> > [ ] -1 Do not release this because...
> >
> >
> > Please consider this email a +1 from my side:
> >
> > Ran some basic table scans
> >
> > Including tables with positional deletes
> >
> > Checked to see if everything still works when PyArrow is not installed
> > Set some table properties
> >
> > Kind regards,
> >
> > Fokko
>

Re: [VOTE] Release PyIceberg 0.4.0 RC1

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (non binding)

Regards
JB

On Mon, Jun 26, 2023 at 11:27 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> Hi Everyone,
>
>
> Excited to start the 0.4.0 PyIceberg release process. The 0.4.0 release is packed with cool features:
>
> Support for converting Parquet schemas into Iceberg ones
> Support for reading data using FSSpec.
> Support fetching a limited number of rows to quickly peek into a dataset.
> Reduced the number of calls to the object store with PyArrow>=12.0.0.
> Speed up queries using the Iceberg metrics.
> Ability to do SQL style filters: row_filter='passengers >= 3'.|
> SigV4 support for the REST catalog.
> A complete makeover of the docs site.
> Support for positional deletes.
> Ability to set table properties.
> And many bugs have been fixed!
>
>  I propose that we release the following RC as the official PyIceberg 0.4.0 release. The commit ID is e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> * This corresponds to the tag: pyiceberg-0.4.0rc1 (c3579a11b4bfa5387e313185e714c40a0ed1ccfe)
>
> * https://github.com/apache/iceberg/releases/tag/pyiceberg-0.4.0rc1
>
> * https://github.com/apache/iceberg/tree/e85ec9447c08c1a21e9ef21278f3237811f3f67f
>
>
> The release tarball, signature, and checksums are here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.4.0rc1/
>
>
> You can find the KEYS file here:
>
>
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on pypi:
>
>
> https://pypi.org/project/pyiceberg/0.4.0rc1/
>
>
> And can be installed using: pip3 install pyiceberg==0.4.0rc1
>
>
> Please download, verify, and test.
>
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as PyIceberg 0.4.0
>
> [ ] +0
>
> [ ] -1 Do not release this because...
>
>
> Please consider this email a +1 from my side:
>
> Ran some basic table scans
>
> Including tables with positional deletes
>
> Checked to see if everything still works when PyArrow is not installed
> Set some table properties
>
> Kind regards,
>
> Fokko