You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Sung Yun <su...@gmail.com> on 2024/08/27 14:56:32 UTC
[DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Hi folks,
We’ve recently had a few discussion threads on components we wanted to
expose as Python Bindings from the iceberg-rust project in the form of the
"pyiceberg_core" package.
Now that the Python bindings for the Transforms have been implemented, I
wanted to open this thread to discuss the first 'pyiceberg_core' release.
Here are some questions I wanted to discuss with the community:
1. Which features do we expose in the first release?
2. Should the version of the pyiceberg_core python package be aligned with
the version of iceberg-rust crate?
3. What is the release cadence we want to settle on in the future?
Here are my answers to the above questions, where my thoughts on the
versioning and release process are inspired by the current practice from
Apache Arrow community <https://arrow.apache.org/release/>:
1. Expose Transforms as Python bindings is ready to release, and we could
group in other features if they are also ready:
https://github.com/apache/iceberg-rust/pull/556
2. I think we could start with the current rust version (0.3.0). Aligning
it with the version of the iceberg-rust crate would make it easier to set
up a CI worklow that releases packages for all available bindings from the
same trigger.
3. We could publish the initial package manually to unblock PyIceberg, but
we should invest in a unified release process on github workflows that
publishes all enabled language bindings in the repository together.
I'm excited to hear your thoughts!
Sung
Prior Discussion Threads:
Transforms: https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
PR for exposing transforms as Python bindings:
https://github.com/apache/iceberg-rust/pull/556
Re: [DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Posted by Renjie Liu <li...@gmail.com>.
>
> OpenDAL tends to release all packages simultaneously but with different
> versions. We can try this approach as well.
I think it's a good approach. But I also agree that we could
release pyiceberg_core separately for now, and rethink the workflow when we
actually have more bindings.
On Thu, Aug 29, 2024 at 2:47 PM Xuanwo <xu...@apache.org> wrote:
> > Decoupling pyiceberg_core with iceberg-rust may be flexible, but my
> concern is that this may not be scalable when we introduce more language
> bindings. I think Xuanwo has a lot of experience when maintaining Apache
> OpenDAL <https://github.com/apache/opendal> ?
>
> OpenDAL tends to release all packages simultaneously but with different
> versions. We can try this approach as well.
>
> It's also fine for me to release pyiceberg_core separately for the first
> time.
>
> On Thu, Aug 29, 2024, at 10:51, Renjie Liu wrote:
>
> Hi:
>
> Thanks Sung for driving this.
>
> Which features do we expose in the first release?
>
>
> I think the transformation is a good start.
>
> Should the version of the pyiceberg_core python package be aligned with
> the version of iceberg-rust crate?
>
>
> Decoupling pyiceberg_core with iceberg-rust may be flexible, but my
> concern is that this may not be scalable when we introduce more language
> bindings. I think Xuanwo has a lot of experience when maintaining Apache
> OpenDAL <https://github.com/apache/opendal> ?
>
> What is the release cadence we want to settle on in the future?
>
>
> I agree that it's valuable to invest in automation workflow for this.
>
>
> On Wed, Aug 28, 2024 at 10:19 PM Fokko Driesprong <fo...@apache.org>
> wrote:
>
> Thanks for driving this Sung, this is very exciting!
>
> 1. The transforms are a good first thing to address.
> 2. I agree with Xuanwo, that for flexibility we can decouple them.
> 3. Automation is probably easier than doing it manually (otherwise we
> would have to document the steps).
>
> Kind regards,
> Fokko
>
> Op di 27 aug 2024 om 17:50 schreef Xuanwo <xu...@apache.org>:
> >
> > Thanks a lot for carrying this!
> >
> > 1. Which features do we expose in the first release?
> >
> >
> > No further comments on this. Pyiceberg is currently the sole user of
> pyiceberg_core. I'm willing to meet whatever pyiceberg's needs.
> >
> > 2. Should the version of the pyiceberg_core python package be aligned
> with the version of iceberg-rust crate?
> >
> >
> > I believe pyiceberg_core should have its own version. The release
> workflow should be very different from the existing Rust workflow, so we
> don't need to align the versions just for this.
> >
> > 3. We could publish the initial package manually to unblock PyIceberg,
> but we should invest in a unified release process on github workflows that
> publishes all enabled language bindings in the repository together.
> >
> >
> > Releasing a Rust-core-based Python package manually is very tedious;
> it's better to start with a GitHub workflow from the first version.
> >
> > I have experience in building OpenDAL Python workflows, and I'm willing
> to help establish the release workflow for PyIceberg Core.
> >
> > See:
> https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml
> >
> >
> > On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote:
> >
> > Hi folks,
> >
> > We’ve recently had a few discussion threads on components we wanted to
> expose as Python Bindings from the iceberg-rust project in the form of the
> "pyiceberg_core" package.
> >
> > Now that the Python bindings for the Transforms have been implemented, I
> wanted to open this thread to discuss the first 'pyiceberg_core' release.
> Here are some questions I wanted to discuss with the community:
> >
> > 1. Which features do we expose in the first release?
> > 2. Should the version of the pyiceberg_core python package be aligned
> with the version of iceberg-rust crate?
> > 3. What is the release cadence we want to settle on in the future?
> >
> > Here are my answers to the above questions, where my thoughts on the
> versioning and release process are inspired by the current practice from
> Apache Arrow community <https://arrow.apache.org/release/>:
> > 1. Expose Transforms as Python bindings is ready to release, and we
> could group in other features if they are also ready:
> https://github.com/apache/iceberg-rust/pull/556
> > 2. I think we could start with the current rust version (0.3.0).
> Aligning it with the version of the iceberg-rust crate would make it easier
> to set up a CI worklow that releases packages for all available bindings
> from the same trigger.
> > 3. We could publish the initial package manually to unblock PyIceberg,
> but we should invest in a unified release process on github workflows that
> publishes all enabled language bindings in the repository together.
> >
> > I'm excited to hear your thoughts!
> >
> > Sung
> >
> > Prior Discussion Threads:
> > Transforms:
> https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
> > FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
> >
> > PR for exposing transforms as Python bindings:
> https://github.com/apache/iceberg-rust/pull/556
> >
> > Xuanwo
> >
> > https://xuanwo.io/
> >
>
>
> Xuanwo
>
> https://xuanwo.io/
>
>
Re: [DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Posted by Xuanwo <xu...@apache.org>.
> Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern is that this may not be scalable when we introduce more language bindings. I think Xuanwo has a lot of experience when maintaining Apache OpenDAL <https://github.com/apache/opendal> ?
OpenDAL tends to release all packages simultaneously but with different versions. We can try this approach as well.
It's also fine for me to release pyiceberg_core separately for the first time.
On Thu, Aug 29, 2024, at 10:51, Renjie Liu wrote:
> Hi:
>
> Thanks Sung for driving this.
>
>> Which features do we expose in the first release?
>
> I think the transformation is a good start.
>
>> Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
>
> Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern is that this may not be scalable when we introduce more language bindings. I think Xuanwo has a lot of experience when maintaining Apache OpenDAL <https://github.com/apache/opendal> ?
>
>> What is the release cadence we want to settle on in the future?
>
> I agree that it's valuable to invest in automation workflow for this.
>
>
> On Wed, Aug 28, 2024 at 10:19 PM Fokko Driesprong <fo...@apache.org> wrote:
>> Thanks for driving this Sung, this is very exciting!
>>
>> 1. The transforms are a good first thing to address.
>> 2. I agree with Xuanwo, that for flexibility we can decouple them.
>> 3. Automation is probably easier than doing it manually (otherwise we
>> would have to document the steps).
>>
>> Kind regards,
>> Fokko
>>
>> Op di 27 aug 2024 om 17:50 schreef Xuanwo <xu...@apache.org>:
>> >
>> > Thanks a lot for carrying this!
>> >
>> > 1. Which features do we expose in the first release?
>> >
>> >
>> > No further comments on this. Pyiceberg is currently the sole user of pyiceberg_core. I'm willing to meet whatever pyiceberg's needs.
>> >
>> > 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
>> >
>> >
>> > I believe pyiceberg_core should have its own version. The release workflow should be very different from the existing Rust workflow, so we don't need to align the versions just for this.
>> >
>> > 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
>> >
>> >
>> > Releasing a Rust-core-based Python package manually is very tedious; it's better to start with a GitHub workflow from the first version.
>> >
>> > I have experience in building OpenDAL Python workflows, and I'm willing to help establish the release workflow for PyIceberg Core.
>> >
>> > See: https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml
>> >
>> >
>> > On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote:
>> >
>> > Hi folks,
>> >
>> > We’ve recently had a few discussion threads on components we wanted to expose as Python Bindings from the iceberg-rust project in the form of the "pyiceberg_core" package.
>> >
>> > Now that the Python bindings for the Transforms have been implemented, I wanted to open this thread to discuss the first 'pyiceberg_core' release. Here are some questions I wanted to discuss with the community:
>> >
>> > 1. Which features do we expose in the first release?
>> > 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
>> > 3. What is the release cadence we want to settle on in the future?
>> >
>> > Here are my answers to the above questions, where my thoughts on the versioning and release process are inspired by the current practice from Apache Arrow community <https://arrow.apache.org/release/>:
>> > 1. Expose Transforms as Python bindings is ready to release, and we could group in other features if they are also ready: https://github.com/apache/iceberg-rust/pull/556
>> > 2. I think we could start with the current rust version (0.3.0). Aligning it with the version of the iceberg-rust crate would make it easier to set up a CI worklow that releases packages for all available bindings from the same trigger.
>> > 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
>> >
>> > I'm excited to hear your thoughts!
>> >
>> > Sung
>> >
>> > Prior Discussion Threads:
>> > Transforms: https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
>> > FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
>> >
>> > PR for exposing transforms as Python bindings: https://github.com/apache/iceberg-rust/pull/556
>> >
>> > Xuanwo
>> >
>> > https://xuanwo.io/
>> >
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Posted by Renjie Liu <li...@gmail.com>.
Hi:
Thanks Sung for driving this.
Which features do we expose in the first release?
I think the transformation is a good start.
Should the version of the pyiceberg_core python package be aligned with the
> version of iceberg-rust crate?
Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern
is that this may not be scalable when we introduce more language bindings.
I think Xuanwo has a lot of experience when maintaining Apache OpenDAL
<https://github.com/apache/opendal> ?
What is the release cadence we want to settle on in the future?
I agree that it's valuable to invest in automation workflow for this.
On Wed, Aug 28, 2024 at 10:19 PM Fokko Driesprong <fo...@apache.org> wrote:
> Thanks for driving this Sung, this is very exciting!
>
> 1. The transforms are a good first thing to address.
> 2. I agree with Xuanwo, that for flexibility we can decouple them.
> 3. Automation is probably easier than doing it manually (otherwise we
> would have to document the steps).
>
> Kind regards,
> Fokko
>
> Op di 27 aug 2024 om 17:50 schreef Xuanwo <xu...@apache.org>:
> >
> > Thanks a lot for carrying this!
> >
> > 1. Which features do we expose in the first release?
> >
> >
> > No further comments on this. Pyiceberg is currently the sole user of
> pyiceberg_core. I'm willing to meet whatever pyiceberg's needs.
> >
> > 2. Should the version of the pyiceberg_core python package be aligned
> with the version of iceberg-rust crate?
> >
> >
> > I believe pyiceberg_core should have its own version. The release
> workflow should be very different from the existing Rust workflow, so we
> don't need to align the versions just for this.
> >
> > 3. We could publish the initial package manually to unblock PyIceberg,
> but we should invest in a unified release process on github workflows that
> publishes all enabled language bindings in the repository together.
> >
> >
> > Releasing a Rust-core-based Python package manually is very tedious;
> it's better to start with a GitHub workflow from the first version.
> >
> > I have experience in building OpenDAL Python workflows, and I'm willing
> to help establish the release workflow for PyIceberg Core.
> >
> > See:
> https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml
> >
> >
> > On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote:
> >
> > Hi folks,
> >
> > We’ve recently had a few discussion threads on components we wanted to
> expose as Python Bindings from the iceberg-rust project in the form of the
> "pyiceberg_core" package.
> >
> > Now that the Python bindings for the Transforms have been implemented, I
> wanted to open this thread to discuss the first 'pyiceberg_core' release.
> Here are some questions I wanted to discuss with the community:
> >
> > 1. Which features do we expose in the first release?
> > 2. Should the version of the pyiceberg_core python package be aligned
> with the version of iceberg-rust crate?
> > 3. What is the release cadence we want to settle on in the future?
> >
> > Here are my answers to the above questions, where my thoughts on the
> versioning and release process are inspired by the current practice from
> Apache Arrow community <https://arrow.apache.org/release/>:
> > 1. Expose Transforms as Python bindings is ready to release, and we
> could group in other features if they are also ready:
> https://github.com/apache/iceberg-rust/pull/556
> > 2. I think we could start with the current rust version (0.3.0).
> Aligning it with the version of the iceberg-rust crate would make it easier
> to set up a CI worklow that releases packages for all available bindings
> from the same trigger.
> > 3. We could publish the initial package manually to unblock PyIceberg,
> but we should invest in a unified release process on github workflows that
> publishes all enabled language bindings in the repository together.
> >
> > I'm excited to hear your thoughts!
> >
> > Sung
> >
> > Prior Discussion Threads:
> > Transforms:
> https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
> > FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
> >
> > PR for exposing transforms as Python bindings:
> https://github.com/apache/iceberg-rust/pull/556
> >
> > Xuanwo
> >
> > https://xuanwo.io/
> >
>
Re: [DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Posted by Fokko Driesprong <fo...@apache.org>.
Thanks for driving this Sung, this is very exciting!
1. The transforms are a good first thing to address.
2. I agree with Xuanwo, that for flexibility we can decouple them.
3. Automation is probably easier than doing it manually (otherwise we
would have to document the steps).
Kind regards,
Fokko
Op di 27 aug 2024 om 17:50 schreef Xuanwo <xu...@apache.org>:
>
> Thanks a lot for carrying this!
>
> 1. Which features do we expose in the first release?
>
>
> No further comments on this. Pyiceberg is currently the sole user of pyiceberg_core. I'm willing to meet whatever pyiceberg's needs.
>
> 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
>
>
> I believe pyiceberg_core should have its own version. The release workflow should be very different from the existing Rust workflow, so we don't need to align the versions just for this.
>
> 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
>
>
> Releasing a Rust-core-based Python package manually is very tedious; it's better to start with a GitHub workflow from the first version.
>
> I have experience in building OpenDAL Python workflows, and I'm willing to help establish the release workflow for PyIceberg Core.
>
> See: https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml
>
>
> On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote:
>
> Hi folks,
>
> We’ve recently had a few discussion threads on components we wanted to expose as Python Bindings from the iceberg-rust project in the form of the "pyiceberg_core" package.
>
> Now that the Python bindings for the Transforms have been implemented, I wanted to open this thread to discuss the first 'pyiceberg_core' release. Here are some questions I wanted to discuss with the community:
>
> 1. Which features do we expose in the first release?
> 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
> 3. What is the release cadence we want to settle on in the future?
>
> Here are my answers to the above questions, where my thoughts on the versioning and release process are inspired by the current practice from Apache Arrow community <https://arrow.apache.org/release/>:
> 1. Expose Transforms as Python bindings is ready to release, and we could group in other features if they are also ready: https://github.com/apache/iceberg-rust/pull/556
> 2. I think we could start with the current rust version (0.3.0). Aligning it with the version of the iceberg-rust crate would make it easier to set up a CI worklow that releases packages for all available bindings from the same trigger.
> 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
>
> I'm excited to hear your thoughts!
>
> Sung
>
> Prior Discussion Threads:
> Transforms: https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
> FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
>
> PR for exposing transforms as Python bindings: https://github.com/apache/iceberg-rust/pull/556
>
> Xuanwo
>
> https://xuanwo.io/
>
Re: [DISCUSS] iceberg-rust: pyiceberg_core 0.1.0 Release
Posted by Xuanwo <xu...@apache.org>.
Thanks a lot for carrying this!
> 1. Which features do we expose in the first release?
No further comments on this. Pyiceberg is currently the sole user of pyiceberg_core. I'm willing to meet whatever pyiceberg's needs.
> 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
I believe pyiceberg_core should have its own version. The release workflow should be very different from the existing Rust workflow, so we don't need to align the versions just for this.
> 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
Releasing a Rust-core-based Python package manually is very tedious; it's better to start with a GitHub workflow from the first version.
I have experience in building OpenDAL Python workflows, and I'm willing to help establish the release workflow for PyIceberg Core.
See: https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml
On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote:
> Hi folks,
>
> We’ve recently had a few discussion threads on components we wanted to expose as Python Bindings from the iceberg-rust project in the form of the "pyiceberg_core" package.
>
> Now that the Python bindings for the Transforms have been implemented, I wanted to open this thread to discuss the first 'pyiceberg_core' release. Here are some questions I wanted to discuss with the community:
>
> 1. Which features do we expose in the first release?
> 2. Should the version of the pyiceberg_core python package be aligned with the version of iceberg-rust crate?
> 3. What is the release cadence we want to settle on in the future?
>
> Here are my answers to the above questions, where my thoughts on the versioning and release process are inspired by the current practice from Apache Arrow community <https://arrow.apache.org/release/>:
> 1. Expose Transforms as Python bindings is ready to release, and we could group in other features if they are also ready: https://github.com/apache/iceberg-rust/pull/556
> 2. I think we could start with the current rust version (0.3.0). Aligning it with the version of the iceberg-rust crate would make it easier to set up a CI worklow that releases packages for all available bindings from the same trigger.
> 3. We could publish the initial package manually to unblock PyIceberg, but we should invest in a unified release process on github workflows that publishes all enabled language bindings in the repository together.
>
> I'm excited to hear your thoughts!
>
> Sung
>
> Prior Discussion Threads:
> Transforms: https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss
> FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc
>
> PR for exposing transforms as Python bindings: https://github.com/apache/iceberg-rust/pull/556
Xuanwo
https://xuanwo.io/