You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Xuanwo <xu...@apache.org> on 2024/08/01 02:28:52 UTC
[DISCUSS] Use iceberg-rust as pyiceberg file io
Hello everyone
I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
Welcome to join in the discussion.
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Ryan Blue <bl...@databricks.com.INVALID>.
I would vote for a Python FileIO implementation using OpenDAL. FileIO tends
to be a simpler interface and we already have support for multiple options
(PyArrow and FSSpec). This would give users the choice.
On Fri, Aug 2, 2024 at 7:53 AM Renjie Liu <li...@gmail.com> wrote:
> Hi:
>
> Thanks Xuanwo for raising this.
>
> As mentioned in another thread, I think using iceberg-rust in pyiceberg is
> a good idea.
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
> I think this is a good first step moving forward to make pyiceberg backed
> iceberg-rust. In the future we can replace components gradually.
>
> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>
>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>> Hi, Raschkowski, good question!
>>
>> It's possible. There is an ongoing project developing fsspec bindings for
>> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
>> directly use opendal through fsspec.
>>
>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
>> be able to use opendalfs as an alternative implementation of the fsspec
>> AbstractFileSystem class.
>>
>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>
>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>>
>> ------------------------------
>>
>> *From:* Joe Stein <cr...@gmail.com>
>> *Sent:* Thursday, August 1, 2024 3:37 AM
>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>
>> *CAUTION:* This email originates from an external party (outside of
>> Palantir). If you believe this message is suspicious in nature, please use
>> the "Report Message" button built into Outlook.
>>
>> Kafka did this with librdkafka and was wildly successful. The underlying
>> bindings being in rust are great with a layer for access in Python +1
>>
>>
>> ~ Joe Stein
>>
>>
>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>
>> Hello everyone
>>
>> I start this thread to discuss the idea about using iceberg-rust as
>> pyiceberg file io.
>>
>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>> [hackmd.io]
>> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>
>> In summary, we can leverage the work from iceberg-rust to help pyiceberg
>> in developing a fast and compact file IO system that benefits users with
>> specific constraints.
>>
>> Welcome to join in the discussion.
>>
>> Xuanwo
>>
>> https://xuanwo.io/ [xuanwo.io]
>> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>>
--
Ryan Blue
Databricks
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Kevin Liu <ke...@gmail.com>.
I had some ideas to refactor the current FileIO implementations in
PyIceberg to consolidate the behaviors for FsSpec and PyArrow.
https://github.com/apache/iceberg-python/issues/310
There are also some additional concerns around URI parsing based on the
specific FileIO implementation.
Perhaps a good litmus test for a well-defined FileIO interface is to
introduce nanoarrow
<https://arrow.apache.org/nanoarrow/latest/getting-started/python.html> and
see what breaks.
Thanks,
Kevin Liu
On Tue, Aug 6, 2024 at 11:21 AM Kevin Liu <ke...@gmail.com> wrote:
> > First, we need to establish a workflow that allows us to gradually
> integrate new features into pyiceberg-core. Additionally, pyiceberg should
> be able to import and optionally use classes from pyiceberg-core in an
> additive manner. While developing this workflow, our community will learn
> how to collaborate, manage releases, and more.
>
> +1 I would like to learn more about how to integrate pyiceberg-core into
> PyIceberg. The initial setup should give us a framework for future
> integrations.
>
> I also think there's some prerequisite work on the Pyiceberg side to
> clean up FileIO. A lot of features are built specifically with PyArrow
> dependency, such as writing to the table.
>
>
> Thanks,
> Kevin
>
>
> On Mon, Aug 5, 2024 at 12:19 AM Xuanwo <xu...@apache.org> wrote:
>
>> For the FileIO part, just curious—since Rust's FileIO currently also uses
>> OpenDAL, will there be any functional differences in terms of supported
>> storage services or configurations (like profile_name, signer, etc.)
>> compared to using opendalfs directly in Python in the future? Will Rust's
>> FileIO introduce any customizations/optimizations/extensions beyond what
>> OpenDAL supports?
>>
>>
>> Hi, Honah
>>
>> I believe there should be no functional differences. We can implement the
>> exact same thing for both pyiceberg_core FileIO and opendalfs fsspec FileIO.
>>
>> The main difference I've noticed is in where the configuration parsing
>> occurs.
>>
>> The pyiceberg_core FileIO directly exposes the FileIO class, which can
>> inherently understand iceberg properties. We can pass these properties
>> directly to initialize file IO without any additional effort on the
>> pyiceberg side.
>>
>> However, for opendalfs fsspec FileIO, we need to parse the properties and
>> convert them into appropriate opendalfs options for it to function properly.
>>
>> On Mon, Aug 5, 2024, at 15:04, Honah J. wrote:
>>
>> Thanks Xuanwo for driving this and everyone for discussing,
>>
>> I like the idea of pushing down low-level logic to Iceberg-rust
>> (pyiceberg_core). It’s great to have another option besides PyArrow for
>> reading and writing data in PyIceberg. Thanks, Xuanwo, for moving this
>> forward with the initial PR to add pyiceberg_core.
>>
>> For the FileIO part, just curious—since Rust's FileIO currently also uses
>> OpenDAL, will there be any functional differences in terms of supported
>> storage services or configurations (like profile_name, signer, etc.)
>> compared to using opendalfs directly in Python in the future? Will Rust's
>> FileIO introduce any customizations/optimizations/extensions beyond what
>> OpenDAL supports?
>>
>> Best regards,
>> Honah
>>
>>
>>
>> On Sat, Aug 3, 2024 at 4:12 PM timoguin@proton.me.INVALID
>> <ti...@proton.me.invalid> wrote:
>>
>> Fantastic work! I think this is a great direction, and this provides a
>> good base to start iterating.
>>
>> It makes the most sense to me for the Python bindings (and others) to
>> live in the same repo as iceberg-rust, especially at this early stage.
>>
>> - Tim O'Guin
>>
>>
>> -------- Original Message --------
>> On 8/3/24 12:33 AM, Xuanwo wrote:
>>
>>
>> Let's rock! Welcome to take a review:
>> https://github.com/apache/iceberg-rust/pull/518
>>
>> On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
>>
>> I also support integrating iceberg-rust with pyiceberg rather than
>> building something new on OpenDAL.
>>
>> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the
>> native fsspec support for OpenDAL, is ready. Users can use opendalfs as a
>> FileIO class directly in pure python. It's not an action item for our
>> community to take.
>>
>> The consensus we've reached is that iceberg-rust will be the core of
>> PyIceberg. The main question now is "How?" How can we implement it without
>> disrupting our valued users? This is my top priority.
>>
>> *Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg
>> core as `pyiceberg-core` until we decide on a project name.*
>>
>> First, we need to establish a workflow that allows us to gradually
>> integrate new features into pyiceberg-core. Additionally, pyiceberg should
>> be able to import and optionally use classes from pyiceberg-core in an
>> additive manner. While developing this workflow, our community will learn
>> how to collaborate, manage releases, and more.
>>
>> We will then incorporate additional Rust-backed features into
>> pyiceberg-core. Eventually, we may make pyiceberg-core our default
>> implementation.
>>
>> My current plan is to implement this pyiceberg-core under iceberg-rust
>> repo under `bindings/python`.
>>
>> - Iceberg-rust is currently under active development. I plan to release
>> pyiceberg-core independently of iceberg-rust's release, as they feature
>> distinct public APIs (and languages!).
>> - Most of the work involves maintaining a few Python stubs and classes,
>> with the majority related to Rust.
>> - The python integration is just a start: we can expect `bindings/nodejs`
>> to happen here too.
>>
>> The setup work has already been started. I will update my PR here once
>> it's ready to review.
>>
>> [1]: https://github.com/fsspec/opendalfs
>>
>> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>>
>> Hi:
>>
>> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's
>> FileIO, rather than directly using OpenDAL. The motivation is that we can
>> use this as a starting point of providing iceberg-rust backed components
>> for pyiceberg, and due to its simplicity, it's a good case. I believe there
>> will be more cases, like Sung mentioned transform in another thread, and
>> table scan mentioned by Fokko.
>>
>> If we want to use OpenDAL directly, we don't need iceberg-rust, since
>> OpenDAL already has python binding:
>> https://opendal.apache.org/docs/python/opendal.html
>>
>> Do you have any experience with this? I see many projects having Rust and
>> Python code in a single repository. There are some exceptions like
>> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
>> <https://github.com/pydantic/pydantic-core>).
>>
>>
>> Well, first I want to say providing a python binding for a library
>> written in rust is a quite common practice. Just to name a few: opendal
>> <https://github.com/apache/opendal>, polars
>> <https://github.com/pola-rs/polars>, datafusion
>> <https://github.com/apache/datafusion>, delta-rs
>> <https://github.com/delta-io/delta-rs>. As far as I know, most of them
>> choose to put python binding with rust in the same repo, only
>> datafusion-python <https://github.com/apache/datafusion-python> lives in
>> another, I'm not sure about the reason, maybe it's too large?
>>
>> I haven't tried to implement one before, but pyo3
>> <https://github.com/PyO3> has great documentation, and there are many
>> existing examples in open source we can learn with.
>>
>> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>>
>> One more thing,
>>
>> About this idea, would you have a more detailed design? For example,
>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>
>>
>> Do you have any experience with this? I see many projects having Rust and
>> Python code in a single repository. There are some exceptions like
>> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
>> <https://github.com/pydantic/pydantic-core>).
>>
>> Kind regards,
>> Fokko
>>
>>
>>
>> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>>
>> Thanks for driving this Xuanwo,
>>
>> I already suggested this in my talk back at the Spark Summit to see if we
>> can spark some interest, and it is exciting to see this materialize.
>>
>> For the IO abstraction, I think the FileIO is the best option. We already
>> have the interface
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
>> in PyIceberg, and also a PyArrowFileIO
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
>> I must admit that the abstraction is less clear in PyIceberg since we rely
>> so much on Arrow for reading/writing data that it is tightly coupled. I
>> would love to see if we can use OpenDAL for reading/writing data, and
>> Iceberg-rust for pushing down the low-level logic. A while ago I did some
>> profiling on the code, and one of the major issues is that Arrow doesn't
>> support proper field-ID projection. Therefore we have to the Parquet file,
>> and do the schema-evolution and type promotion afterwards in Python
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
>> which causes a lot of congestion on the GIL.
>>
>> Kind regards,
>> Fokko
>>
>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>
>> +1 for an OpenDALFileIO
>>
>> -Jack
>>
>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>
>>
>> Hi, renjie
>>
>> Thank you for your support. I'll delve into the details and first build a
>> PoC PR to make it clear.
>>
>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>
>> Hi:
>>
>> Thanks Xuanwo for raising this.
>>
>> As mentioned in another thread, I think using iceberg-rust in pyiceberg
>> is a good idea.
>>
>> About this idea, would you have a more detailed design? For example,
>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>
>> I think this is a good first step moving forward to make pyiceberg backed
>> iceberg-rust. In the future we can replace components gradually.
>>
>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>
>>
>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>> Hi, Raschkowski, good question!
>>
>> It's possible. There is an ongoing project developing fsspec bindings for
>> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
>> directly use opendal through fsspec.
>>
>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
>> be able to use opendalfs as an alternative implementation of the fsspec
>> AbstractFileSystem class.
>>
>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>
>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>>
>> ------------------------------
>>
>> *From:* Joe Stein <cr...@gmail.com>
>> *Sent:* Thursday, August 1, 2024 3:37 AM
>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>
>> *CAUTION:* This email originates from an external party (outside of
>> Palantir). If you believe this message is suspicious in nature, please use
>> the "Report Message" button built into Outlook.
>>
>> Kafka did this with librdkafka and was wildly successful. The underlying
>> bindings being in rust are great with a layer for access in Python +1
>>
>>
>> ~ Joe Stein
>>
>>
>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>
>> Hello everyone
>>
>> I start this thread to discuss the idea about using iceberg-rust as
>> pyiceberg file io.
>>
>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>> [hackmd.io]
>> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>
>> In summary, we can leverage the work from iceberg-rust to help pyiceberg
>> in developing a fast and compact file IO system that benefits users with
>> specific constraints.
>>
>> Welcome to join in the discussion.
>>
>> Xuanwo
>>
>> https://xuanwo.io/ [xuanwo.io]
>> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Kevin Liu <ke...@gmail.com>.
> First, we need to establish a workflow that allows us to gradually
integrate new features into pyiceberg-core. Additionally, pyiceberg should
be able to import and optionally use classes from pyiceberg-core in an
additive manner. While developing this workflow, our community will learn
how to collaborate, manage releases, and more.
+1 I would like to learn more about how to integrate pyiceberg-core into
PyIceberg. The initial setup should give us a framework for future
integrations.
I also think there's some prerequisite work on the Pyiceberg side to clean
up FileIO. A lot of features are built specifically with PyArrow
dependency, such as writing to the table.
Thanks,
Kevin
On Mon, Aug 5, 2024 at 12:19 AM Xuanwo <xu...@apache.org> wrote:
> For the FileIO part, just curious—since Rust's FileIO currently also uses
> OpenDAL, will there be any functional differences in terms of supported
> storage services or configurations (like profile_name, signer, etc.)
> compared to using opendalfs directly in Python in the future? Will Rust's
> FileIO introduce any customizations/optimizations/extensions beyond what
> OpenDAL supports?
>
>
> Hi, Honah
>
> I believe there should be no functional differences. We can implement the
> exact same thing for both pyiceberg_core FileIO and opendalfs fsspec FileIO.
>
> The main difference I've noticed is in where the configuration parsing
> occurs.
>
> The pyiceberg_core FileIO directly exposes the FileIO class, which can
> inherently understand iceberg properties. We can pass these properties
> directly to initialize file IO without any additional effort on the
> pyiceberg side.
>
> However, for opendalfs fsspec FileIO, we need to parse the properties and
> convert them into appropriate opendalfs options for it to function properly.
>
> On Mon, Aug 5, 2024, at 15:04, Honah J. wrote:
>
> Thanks Xuanwo for driving this and everyone for discussing,
>
> I like the idea of pushing down low-level logic to Iceberg-rust
> (pyiceberg_core). It’s great to have another option besides PyArrow for
> reading and writing data in PyIceberg. Thanks, Xuanwo, for moving this
> forward with the initial PR to add pyiceberg_core.
>
> For the FileIO part, just curious—since Rust's FileIO currently also uses
> OpenDAL, will there be any functional differences in terms of supported
> storage services or configurations (like profile_name, signer, etc.)
> compared to using opendalfs directly in Python in the future? Will Rust's
> FileIO introduce any customizations/optimizations/extensions beyond what
> OpenDAL supports?
>
> Best regards,
> Honah
>
>
>
> On Sat, Aug 3, 2024 at 4:12 PM timoguin@proton.me.INVALID
> <ti...@proton.me.invalid> wrote:
>
> Fantastic work! I think this is a great direction, and this provides a
> good base to start iterating.
>
> It makes the most sense to me for the Python bindings (and others) to live
> in the same repo as iceberg-rust, especially at this early stage.
>
> - Tim O'Guin
>
>
> -------- Original Message --------
> On 8/3/24 12:33 AM, Xuanwo wrote:
>
>
> Let's rock! Welcome to take a review:
> https://github.com/apache/iceberg-rust/pull/518
>
> On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
>
> I also support integrating iceberg-rust with pyiceberg rather than
> building something new on OpenDAL.
>
> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the
> native fsspec support for OpenDAL, is ready. Users can use opendalfs as a
> FileIO class directly in pure python. It's not an action item for our
> community to take.
>
> The consensus we've reached is that iceberg-rust will be the core of
> PyIceberg. The main question now is "How?" How can we implement it without
> disrupting our valued users? This is my top priority.
>
> *Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg
> core as `pyiceberg-core` until we decide on a project name.*
>
> First, we need to establish a workflow that allows us to gradually
> integrate new features into pyiceberg-core. Additionally, pyiceberg should
> be able to import and optionally use classes from pyiceberg-core in an
> additive manner. While developing this workflow, our community will learn
> how to collaborate, manage releases, and more.
>
> We will then incorporate additional Rust-backed features into
> pyiceberg-core. Eventually, we may make pyiceberg-core our default
> implementation.
>
> My current plan is to implement this pyiceberg-core under iceberg-rust
> repo under `bindings/python`.
>
> - Iceberg-rust is currently under active development. I plan to release
> pyiceberg-core independently of iceberg-rust's release, as they feature
> distinct public APIs (and languages!).
> - Most of the work involves maintaining a few Python stubs and classes,
> with the majority related to Rust.
> - The python integration is just a start: we can expect `bindings/nodejs`
> to happen here too.
>
> The setup work has already been started. I will update my PR here once
> it's ready to review.
>
> [1]: https://github.com/fsspec/opendalfs
>
> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>
> Hi:
>
> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's
> FileIO, rather than directly using OpenDAL. The motivation is that we can
> use this as a starting point of providing iceberg-rust backed components
> for pyiceberg, and due to its simplicity, it's a good case. I believe there
> will be more cases, like Sung mentioned transform in another thread, and
> table scan mentioned by Fokko.
>
> If we want to use OpenDAL directly, we don't need iceberg-rust, since
> OpenDAL already has python binding:
> https://opendal.apache.org/docs/python/opendal.html
>
> Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
>
>
> Well, first I want to say providing a python binding for a library
> written in rust is a quite common practice. Just to name a few: opendal
> <https://github.com/apache/opendal>, polars
> <https://github.com/pola-rs/polars>, datafusion
> <https://github.com/apache/datafusion>, delta-rs
> <https://github.com/delta-io/delta-rs>. As far as I know, most of them
> choose to put python binding with rust in the same repo, only
> datafusion-python <https://github.com/apache/datafusion-python> lives in
> another, I'm not sure about the reason, maybe it's too large?
>
> I haven't tried to implement one before, but pyo3
> <https://github.com/PyO3> has great documentation, and there are many
> existing examples in open source we can learn with.
>
> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> One more thing,
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
>
> Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
>
> Kind regards,
> Fokko
>
>
>
> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>
> Thanks for driving this Xuanwo,
>
> I already suggested this in my talk back at the Spark Summit to see if we
> can spark some interest, and it is exciting to see this materialize.
>
> For the IO abstraction, I think the FileIO is the best option. We already
> have the interface
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
> in PyIceberg, and also a PyArrowFileIO
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
> I must admit that the abstraction is less clear in PyIceberg since we rely
> so much on Arrow for reading/writing data that it is tightly coupled. I
> would love to see if we can use OpenDAL for reading/writing data, and
> Iceberg-rust for pushing down the low-level logic. A while ago I did some
> profiling on the code, and one of the major issues is that Arrow doesn't
> support proper field-ID projection. Therefore we have to the Parquet file,
> and do the schema-evolution and type promotion afterwards in Python
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
> which causes a lot of congestion on the GIL.
>
> Kind regards,
> Fokko
>
> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>
> +1 for an OpenDALFileIO
>
> -Jack
>
> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>
>
> Hi, renjie
>
> Thank you for your support. I'll delve into the details and first build a
> PoC PR to make it clear.
>
> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>
> Hi:
>
> Thanks Xuanwo for raising this.
>
> As mentioned in another thread, I think using iceberg-rust in pyiceberg is
> a good idea.
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
> I think this is a good first step moving forward to make pyiceberg backed
> iceberg-rust. In the future we can replace components gradually.
>
> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>
>
> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
> Hi, Raschkowski, good question!
>
> It's possible. There is an ongoing project developing fsspec bindings for
> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
> directly use opendal through fsspec.
>
> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
> be able to use opendalfs as an alternative implementation of the fsspec
> AbstractFileSystem class.
>
> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
>
> ------------------------------
>
> *From:* Joe Stein <cr...@gmail.com>
> *Sent:* Thursday, August 1, 2024 3:37 AM
> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>
> *CAUTION:* This email originates from an external party (outside of
> Palantir). If you believe this message is suspicious in nature, please use
> the "Report Message" button built into Outlook.
>
> Kafka did this with librdkafka and was wildly successful. The underlying
> bindings being in rust are great with a layer for access in Python +1
>
>
> ~ Joe Stein
>
>
> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>
> Hello everyone
>
> I start this thread to discuss the idea about using iceberg-rust as
> pyiceberg file io.
>
> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
> [hackmd.io]
> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>
> In summary, we can leverage the work from iceberg-rust to help pyiceberg
> in developing a fast and compact file IO system that benefits users with
> specific constraints.
>
> Welcome to join in the discussion.
>
> Xuanwo
>
> https://xuanwo.io/ [xuanwo.io]
> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
>
> Xuanwo
>
> https://xuanwo.io/
>
>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Xuanwo <xu...@apache.org>.
> For the FileIO part, just curious—since Rust's FileIO currently also uses OpenDAL, will there be any functional differences in terms of supported storage services or configurations (like profile_name, signer, etc.) compared to using opendalfs directly in Python in the future? Will Rust's FileIO introduce any customizations/optimizations/extensions beyond what OpenDAL supports?
Hi, Honah
I believe there should be no functional differences. We can implement the exact same thing for both pyiceberg_core FileIO and opendalfs fsspec FileIO.
The main difference I've noticed is in where the configuration parsing occurs.
The pyiceberg_core FileIO directly exposes the FileIO class, which can inherently understand iceberg properties. We can pass these properties directly to initialize file IO without any additional effort on the pyiceberg side.
However, for opendalfs fsspec FileIO, we need to parse the properties and convert them into appropriate opendalfs options for it to function properly.
On Mon, Aug 5, 2024, at 15:04, Honah J. wrote:
> Thanks Xuanwo for driving this and everyone for discussing,
>
> I like the idea of pushing down low-level logic to Iceberg-rust (pyiceberg_core). It’s great to have another option besides PyArrow for reading and writing data in PyIceberg. Thanks, Xuanwo, for moving this forward with the initial PR to add pyiceberg_core.
>
> For the FileIO part, just curious—since Rust's FileIO currently also uses OpenDAL, will there be any functional differences in terms of supported storage services or configurations (like profile_name, signer, etc.) compared to using opendalfs directly in Python in the future? Will Rust's FileIO introduce any customizations/optimizations/extensions beyond what OpenDAL supports?
>
> Best regards,
> Honah
>
>
>
> On Sat, Aug 3, 2024 at 4:12 PM timoguin@proton.me.INVALID <ti...@proton.me.invalid> wrote:
>> Fantastic work! I think this is a great direction, and this provides a good base to start iterating.
>>
>> It makes the most sense to me for the Python bindings (and others) to live in the same repo as iceberg-rust, especially at this early stage.
>>
>> - Tim O'Guin
>>
>>
>>
>> -------- Original Message --------
>> On 8/3/24 12:33 AM, Xuanwo __ wrote:
>>> __
>>> Let's rock! Welcome to take a review: https://github.com/apache/iceberg-rust/pull/518
>>>
>>> On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
>>>> I also support integrating iceberg-rust with pyiceberg rather than building something new on OpenDAL.
>>>>
>>>> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the native fsspec support for OpenDAL, is ready. Users can use opendalfs as a FileIO class directly in pure python. It's not an action item for our community to take.
>>>>
>>>> The consensus we've reached is that iceberg-rust will be the core of PyIceberg. The main question now is "How?" How can we implement it without disrupting our valued users? This is my top priority.
>>>>
>>>> *Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg core as `*pyiceberg-core*` until we decide on a project name.*
>>>>
>>>> First, we need to establish a workflow that allows us to gradually integrate new features into pyiceberg-core. Additionally, pyiceberg should be able to import and optionally use classes from pyiceberg-core in an additive manner. While developing this workflow, our community will learn how to collaborate, manage releases, and more.
>>>>
>>>> We will then incorporate additional Rust-backed features into pyiceberg-core. Eventually, we may make pyiceberg-core our default implementation.
>>>>
>>>> My current plan is to implement this pyiceberg-core under iceberg-rust repo under `bindings/python`.
>>>>
>>>> - Iceberg-rust is currently under active development. I plan to release pyiceberg-core independently of iceberg-rust's release, as they feature distinct public APIs (and languages!).
>>>> - Most of the work involves maintaining a few Python stubs and classes, with the majority related to Rust.
>>>> - The python integration is just a start: we can expect `bindings/nodejs` to happen here too.
>>>>
>>>> The setup work has already been started. I will update my PR here once it's ready to review.
>>>>
>>>> [1]: https://github.com/fsspec/opendalfs
>>>>
>>>> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>>>>> Hi:
>>>>>
>>>>> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's FileIO, rather than directly using OpenDAL. The motivation is that we can use this as a starting point of providing iceberg-rust backed components for pyiceberg, and due to its simplicity, it's a good case. I believe there will be more cases, like Sung mentioned transform in another thread, and table scan mentioned by Fokko.
>>>>>
>>>>> If we want to use OpenDAL directly, we don't need iceberg-rust, since OpenDAL already has python binding: https://opendal.apache.org/docs/python/opendal.html
>>>>>
>>>>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>>>>>
>>>>> Well, first I want to say providing a python binding for a library written in rust is a quite common practice. Just to name a few: opendal <https://github.com/apache/opendal>, polars <https://github.com/pola-rs/polars>, datafusion <https://github.com/apache/datafusion>, delta-rs <https://github.com/delta-io/delta-rs>. As far as I know, most of them choose to put python binding with rust in the same repo, only datafusion-python <https://github.com/apache/datafusion-python> lives in another, I'm not sure about the reason, maybe it's too large?
>>>>>
>>>>> I haven't tried to implement one before, but pyo3 <https://github.com/PyO3> has great documentation, and there are many existing examples in open source we can learn with.
>>>>>
>>>>> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>>>>>> One more thing,
>>>>>>
>>>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>>>
>>>>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>>>>>>
>>>>>> Kind regards,
>>>>>> Fokko
>>>>>>
>>>>>>
>>>>>>
>>>>>> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>>>>>>> Thanks for driving this Xuanwo,
>>>>>>>
>>>>>>> I already suggested this in my talk back at the Spark Summit to see if we can spark some interest, and it is exciting to see this materialize.
>>>>>>>
>>>>>>> For the IO abstraction, I think the FileIO is the best option. We already have the interface <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239> in PyIceberg, and also a PyArrowFileIO <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>. I must admit that the abstraction is less clear in PyIceberg since we rely so much on Arrow for reading/writing data that it is tightly coupled. I would love to see if we can use OpenDAL for reading/writing data, and Iceberg-rust for pushing down the low-level logic. A while ago I did some profiling on the code, and one of the major issues is that Arrow doesn't support proper field-ID projection. Therefore we have to the Parquet file, and do the schema-evolution and type promotion afterwards in Python <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>, which causes a lot of congestion on the GIL.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Fokko
>>>>>>>
>>>>>>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>>>>>>> +1 for an OpenDALFileIO
>>>>>>>>
>>>>>>>> -Jack
>>>>>>>>
>>>>>>>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>> __
>>>>>>>>> Hi, renjie
>>>>>>>>>
>>>>>>>>> Thank you for your support. I'll delve into the details and first build a PoC PR to make it clear.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>>>>>>>> Hi:
>>>>>>>>>>
>>>>>>>>>> Thanks Xuanwo for raising this.
>>>>>>>>>>
>>>>>>>>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg is a good idea.
>>>>>>>>>>
>>>>>>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>>>>>>>
>>>>>>>>>> I think this is a good first step moving forward to make pyiceberg backed iceberg-rust. In the future we can replace components gradually.
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>>>> __
>>>>>>>>>>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>>>>>
>>>>>>>>>>> Hi, Raschkowski, good question!
>>>>>>>>>>>
>>>>>>>>>>> It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
>>>>>>>>>>>
>>>>>>>>>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>>>>>>>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *From:* Joe Stein <cr...@gmail.com>
>>>>>>>>>>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>>>>>>>>>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>>>>>>>>>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>>>>>>>>>>
>>>>>>>>>>>> *CAUTION:* This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>>>>>>>>>>>>
>>>>>>>>>>>> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~ Joe Stein
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>>>>>> Hello everyone
>>>>>>>>>>>>>
>>>>>>>>>>>>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io] <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Welcome to join in the discussion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Xuanwo
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://xuanwo.io/ [xuanwo.io] <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>>>>>>>>>> Xuanwo
>>>>>>>>>>>
>>>>>>>>>>> https://xuanwo.io/
>>>>>>>>>>>
>>>>>>>>> Xuanwo
>>>>>>>>>
>>>>>>>>> https://xuanwo.io/
>>>>>>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/
>>>>
>>> Xuanwo
>>>
>>> https://xuanwo.io/
>>>
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by "Honah J." <ho...@apache.org>.
Thanks Xuanwo for driving this and everyone for discussing,
I like the idea of pushing down low-level logic to Iceberg-rust
(pyiceberg_core). It’s great to have another option besides PyArrow for
reading and writing data in PyIceberg. Thanks, Xuanwo, for moving this
forward with the initial PR to add pyiceberg_core.
For the FileIO part, just curious—since Rust's FileIO currently also uses
OpenDAL, will there be any functional differences in terms of supported
storage services or configurations (like profile_name, signer, etc.)
compared to using opendalfs directly in Python in the future? Will Rust's
FileIO introduce any customizations/optimizations/extensions beyond what
OpenDAL supports?
Best regards,
Honah
On Sat, Aug 3, 2024 at 4:12 PM timoguin@proton.me.INVALID
<ti...@proton.me.invalid> wrote:
> Fantastic work! I think this is a great direction, and this provides a
> good base to start iterating.
>
> It makes the most sense to me for the Python bindings (and others) to live
> in the same repo as iceberg-rust, especially at this early stage.
>
> - Tim O'Guin
>
>
> -------- Original Message --------
> On 8/3/24 12:33 AM, Xuanwo wrote:
>
> Let's rock! Welcome to take a review:
> https://github.com/apache/iceberg-rust/pull/518
>
> On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
>
> I also support integrating iceberg-rust with pyiceberg rather than
> building something new on OpenDAL.
>
> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the
> native fsspec support for OpenDAL, is ready. Users can use opendalfs as a
> FileIO class directly in pure python. It's not an action item for our
> community to take.
>
> The consensus we've reached is that iceberg-rust will be the core of
> PyIceberg. The main question now is "How?" How can we implement it without
> disrupting our valued users? This is my top priority.
>
> *Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg
> core as `pyiceberg-core` until we decide on a project name.*
>
> First, we need to establish a workflow that allows us to gradually
> integrate new features into pyiceberg-core. Additionally, pyiceberg should
> be able to import and optionally use classes from pyiceberg-core in an
> additive manner. While developing this workflow, our community will learn
> how to collaborate, manage releases, and more.
>
> We will then incorporate additional Rust-backed features into
> pyiceberg-core. Eventually, we may make pyiceberg-core our default
> implementation.
>
> My current plan is to implement this pyiceberg-core under iceberg-rust
> repo under `bindings/python`.
>
> - Iceberg-rust is currently under active development. I plan to release
> pyiceberg-core independently of iceberg-rust's release, as they feature
> distinct public APIs (and languages!).
> - Most of the work involves maintaining a few Python stubs and classes,
> with the majority related to Rust.
> - The python integration is just a start: we can expect `bindings/nodejs`
> to happen here too.
>
> The setup work has already been started. I will update my PR here once
> it's ready to review.
>
> [1]: https://github.com/fsspec/opendalfs
>
> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>
> Hi:
>
> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's
> FileIO, rather than directly using OpenDAL. The motivation is that we can
> use this as a starting point of providing iceberg-rust backed components
> for pyiceberg, and due to its simplicity, it's a good case. I believe there
> will be more cases, like Sung mentioned transform in another thread, and
> table scan mentioned by Fokko.
>
> If we want to use OpenDAL directly, we don't need iceberg-rust, since
> OpenDAL already has python binding:
> https://opendal.apache.org/docs/python/opendal.html
>
> Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
>
>
> Well, first I want to say providing a python binding for a library
> written in rust is a quite common practice. Just to name a few: opendal
> <https://github.com/apache/opendal>, polars
> <https://github.com/pola-rs/polars>, datafusion
> <https://github.com/apache/datafusion>, delta-rs
> <https://github.com/delta-io/delta-rs>. As far as I know, most of them
> choose to put python binding with rust in the same repo, only
> datafusion-python <https://github.com/apache/datafusion-python> lives in
> another, I'm not sure about the reason, maybe it's too large?
>
> I haven't tried to implement one before, but pyo3
> <https://github.com/PyO3> has great documentation, and there are many
> existing examples in open source we can learn with.
>
> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> One more thing,
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
>
> Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
>
> Kind regards,
> Fokko
>
>
>
> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>
> Thanks for driving this Xuanwo,
>
> I already suggested this in my talk back at the Spark Summit to see if we
> can spark some interest, and it is exciting to see this materialize.
>
> For the IO abstraction, I think the FileIO is the best option. We already
> have the interface
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
> in PyIceberg, and also a PyArrowFileIO
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
> I must admit that the abstraction is less clear in PyIceberg since we rely
> so much on Arrow for reading/writing data that it is tightly coupled. I
> would love to see if we can use OpenDAL for reading/writing data, and
> Iceberg-rust for pushing down the low-level logic. A while ago I did some
> profiling on the code, and one of the major issues is that Arrow doesn't
> support proper field-ID projection. Therefore we have to the Parquet file,
> and do the schema-evolution and type promotion afterwards in Python
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
> which causes a lot of congestion on the GIL.
>
> Kind regards,
> Fokko
>
> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>
> +1 for an OpenDALFileIO
>
> -Jack
>
> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>
>
> Hi, renjie
>
> Thank you for your support. I'll delve into the details and first build a
> PoC PR to make it clear.
>
> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>
> Hi:
>
> Thanks Xuanwo for raising this.
>
> As mentioned in another thread, I think using iceberg-rust in pyiceberg is
> a good idea.
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
> I think this is a good first step moving forward to make pyiceberg backed
> iceberg-rust. In the future we can replace components gradually.
>
> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>
>
> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
> Hi, Raschkowski, good question!
>
> It's possible. There is an ongoing project developing fsspec bindings for
> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
> directly use opendal through fsspec.
>
> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
> be able to use opendalfs as an alternative implementation of the fsspec
> AbstractFileSystem class.
>
> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
>
> ------------------------------
>
> *From:* Joe Stein <cr...@gmail.com>
> *Sent:* Thursday, August 1, 2024 3:37 AM
> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>
> *CAUTION:* This email originates from an external party (outside of
> Palantir). If you believe this message is suspicious in nature, please use
> the "Report Message" button built into Outlook.
>
> Kafka did this with librdkafka and was wildly successful. The underlying
> bindings being in rust are great with a layer for access in Python +1
>
>
> ~ Joe Stein
>
>
> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>
> Hello everyone
>
> I start this thread to discuss the idea about using iceberg-rust as
> pyiceberg file io.
>
> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
> [hackmd.io]
> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>
> In summary, we can leverage the work from iceberg-rust to help pyiceberg
> in developing a fast and compact file IO system that benefits users with
> specific constraints.
>
> Welcome to join in the discussion.
>
> Xuanwo
>
> https://xuanwo.io/ [xuanwo.io]
> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by "timoguin@proton.me.INVALID" <ti...@proton.me.INVALID>.
Fantastic work! I think this is a great direction, and this provides a good base to start iterating.
It makes the most sense to me for the Python bindings (and others) to live in the same repo as iceberg-rust, especially at this early stage.
- Tim O'Guin
-------- Original Message --------
On 8/3/24 12:33 AM, Xuanwo wrote:
> Let's rock! Welcome to take a review: https://github.com/apache/iceberg-rust/pull/518
>
> On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
>
>> I also support integrating iceberg-rust with pyiceberg rather than building something new on OpenDAL.
>>
>> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the native fsspec support for OpenDAL, is ready. Users can use opendalfs as a FileIO class directly in pure python. It's not an action item for our community to take.
>>
>> The consensus we've reached is that iceberg-rust will be the core of PyIceberg. The main question now is "How?" How can we implement it without disrupting our valued users? This is my top priority.
>>
>> Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg core as `pyiceberg-core` until we decide on a project name.
>>
>> First, we need to establish a workflow that allows us to gradually integrate new features into pyiceberg-core. Additionally, pyiceberg should be able to import and optionally use classes from pyiceberg-core in an additive manner. While developing this workflow, our community will learn how to collaborate, manage releases, and more.
>>
>> We will then incorporate additional Rust-backed features into pyiceberg-core. Eventually, we may make pyiceberg-core our default implementation.
>>
>> My current plan is to implement this pyiceberg-core under iceberg-rust repo under `bindings/python`.
>>
>> - Iceberg-rust is currently under active development. I plan to release pyiceberg-core independently of iceberg-rust's release, as they feature distinct public APIs (and languages!).
>> - Most of the work involves maintaining a few Python stubs and classes, with the majority related to Rust.
>> - The python integration is just a start: we can expect `bindings/nodejs` to happen here too.
>>
>> The setup work has already been started. I will update my PR here once it's ready to review.
>>
>> [1]: https://github.com/fsspec/opendalfs
>>
>> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>>
>>> Hi:
>>>
>>> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's FileIO, rather than directly using OpenDAL. The motivation is that we can use this as a starting point of providing iceberg-rust backed components for pyiceberg, and due to its simplicity, it's a good case. I believe there will be more cases, like Sung mentioned transform in another thread, and table scan mentioned by Fokko.
>>>
>>> If we want to use OpenDAL directly, we don't need iceberg-rust, since OpenDAL already has python binding: https://opendal.apache.org/docs/python/opendal.html
>>>
>>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic ([pydantic](https://github.com/pydantic/pydantic), [pydantic-core](https://github.com/pydantic/pydantic-core)).
>>>
>>> Well, first I want to say providing a python binding for a library written in rust is a quite common practice. Just to name a few: [opendal](https://github.com/apache/opendal), [polars](https://github.com/pola-rs/polars), [datafusion](https://github.com/apache/datafusion), [delta-rs](https://github.com/delta-io/delta-rs). As far as I know, most of them choose to put python binding with rust in the same repo, only [datafusion-python](https://github.com/apache/datafusion-python) lives in another, I'm not sure about the reason, maybe it's too large?
>>>
>>> I haven't tried to implement one before, but [pyo3](https://github.com/PyO3) has great documentation, and there are many existing examples in open source we can learn with.
>>>
>>> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>>>
>>>> One more thing,
>>>>
>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>
>>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic ([pydantic](https://github.com/pydantic/pydantic), [pydantic-core](https://github.com/pydantic/pydantic-core)).
>>>>
>>>> Kind regards,
>>>> Fokko
>>>>
>>>> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>>>>
>>>>> Thanks for driving this Xuanwo,
>>>>>
>>>>> I already suggested this in my talk back at the Spark Summit to see if we can spark some interest, and it is exciting to see this materialize.
>>>>>
>>>>> For the IO abstraction, I think the FileIO is the best option. We already have the [interface](https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239) in PyIceberg, and also a [PyArrowFileIO](https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327). I must admit that the abstraction is less clear in PyIceberg since we rely so much on Arrow for reading/writing data that it is tightly coupled. I would love to see if we can use OpenDAL for reading/writing data, and Iceberg-rust for pushing down the low-level logic. A while ago I did some profiling on the code, and one of the major issues is that Arrow doesn't support proper field-ID projection. Therefore we have to the Parquet file, and do the schema-evolution and type promotion afterwards [in Python](https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458), which causes a lot of congestion on the GIL.
>>>>>
>>>>> Kind regards,
>>>>> Fokko
>>>>>
>>>>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>>>>
>>>>>> +1 for an OpenDALFileIO
>>>>>>
>>>>>> -Jack
>>>>>>
>>>>>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>>>>>
>>>>>>> Hi, renjie
>>>>>>>
>>>>>>> Thank you for your support. I'll delve into the details and first build a PoC PR to make it clear.
>>>>>>>
>>>>>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>>>>>
>>>>>>>> Hi:
>>>>>>>>
>>>>>>>> Thanks Xuanwo for raising this.
>>>>>>>>
>>>>>>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg is a good idea.
>>>>>>>>
>>>>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>>>>>
>>>>>>>> I think this is a good first step moving forward to make pyiceberg backed iceberg-rust. In the future we can replace components gradually.
>>>>>>>>
>>>>>>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>
>>>>>>>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / [fsspec bindings](https://github.com/apache/opendal/issues/4511)?
>>>>>>>>>
>>>>>>>>> Hi, Raschkowski, good question!
>>>>>>>>>
>>>>>>>>> It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
>>>>>>>>>
>>>>>>>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>>>>>>>
>>>>>>>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / [fsspec bindings](https://github.com/apache/opendal/issues/4511)?
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> From: Joe Stein <cr...@gmail.com>
>>>>>>>>>> Sent: Thursday, August 1, 2024 3:37 AM
>>>>>>>>>> To: dev@iceberg.apache.org <de...@iceberg.apache.org>
>>>>>>>>>> Subject: Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>>>>>>>>
>>>>>>>>>> CAUTION: This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>>>>>>>>>>
>>>>>>>>>> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>>>>>>>>>>
>>>>>>>>>> ~ Joe Stein
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello everyone
>>>>>>>>>>>
>>>>>>>>>>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>>>>>>>>>>
>>>>>>>>>>> The idea is living at [https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io]](https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$)
>>>>>>>>>>>
>>>>>>>>>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>>>>>>>>>>
>>>>>>>>>>> Welcome to join in the discussion.
>>>>>>>>>>>
>>>>>>>>>>> Xuanwo
>>>>>>>>>>>
>>>>>>>>>>> [https://xuanwo.io/ [xuanwo.io]](https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$)
>>>>>>>>>
>>>>>>>>> Xuanwo
>>>>>>>>>
>>>>>>>>> https://xuanwo.io/
>>>>>>>
>>>>>>> Xuanwo
>>>>>>>
>>>>>>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Xuanwo <xu...@apache.org>.
Let's rock! Welcome to take a review: https://github.com/apache/iceberg-rust/pull/518
On Sat, Aug 3, 2024, at 12:13, Xuanwo wrote:
> I also support integrating iceberg-rust with pyiceberg rather than building something new on OpenDAL.
>
> OpenDAL backed FileIO will be usable in Python once opendalfs[1], the native fsspec support for OpenDAL, is ready. Users can use opendalfs as a FileIO class directly in pure python. It's not an action item for our community to take.
>
> The consensus we've reached is that iceberg-rust will be the core of PyIceberg. The main question now is "How?" How can we implement it without disrupting our valued users? This is my top priority.
>
> *Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg core as `*pyiceberg-core*` until we decide on a project name.*
>
> First, we need to establish a workflow that allows us to gradually integrate new features into pyiceberg-core. Additionally, pyiceberg should be able to import and optionally use classes from pyiceberg-core in an additive manner. While developing this workflow, our community will learn how to collaborate, manage releases, and more.
>
> We will then incorporate additional Rust-backed features into pyiceberg-core. Eventually, we may make pyiceberg-core our default implementation.
>
> My current plan is to implement this pyiceberg-core under iceberg-rust repo under `bindings/python`.
>
> - Iceberg-rust is currently under active development. I plan to release pyiceberg-core independently of iceberg-rust's release, as they feature distinct public APIs (and languages!).
> - Most of the work involves maintaining a few Python stubs and classes, with the majority related to Rust.
> - The python integration is just a start: we can expect `bindings/nodejs` to happen here too.
>
> The setup work has already been started. I will update my PR here once it's ready to review.
>
> [1]: https://github.com/fsspec/opendalfs
>
> On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
>> Hi:
>>
>> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's FileIO, rather than directly using OpenDAL. The motivation is that we can use this as a starting point of providing iceberg-rust backed components for pyiceberg, and due to its simplicity, it's a good case. I believe there will be more cases, like Sung mentioned transform in another thread, and table scan mentioned by Fokko.
>>
>> If we want to use OpenDAL directly, we don't need iceberg-rust, since OpenDAL already has python binding: https://opendal.apache.org/docs/python/opendal.html
>>
>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>>
>> Well, first I want to say providing a python binding for a library written in rust is a quite common practice. Just to name a few: opendal <https://github.com/apache/opendal>, polars <https://github.com/pola-rs/polars>, datafusion <https://github.com/apache/datafusion>, delta-rs <https://github.com/delta-io/delta-rs>. As far as I know, most of them choose to put python binding with rust in the same repo, only datafusion-python <https://github.com/apache/datafusion-python> lives in another, I'm not sure about the reason, maybe it's too large?
>>
>> I haven't tried to implement one before, but pyo3 <https://github.com/PyO3> has great documentation, and there are many existing examples in open source we can learn with.
>>
>> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>>> One more thing,
>>>
>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>
>>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>>>
>>> Kind regards,
>>> Fokko
>>>
>>>
>>>
>>> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>>>> Thanks for driving this Xuanwo,
>>>>
>>>> I already suggested this in my talk back at the Spark Summit to see if we can spark some interest, and it is exciting to see this materialize.
>>>>
>>>> For the IO abstraction, I think the FileIO is the best option. We already have the interface <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239> in PyIceberg, and also a PyArrowFileIO <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>. I must admit that the abstraction is less clear in PyIceberg since we rely so much on Arrow for reading/writing data that it is tightly coupled. I would love to see if we can use OpenDAL for reading/writing data, and Iceberg-rust for pushing down the low-level logic. A while ago I did some profiling on the code, and one of the major issues is that Arrow doesn't support proper field-ID projection. Therefore we have to the Parquet file, and do the schema-evolution and type promotion afterwards in Python <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>, which causes a lot of congestion on the GIL.
>>>>
>>>> Kind regards,
>>>> Fokko
>>>>
>>>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>>>> +1 for an OpenDALFileIO
>>>>>
>>>>> -Jack
>>>>>
>>>>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>>>>> __
>>>>>> Hi, renjie
>>>>>>
>>>>>> Thank you for your support. I'll delve into the details and first build a PoC PR to make it clear.
>>>>>>
>>>>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>>>>> Hi:
>>>>>>>
>>>>>>> Thanks Xuanwo for raising this.
>>>>>>>
>>>>>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg is a good idea.
>>>>>>>
>>>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>>>>
>>>>>>> I think this is a good first step moving forward to make pyiceberg backed iceberg-rust. In the future we can replace components gradually.
>>>>>>>
>>>>>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>> __
>>>>>>>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>>
>>>>>>>> Hi, Raschkowski, good question!
>>>>>>>>
>>>>>>>> It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
>>>>>>>>
>>>>>>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
>>>>>>>>
>>>>>>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>>>>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* Joe Stein <cr...@gmail.com>
>>>>>>>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>>>>>>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>>>>>>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>>>>>>>
>>>>>>>>> *CAUTION:* This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>>>>>>>>>
>>>>>>>>> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ~ Joe Stein
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>>> Hello everyone
>>>>>>>>>>
>>>>>>>>>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>>>>>>>>>
>>>>>>>>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io] <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>>>>>>>>
>>>>>>>>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>>>>>>>>>
>>>>>>>>>> Welcome to join in the discussion.
>>>>>>>>>>
>>>>>>>>>> Xuanwo
>>>>>>>>>>
>>>>>>>>>> https://xuanwo.io/ [xuanwo.io] <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>>>>>>> Xuanwo
>>>>>>>>
>>>>>>>> https://xuanwo.io/
>>>>>>>>
>>>>>> Xuanwo
>>>>>>
>>>>>> https://xuanwo.io/
>>>>>>
> Xuanwo
>
> https://xuanwo.io/
>
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Xuanwo <xu...@apache.org>.
I also support integrating iceberg-rust with pyiceberg rather than building something new on OpenDAL.
OpenDAL backed FileIO will be usable in Python once opendalfs[1], the native fsspec support for OpenDAL, is ready. Users can use opendalfs as a FileIO class directly in pure python. It's not an action item for our community to take.
The consensus we've reached is that iceberg-rust will be the core of PyIceberg. The main question now is "How?" How can we implement it without disrupting our valued users? This is my top priority.
*Naming is so hard! Let's refer to the new iceberg-rust based pyiceberg core as `*pyiceberg-core*` until we decide on a project name.*
First, we need to establish a workflow that allows us to gradually integrate new features into pyiceberg-core. Additionally, pyiceberg should be able to import and optionally use classes from pyiceberg-core in an additive manner. While developing this workflow, our community will learn how to collaborate, manage releases, and more.
We will then incorporate additional Rust-backed features into pyiceberg-core. Eventually, we may make pyiceberg-core our default implementation.
My current plan is to implement this pyiceberg-core under iceberg-rust repo under `bindings/python`.
- Iceberg-rust is currently under active development. I plan to release pyiceberg-core independently of iceberg-rust's release, as they feature distinct public APIs (and languages!).
- Most of the work involves maintaining a few Python stubs and classes, with the majority related to Rust.
- The python integration is just a start: we can expect `bindings/nodejs` to happen here too.
The setup work has already been started. I will update my PR here once it's ready to review.
[1]: https://github.com/fsspec/opendalfs
On Sat, Aug 3, 2024, at 09:57, Renjie Liu wrote:
> Hi:
>
> I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's FileIO, rather than directly using OpenDAL. The motivation is that we can use this as a starting point of providing iceberg-rust backed components for pyiceberg, and due to its simplicity, it's a good case. I believe there will be more cases, like Sung mentioned transform in another thread, and table scan mentioned by Fokko.
>
> If we want to use OpenDAL directly, we don't need iceberg-rust, since OpenDAL already has python binding: https://opendal.apache.org/docs/python/opendal.html
>
>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>
> Well, first I want to say providing a python binding for a library written in rust is a quite common practice. Just to name a few: opendal <https://github.com/apache/opendal>, polars <https://github.com/pola-rs/polars>, datafusion <https://github.com/apache/datafusion>, delta-rs <https://github.com/delta-io/delta-rs>. As far as I know, most of them choose to put python binding with rust in the same repo, only datafusion-python <https://github.com/apache/datafusion-python> lives in another, I'm not sure about the reason, maybe it's too large?
>
> I haven't tried to implement one before, but pyo3 <https://github.com/PyO3> has great documentation, and there are many existing examples in open source we can learn with.
>
> On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
>> One more thing,
>>
>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>
>> Do you have any experience with this? I see many projects having Rust and Python code in a single repository. There are some exceptions like Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core <https://github.com/pydantic/pydantic-core>).
>>
>> Kind regards,
>> Fokko
>>
>>
>>
>> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>>> Thanks for driving this Xuanwo,
>>>
>>> I already suggested this in my talk back at the Spark Summit to see if we can spark some interest, and it is exciting to see this materialize.
>>>
>>> For the IO abstraction, I think the FileIO is the best option. We already have the interface <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239> in PyIceberg, and also a PyArrowFileIO <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>. I must admit that the abstraction is less clear in PyIceberg since we rely so much on Arrow for reading/writing data that it is tightly coupled. I would love to see if we can use OpenDAL for reading/writing data, and Iceberg-rust for pushing down the low-level logic. A while ago I did some profiling on the code, and one of the major issues is that Arrow doesn't support proper field-ID projection. Therefore we have to the Parquet file, and do the schema-evolution and type promotion afterwards in Python <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>, which causes a lot of congestion on the GIL.
>>>
>>> Kind regards,
>>> Fokko
>>>
>>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>>> +1 for an OpenDALFileIO
>>>>
>>>> -Jack
>>>>
>>>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>>>> __
>>>>> Hi, renjie
>>>>>
>>>>> Thank you for your support. I'll delve into the details and first build a PoC PR to make it clear.
>>>>>
>>>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>>>> Hi:
>>>>>>
>>>>>> Thanks Xuanwo for raising this.
>>>>>>
>>>>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg is a good idea.
>>>>>>
>>>>>> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>>>
>>>>>> I think this is a good first step moving forward to make pyiceberg backed iceberg-rust. In the future we can replace components gradually.
>>>>>>
>>>>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>> __
>>>>>>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>
>>>>>>> Hi, Raschkowski, good question!
>>>>>>>
>>>>>>> It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
>>>>>>>
>>>>>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
>>>>>>>
>>>>>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>>>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From:* Joe Stein <cr...@gmail.com>
>>>>>>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>>>>>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>>>>>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>>>>>>
>>>>>>>> *CAUTION:* This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>>>>>>>>
>>>>>>>> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>>>>>>>>
>>>>>>>>
>>>>>>>> ~ Joe Stein
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>>>>>>> Hello everyone
>>>>>>>>>
>>>>>>>>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>>>>>>>>
>>>>>>>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io] <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>>>>>>>
>>>>>>>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>>>>>>>>
>>>>>>>>> Welcome to join in the discussion.
>>>>>>>>>
>>>>>>>>> Xuanwo
>>>>>>>>>
>>>>>>>>> https://xuanwo.io/ [xuanwo.io] <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>>>>>> Xuanwo
>>>>>>>
>>>>>>> https://xuanwo.io/
>>>>>>>
>>>>> Xuanwo
>>>>>
>>>>> https://xuanwo.io/
>>>>>
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Renjie Liu <li...@gmail.com>.
Hi:
I lean towards implementing pyiceberg's FileIO backed by iceberg-rust's
FileIO, rather than directly using OpenDAL. The motivation is that we can
use this as a starting point of providing iceberg-rust backed components
for pyiceberg, and due to its simplicity, it's a good case. I believe there
will be more cases, like Sung mentioned transform in another thread, and
table scan mentioned by Fokko.
If we want to use OpenDAL directly, we don't need iceberg-rust, since
OpenDAL already has python binding:
https://opendal.apache.org/docs/python/opendal.html
Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
Well, first I want to say providing a python binding for a library
written in rust is a quite common practice. Just to name a few: opendal
<https://github.com/apache/opendal>, polars
<https://github.com/pola-rs/polars>, datafusion
<https://github.com/apache/datafusion>, delta-rs
<https://github.com/delta-io/delta-rs>. As far as I know, most of them
choose to put python binding with rust in the same repo, only
datafusion-python <https://github.com/apache/datafusion-python> lives in
another, I'm not sure about the reason, maybe it's too large?
I haven't tried to implement one before, but pyo3 <https://github.com/PyO3> has
great documentation, and there are many existing examples in open source we
can learn with.
On Sat, Aug 3, 2024 at 2:23 AM Fokko Driesprong <fo...@apache.org> wrote:
> One more thing,
>
> About this idea, would you have a more detailed design? For example,
>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
>
> Do you have any experience with this? I see many projects having Rust and
> Python code in a single repository. There are some exceptions like
> Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
> <https://github.com/pydantic/pydantic-core>).
>
> Kind regards,
> Fokko
>
>
>
> Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
>
>> Thanks for driving this Xuanwo,
>>
>> I already suggested this in my talk back at the Spark Summit to see if we
>> can spark some interest, and it is exciting to see this materialize.
>>
>> For the IO abstraction, I think the FileIO is the best option. We already
>> have the interface
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
>> in PyIceberg, and also a PyArrowFileIO
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
>> I must admit that the abstraction is less clear in PyIceberg since we rely
>> so much on Arrow for reading/writing data that it is tightly coupled. I
>> would love to see if we can use OpenDAL for reading/writing data, and
>> Iceberg-rust for pushing down the low-level logic. A while ago I did some
>> profiling on the code, and one of the major issues is that Arrow doesn't
>> support proper field-ID projection. Therefore we have to the Parquet file,
>> and do the schema-evolution and type promotion afterwards in Python
>> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
>> which causes a lot of congestion on the GIL.
>>
>> Kind regards,
>> Fokko
>>
>> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>>
>>> +1 for an OpenDALFileIO
>>>
>>> -Jack
>>>
>>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>>
>>>> Hi, renjie
>>>>
>>>> Thank you for your support. I'll delve into the details and first build
>>>> a PoC PR to make it clear.
>>>>
>>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>>
>>>> Hi:
>>>>
>>>> Thanks Xuanwo for raising this.
>>>>
>>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg
>>>> is a good idea.
>>>>
>>>> About this idea, would you have a more detailed design? For example,
>>>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>>>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>>
>>>> I think this is a good first step moving forward to make pyiceberg
>>>> backed iceberg-rust. In the future we can replace components gradually.
>>>>
>>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>>
>>>>
>>>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>>>> implementations via pyo3 / fsspec bindings
>>>> <https://github.com/apache/opendal/issues/4511>?
>>>>
>>>> Hi, Raschkowski, good question!
>>>>
>>>> It's possible. There is an ongoing project developing fsspec bindings
>>>> for opendal at https://github.com/fsspec/opendalfs. Once complete, we
>>>> can directly use opendal through fsspec.
>>>>
>>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users
>>>> should be able to use opendalfs as an alternative implementation of the
>>>> fsspec AbstractFileSystem class.
>>>>
>>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>>
>>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>>>> implementations via pyo3 / fsspec bindings
>>>> <https://github.com/apache/opendal/issues/4511>?
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> *From:* Joe Stein <cr...@gmail.com>
>>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>>
>>>> *CAUTION:* This email originates from an external party (outside of
>>>> Palantir). If you believe this message is suspicious in nature, please use
>>>> the "Report Message" button built into Outlook.
>>>>
>>>> Kafka did this with librdkafka and was wildly successful. The
>>>> underlying bindings being in rust are great with a layer for access in
>>>> Python +1
>>>>
>>>>
>>>> ~ Joe Stein
>>>>
>>>>
>>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>>
>>>> Hello everyone
>>>>
>>>> I start this thread to discuss the idea about using iceberg-rust as
>>>> pyiceberg file io.
>>>>
>>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>>>> [hackmd.io]
>>>> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>>
>>>> In summary, we can leverage the work from iceberg-rust to help
>>>> pyiceberg in developing a fast and compact file IO system that benefits
>>>> users with specific constraints.
>>>>
>>>> Welcome to join in the discussion.
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/ [xuanwo.io]
>>>> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/
>>>>
>>>>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Fokko Driesprong <fo...@apache.org>.
One more thing,
About this idea, would you have a more detailed design? For example, where
> should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of
> interface should we provide to pyiceberg, FileIO or OpenDAL?
Do you have any experience with this? I see many projects having Rust and
Python code in a single repository. There are some exceptions like
Pydantic (pydantic <https://github.com/pydantic/pydantic>, pydantic-core
<https://github.com/pydantic/pydantic-core>).
Kind regards,
Fokko
Op vr 2 aug 2024 om 20:11 schreef Fokko Driesprong <fo...@apache.org>:
> Thanks for driving this Xuanwo,
>
> I already suggested this in my talk back at the Spark Summit to see if we
> can spark some interest, and it is exciting to see this materialize.
>
> For the IO abstraction, I think the FileIO is the best option. We already
> have the interface
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
> in PyIceberg, and also a PyArrowFileIO
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
> I must admit that the abstraction is less clear in PyIceberg since we rely
> so much on Arrow for reading/writing data that it is tightly coupled. I
> would love to see if we can use OpenDAL for reading/writing data, and
> Iceberg-rust for pushing down the low-level logic. A while ago I did some
> profiling on the code, and one of the major issues is that Arrow doesn't
> support proper field-ID projection. Therefore we have to the Parquet file,
> and do the schema-evolution and type promotion afterwards in Python
> <https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
> which causes a lot of congestion on the GIL.
>
> Kind regards,
> Fokko
>
> Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
>
>> +1 for an OpenDALFileIO
>>
>> -Jack
>>
>> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>>
>>> Hi, renjie
>>>
>>> Thank you for your support. I'll delve into the details and first build
>>> a PoC PR to make it clear.
>>>
>>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>>
>>> Hi:
>>>
>>> Thanks Xuanwo for raising this.
>>>
>>> As mentioned in another thread, I think using iceberg-rust in pyiceberg
>>> is a good idea.
>>>
>>> About this idea, would you have a more detailed design? For example,
>>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>>
>>> I think this is a good first step moving forward to make pyiceberg
>>> backed iceberg-rust. In the future we can replace components gradually.
>>>
>>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>>
>>>
>>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>>> implementations via pyo3 / fsspec bindings
>>> <https://github.com/apache/opendal/issues/4511>?
>>>
>>> Hi, Raschkowski, good question!
>>>
>>> It's possible. There is an ongoing project developing fsspec bindings
>>> for opendal at https://github.com/fsspec/opendalfs. Once complete, we
>>> can directly use opendal through fsspec.
>>>
>>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users
>>> should be able to use opendalfs as an alternative implementation of the
>>> fsspec AbstractFileSystem class.
>>>
>>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>>
>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>>> implementations via pyo3 / fsspec bindings
>>> <https://github.com/apache/opendal/issues/4511>?
>>>
>>>
>>> ------------------------------
>>>
>>> *From:* Joe Stein <cr...@gmail.com>
>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>
>>> *CAUTION:* This email originates from an external party (outside of
>>> Palantir). If you believe this message is suspicious in nature, please use
>>> the "Report Message" button built into Outlook.
>>>
>>> Kafka did this with librdkafka and was wildly successful. The underlying
>>> bindings being in rust are great with a layer for access in Python +1
>>>
>>>
>>> ~ Joe Stein
>>>
>>>
>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>
>>> Hello everyone
>>>
>>> I start this thread to discuss the idea about using iceberg-rust as
>>> pyiceberg file io.
>>>
>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>>> [hackmd.io]
>>> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>
>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg
>>> in developing a fast and compact file IO system that benefits users with
>>> specific constraints.
>>>
>>> Welcome to join in the discussion.
>>>
>>> Xuanwo
>>>
>>> https://xuanwo.io/ [xuanwo.io]
>>> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>>
>>> Xuanwo
>>>
>>> https://xuanwo.io/
>>>
>>> Xuanwo
>>>
>>> https://xuanwo.io/
>>>
>>>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Fokko Driesprong <fo...@apache.org>.
Thanks for driving this Xuanwo,
I already suggested this in my talk back at the Spark Summit to see if we
can spark some interest, and it is exciting to see this materialize.
For the IO abstraction, I think the FileIO is the best option. We already
have the interface
<https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/__init__.py#L239>
in PyIceberg, and also a PyArrowFileIO
<https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L327>.
I must admit that the abstraction is less clear in PyIceberg since we rely
so much on Arrow for reading/writing data that it is tightly coupled. I
would love to see if we can use OpenDAL for reading/writing data, and
Iceberg-rust for pushing down the low-level logic. A while ago I did some
profiling on the code, and one of the major issues is that Arrow doesn't
support proper field-ID projection. Therefore we have to the Parquet file,
and do the schema-evolution and type promotion afterwards in Python
<https://github.com/apache/iceberg-python/blob/6c0d307032608967ccd00cfe72d8815e6e7e01cc/pyiceberg/io/pyarrow.py#L1444-L1458>,
which causes a lot of congestion on the GIL.
Kind regards,
Fokko
Op vr 2 aug 2024 om 17:46 schreef Jack Ye <ye...@gmail.com>:
> +1 for an OpenDALFileIO
>
> -Jack
>
> On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
>
>> Hi, renjie
>>
>> Thank you for your support. I'll delve into the details and first build a
>> PoC PR to make it clear.
>>
>> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>>
>> Hi:
>>
>> Thanks Xuanwo for raising this.
>>
>> As mentioned in another thread, I think using iceberg-rust in pyiceberg
>> is a good idea.
>>
>> About this idea, would you have a more detailed design? For example,
>> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
>> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>>
>> I think this is a good first step moving forward to make pyiceberg backed
>> iceberg-rust. In the future we can replace components gradually.
>>
>> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>>
>>
>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>> Hi, Raschkowski, good question!
>>
>> It's possible. There is an ongoing project developing fsspec bindings for
>> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
>> directly use opendal through fsspec.
>>
>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
>> be able to use opendalfs as an alternative implementation of the fsspec
>> AbstractFileSystem class.
>>
>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>
>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
>> implementations via pyo3 / fsspec bindings
>> <https://github.com/apache/opendal/issues/4511>?
>>
>>
>> ------------------------------
>>
>> *From:* Joe Stein <cr...@gmail.com>
>> *Sent:* Thursday, August 1, 2024 3:37 AM
>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>
>> *CAUTION:* This email originates from an external party (outside of
>> Palantir). If you believe this message is suspicious in nature, please use
>> the "Report Message" button built into Outlook.
>>
>> Kafka did this with librdkafka and was wildly successful. The underlying
>> bindings being in rust are great with a layer for access in Python +1
>>
>>
>> ~ Joe Stein
>>
>>
>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>
>> Hello everyone
>>
>> I start this thread to discuss the idea about using iceberg-rust as
>> pyiceberg file io.
>>
>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>> [hackmd.io]
>> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>
>> In summary, we can leverage the work from iceberg-rust to help pyiceberg
>> in developing a fast and compact file IO system that benefits users with
>> specific constraints.
>>
>> Welcome to join in the discussion.
>>
>> Xuanwo
>>
>> https://xuanwo.io/ [xuanwo.io]
>> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Jack Ye <ye...@gmail.com>.
+1 for an OpenDALFileIO
-Jack
On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xu...@apache.org> wrote:
> Hi, renjie
>
> Thank you for your support. I'll delve into the details and first build a
> PoC PR to make it clear.
>
> On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
>
> Hi:
>
> Thanks Xuanwo for raising this.
>
> As mentioned in another thread, I think using iceberg-rust in pyiceberg is
> a good idea.
>
> About this idea, would you have a more detailed design? For example,
> where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What
> kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
> I think this is a good first step moving forward to make pyiceberg backed
> iceberg-rust. In the future we can replace components gradually.
>
> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>
>
> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
> Hi, Raschkowski, good question!
>
> It's possible. There is an ongoing project developing fsspec bindings for
> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
> directly use opendal through fsspec.
>
> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
> be able to use opendalfs as an alternative implementation of the fsspec
> AbstractFileSystem class.
>
> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
>
> ------------------------------
>
> *From:* Joe Stein <cr...@gmail.com>
> *Sent:* Thursday, August 1, 2024 3:37 AM
> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>
> *CAUTION:* This email originates from an external party (outside of
> Palantir). If you believe this message is suspicious in nature, please use
> the "Report Message" button built into Outlook.
>
> Kafka did this with librdkafka and was wildly successful. The underlying
> bindings being in rust are great with a layer for access in Python +1
>
>
> ~ Joe Stein
>
>
> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>
> Hello everyone
>
> I start this thread to discuss the idea about using iceberg-rust as
> pyiceberg file io.
>
> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
> [hackmd.io]
> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>
> In summary, we can leverage the work from iceberg-rust to help pyiceberg
> in developing a fast and compact file IO system that benefits users with
> specific constraints.
>
> Welcome to join in the discussion.
>
> Xuanwo
>
> https://xuanwo.io/ [xuanwo.io]
> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Xuanwo <xu...@apache.org>.
Hi, renjie
Thank you for your support. I'll delve into the details and first build a PoC PR to make it clear.
On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote:
> Hi:
>
> Thanks Xuanwo for raising this.
>
> As mentioned in another thread, I think using iceberg-rust in pyiceberg is a good idea.
>
> About this idea, would you have a more detailed design? For example, where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of interface should we provide to pyiceberg, FileIO or OpenDAL?
>
> I think this is a good first step moving forward to make pyiceberg backed iceberg-rust. In the future we can replace components gradually.
>
> On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
>> __
>> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>
>> Hi, Raschkowski, good question!
>>
>> It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
>>
>> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
>>
>> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>>> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>>>
>>>
>>>
>>> *From:* Joe Stein <cr...@gmail.com>
>>> *Sent:* Thursday, August 1, 2024 3:37 AM
>>> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
>>> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>>>
>>> *CAUTION:* This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>>>
>>> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>>>
>>>
>>> ~ Joe Stein
>>>
>>>
>>> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>>>> Hello everyone
>>>>
>>>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>>>
>>>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io] <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>>>
>>>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>>>
>>>> Welcome to join in the discussion.
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/ [xuanwo.io] <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Renjie Liu <li...@gmail.com>.
Hi:
Thanks Xuanwo for raising this.
As mentioned in another thread, I think using iceberg-rust in pyiceberg is
a good idea.
About this idea, would you have a more detailed design? For example, where
should the pyo3 codes live, in iceberg-rust or in pyiceberg? What kind of
interface should we provide to pyiceberg, FileIO or OpenDAL?
I think this is a good first step moving forward to make pyiceberg backed
iceberg-rust. In the future we can replace components gradually.
On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xu...@apache.org> wrote:
> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
> Hi, Raschkowski, good question!
>
> It's possible. There is an ongoing project developing fsspec bindings for
> opendal at https://github.com/fsspec/opendalfs. Once complete, we can
> directly use opendal through fsspec.
>
> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should
> be able to use opendalfs as an alternative implementation of the fsspec
> AbstractFileSystem class.
>
> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
>
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL
> implementations via pyo3 / fsspec bindings
> <https://github.com/apache/opendal/issues/4511>?
>
>
> ------------------------------
>
> *From:* Joe Stein <cr...@gmail.com>
> *Sent:* Thursday, August 1, 2024 3:37 AM
> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>
> *CAUTION:* This email originates from an external party (outside of
> Palantir). If you believe this message is suspicious in nature, please use
> the "Report Message" button built into Outlook.
>
> Kafka did this with librdkafka and was wildly successful. The underlying
> bindings being in rust are great with a layer for access in Python +1
>
>
> ~ Joe Stein
>
>
> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>
> Hello everyone
>
> I start this thread to discuss the idea about using iceberg-rust as
> pyiceberg file io.
>
> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
> [hackmd.io]
> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>
> In summary, we can leverage the work from iceberg-rust to help pyiceberg
> in developing a fast and compact file IO system that benefits users with
> specific constraints.
>
> Welcome to join in the discussion.
>
> Xuanwo
>
> https://xuanwo.io/ [xuanwo.io]
> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
>
> Xuanwo
>
> https://xuanwo.io/
>
>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Xuanwo <xu...@apache.org>.
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
Hi, Raschkowski, good question!
It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec.
This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class.
On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote:
> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings <https://github.com/apache/opendal/issues/4511>?
>
>
>
> *From:* Joe Stein <cr...@gmail.com>
> *Sent:* Thursday, August 1, 2024 3:37 AM
> *To:* dev@iceberg.apache.org <de...@iceberg.apache.org>
> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
>
> *CAUTION:* This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
>
> Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
>
>
> ~ Joe Stein
>
>
> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
>> Hello everyone
>>
>> I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
>>
>> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io] <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
>>
>> In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
>>
>> Welcome to join in the discussion.
>>
>> Xuanwo
>>
>> https://xuanwo.io/ [xuanwo.io] <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Will Raschkowski <wr...@palantir.com.INVALID>.
Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL implementations via pyo3 / fsspec bindings<https://github.com/apache/opendal/issues/4511>?
________________________________
From: Joe Stein <cr...@gmail.com>
Sent: Thursday, August 1, 2024 3:37 AM
To: dev@iceberg.apache.org <de...@iceberg.apache.org>
Subject: Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
CAUTION: This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Message" button built into Outlook.
Kafka did this with librdkafka and was wildly successful. The underlying bindings being in rust are great with a layer for access in Python +1
~ Joe Stein
On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org>> wrote:
Hello everyone
I start this thread to discuss the idea about using iceberg-rust as pyiceberg file io.
The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io [hackmd.io]<https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$>
In summary, we can leverage the work from iceberg-rust to help pyiceberg in developing a fast and compact file IO system that benefits users with specific constraints.
Welcome to join in the discussion.
Xuanwo
https://xuanwo.io/ [xuanwo.io]<https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$>
Re: [DISCUSS] Use iceberg-rust as pyiceberg file io
Posted by Joe Stein <cr...@gmail.com>.
Kafka did this with librdkafka and was wildly successful. The underlying
bindings being in rust are great with a layer for access in Python +1
~ Joe Stein
On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xu...@apache.org> wrote:
> Hello everyone
>
> I start this thread to discuss the idea about using iceberg-rust as
> pyiceberg file io.
>
> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io
>
> In summary, we can leverage the work from iceberg-rust to help pyiceberg
> in developing a fast and compact file IO system that benefits users with
> specific constraints.
>
> Welcome to join in the discussion.
>
> Xuanwo
>
> https://xuanwo.io/
>