You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Sung Yun <su...@gmail.com> on 2024/08/01 13:37:42 UTC
[DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Hi everyone,
This is something I've been mulling about for a while and I thought this
would be the right forum to discuss this topic as a follow up to a similar
topic discussion thread on using python bindings from iceberg-rust to
support pyiceberg.
As soon as we released 0.7.0 which supports writes into tables with
TimeTransform partitions
<https://github.com/apache/iceberg-python/pull/784/files>, our prospective
users started asking about the support for Bucket Transform partitions.
Iceberg has a custom logic for Bucket partitions (Thanks for the link
<https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I took
a look into the Java code
<https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
and I think it looks somewhat like:
* mmh3_hash(val) mod (num_buckets)
And has field type specific logic so that each type is hashed appropriately.
Unfortunately there is no existing pyarrow compute function that does this,
so I'd like to propose that we write the function in iceberg-rust that
takes an Arrow Array reference and the bucket number as the input, that
returns a new Arrow Array reference with the bucket values evaluated that
corresponds to the input Arrow Array in the same order.
When iceberg-rust becomes more mature, I believe that the same underlying
transform function can be reused for bucket partitions within this
repository, and in the interim we could support writes into Bucket
partitioned tables on PyIceberg by exposing this function as a Python
binding that we import into PyIceberg.
I'd love to hear how folks feel about this idea!
Cross posted Discussion on iceberg-rust: #514
<https://github.com/apache/iceberg-rust/discussions/514>
Sung
Re: [DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Posted by Sung Yun <su...@gmail.com>.
Hi folks, thank you all for your input. It looks like there's a general
agreement that this is a good idea. Given that Xuanwo has already put the
scaffolding in place for pyiceberg_core
<https://github.com/apache/iceberg-rust/pull/518>, I've put up this PR to
implement bucket_transform <https://github.com/apache/iceberg-rust/pull/556>
in hopes of exposing it to PyIceberg as soon as we can.
> To be honest, I don't think bucket transforms alone is not a good
starting point, maybe a more general approach would be to provide another
set of transform implementations backed by iceberg-rust?
That's a valid point. I know we are in the process of defining the
integration points between PyIceberg and pyiceberg_core (rust python
binding), so I think we have a good opportunity to replace all the
transforms.
At the same time, given that pyiceberg_core is meant to be used as a
private package of PyIceberg, I think it there's also an argument for
exposing the functionality as soon as we can, and then taking our time to
properly define the integration points to replace existing, already
supported features within PyIceberg.
On Fri, Aug 2, 2024 at 10:07 PM Renjie Liu <li...@gmail.com> wrote:
> Hi:
>
> Would pushing the bucket transforms into Rust a good exercise to get the
>> scaffolding in place?
>
>
> To be honest, I don't think bucket transforms alone is not a good starting
> point, maybe a more general approach would be to provide another set of
> transform implementations backed by iceberg-rust? The class hierarchy may
> look this following:
> ```
> Transform
> RustTransform
> BucketRustTransform
> YeanRustTransform
> ....
> ```
>
> I think it would be a win-win situation where we can combine the momentum
>> of PyIceberg and push the things that Python doesn't do well (mostly
>> multithreading and heavy lifting) to Iceberg-Rust, and also have
>> Iceberg-Rust as a library for Rust query engines to interact with Iceberg
>> tables directly.
>
>
> +1.
>
>
>
>
> On Sat, Aug 3, 2024 at 2:20 AM Fokko Driesprong <fo...@apache.org> wrote:
>
>> Hey everyone,
>>
>> In the beginning of PyIceberg, one of the goals was to keep PyIceberg
>> pure Python. At some point, we've added a Cython Avro decoder because of
>> performance reasons, but we still have a pure Python fallback. Today you
>> can still do metadata operating using s3fs without any native code. Once
>> you start reading/writing data, you'll need to pull in PyArrow.
>>
>> I think it would be a win-win situation where we can combine the momentum
>> of PyIceberg and push the things that Python doesn't do well (mostly
>> multithreading and heavy lifting) to Iceberg-Rust, and also have
>> Iceberg-Rust as a library for Rust query engines to interact with Iceberg
>> tables directly.
>>
>> My main question here is about sequencing. Would pushing the bucket
>> transforms into Rust a good exercise to get the scaffolding in place?
>>
>> Kind regards,
>> Fokko
>>
>>
>> Op vr 2 aug 2024 om 16:38 schreef Renjie Liu <li...@gmail.com>:
>>
>>> Hi:
>>>
>>> Thanks Sung for raising this. Just as Ryan said, I'm also +1 for using
>>> more iceberg-rust in pyiceberg.
>>>
>>> Is it that we would introduce a hard dependency on iceberg-rust?
>>>
>>>
>>> I think so.
>>>
>>> Is that a risk that could make PyIceberg unusable for some people? I
>>>> don't think that would be a problem since we already have a requirement for
>>>> pyarrow for these cases.
>>>
>>>
>>> +1.
>>>
>>> In fact, what I think more about is making iceberg-rust the backend of
>>> pyiceberg, and this is what me, xuanwo and Fokko had talked about in
>>> brainstorming several times. Not only bucket transform, but also not only
>>> FileIO in another thread initiated by xuanwo. I think these are all
>>> intermediate steps to our final goal.
>>>
>>>
>>> On Fri, Aug 2, 2024 at 3:21 AM Ryan Blue <bl...@databricks.com.invalid>
>>> wrote:
>>>
>>>> In general, I think the idea of using iceberg-rust more from PyIceberg
>>>> is great. I think it will be a good path to pushing more things down to
>>>> native code.
>>>>
>>>> What are the trade-offs of doing it this way? Is it that we would
>>>> introduce a hard dependency on iceberg-rust? Is that a risk that could make
>>>> PyIceberg unusable for some people? I don't think that would be a problem
>>>> since we already have a requirement for pyarrow for these cases.
>>>>
>>>> On Thu, Aug 1, 2024 at 6:39 AM Sung Yun <su...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> This is something I've been mulling about for a while and I thought
>>>>> this would be the right forum to discuss this topic as a follow up to a
>>>>> similar topic discussion thread on using python bindings from iceberg-rust
>>>>> to support pyiceberg.
>>>>>
>>>>> As soon as we released 0.7.0 which supports writes into tables with
>>>>> TimeTransform partitions
>>>>> <https://github.com/apache/iceberg-python/pull/784/files>, our
>>>>> prospective users started asking about the support for Bucket Transform
>>>>> partitions.
>>>>>
>>>>> Iceberg has a custom logic for Bucket partitions (Thanks for the link
>>>>> <https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I
>>>>> took a look into the Java code
>>>>> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
>>>>> and I think it looks somewhat like:
>>>>>
>>>>> * mmh3_hash(val) mod (num_buckets)
>>>>>
>>>>> And has field type specific logic so that each type is hashed
>>>>> appropriately.
>>>>>
>>>>> Unfortunately there is no existing pyarrow compute function that does
>>>>> this, so I'd like to propose that we write the function in iceberg-rust
>>>>> that takes an Arrow Array reference and the bucket number as the input,
>>>>> that returns a new Arrow Array reference with the bucket values evaluated
>>>>> that corresponds to the input Arrow Array in the same order.
>>>>>
>>>>> When iceberg-rust becomes more mature, I believe that the same
>>>>> underlying transform function can be reused for bucket partitions within
>>>>> this repository, and in the interim we could support writes into Bucket
>>>>> partitioned tables on PyIceberg by exposing this function as a Python
>>>>> binding that we import into PyIceberg.
>>>>>
>>>>> I'd love to hear how folks feel about this idea!
>>>>>
>>>>>
>>>>> Cross posted Discussion on iceberg-rust: #514
>>>>> <https://github.com/apache/iceberg-rust/discussions/514>
>>>>>
>>>>>
>>>>> Sung
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Databricks
>>>>
>>>
Re: [DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Posted by Renjie Liu <li...@gmail.com>.
Hi:
Would pushing the bucket transforms into Rust a good exercise to get the
> scaffolding in place?
To be honest, I don't think bucket transforms alone is not a good starting
point, maybe a more general approach would be to provide another set of
transform implementations backed by iceberg-rust? The class hierarchy may
look this following:
```
Transform
RustTransform
BucketRustTransform
YeanRustTransform
....
```
I think it would be a win-win situation where we can combine the momentum
> of PyIceberg and push the things that Python doesn't do well (mostly
> multithreading and heavy lifting) to Iceberg-Rust, and also have
> Iceberg-Rust as a library for Rust query engines to interact with Iceberg
> tables directly.
+1.
On Sat, Aug 3, 2024 at 2:20 AM Fokko Driesprong <fo...@apache.org> wrote:
> Hey everyone,
>
> In the beginning of PyIceberg, one of the goals was to keep PyIceberg pure
> Python. At some point, we've added a Cython Avro decoder because of
> performance reasons, but we still have a pure Python fallback. Today you
> can still do metadata operating using s3fs without any native code. Once
> you start reading/writing data, you'll need to pull in PyArrow.
>
> I think it would be a win-win situation where we can combine the momentum
> of PyIceberg and push the things that Python doesn't do well (mostly
> multithreading and heavy lifting) to Iceberg-Rust, and also have
> Iceberg-Rust as a library for Rust query engines to interact with Iceberg
> tables directly.
>
> My main question here is about sequencing. Would pushing the bucket
> transforms into Rust a good exercise to get the scaffolding in place?
>
> Kind regards,
> Fokko
>
>
> Op vr 2 aug 2024 om 16:38 schreef Renjie Liu <li...@gmail.com>:
>
>> Hi:
>>
>> Thanks Sung for raising this. Just as Ryan said, I'm also +1 for using
>> more iceberg-rust in pyiceberg.
>>
>> Is it that we would introduce a hard dependency on iceberg-rust?
>>
>>
>> I think so.
>>
>> Is that a risk that could make PyIceberg unusable for some people? I
>>> don't think that would be a problem since we already have a requirement for
>>> pyarrow for these cases.
>>
>>
>> +1.
>>
>> In fact, what I think more about is making iceberg-rust the backend of
>> pyiceberg, and this is what me, xuanwo and Fokko had talked about in
>> brainstorming several times. Not only bucket transform, but also not only
>> FileIO in another thread initiated by xuanwo. I think these are all
>> intermediate steps to our final goal.
>>
>>
>> On Fri, Aug 2, 2024 at 3:21 AM Ryan Blue <bl...@databricks.com.invalid>
>> wrote:
>>
>>> In general, I think the idea of using iceberg-rust more from PyIceberg
>>> is great. I think it will be a good path to pushing more things down to
>>> native code.
>>>
>>> What are the trade-offs of doing it this way? Is it that we would
>>> introduce a hard dependency on iceberg-rust? Is that a risk that could make
>>> PyIceberg unusable for some people? I don't think that would be a problem
>>> since we already have a requirement for pyarrow for these cases.
>>>
>>> On Thu, Aug 1, 2024 at 6:39 AM Sung Yun <su...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> This is something I've been mulling about for a while and I thought
>>>> this would be the right forum to discuss this topic as a follow up to a
>>>> similar topic discussion thread on using python bindings from iceberg-rust
>>>> to support pyiceberg.
>>>>
>>>> As soon as we released 0.7.0 which supports writes into tables with
>>>> TimeTransform partitions
>>>> <https://github.com/apache/iceberg-python/pull/784/files>, our
>>>> prospective users started asking about the support for Bucket Transform
>>>> partitions.
>>>>
>>>> Iceberg has a custom logic for Bucket partitions (Thanks for the link
>>>> <https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I
>>>> took a look into the Java code
>>>> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
>>>> and I think it looks somewhat like:
>>>>
>>>> * mmh3_hash(val) mod (num_buckets)
>>>>
>>>> And has field type specific logic so that each type is hashed
>>>> appropriately.
>>>>
>>>> Unfortunately there is no existing pyarrow compute function that does
>>>> this, so I'd like to propose that we write the function in iceberg-rust
>>>> that takes an Arrow Array reference and the bucket number as the input,
>>>> that returns a new Arrow Array reference with the bucket values evaluated
>>>> that corresponds to the input Arrow Array in the same order.
>>>>
>>>> When iceberg-rust becomes more mature, I believe that the same
>>>> underlying transform function can be reused for bucket partitions within
>>>> this repository, and in the interim we could support writes into Bucket
>>>> partitioned tables on PyIceberg by exposing this function as a Python
>>>> binding that we import into PyIceberg.
>>>>
>>>> I'd love to hear how folks feel about this idea!
>>>>
>>>>
>>>> Cross posted Discussion on iceberg-rust: #514
>>>> <https://github.com/apache/iceberg-rust/discussions/514>
>>>>
>>>>
>>>> Sung
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Databricks
>>>
>>
Re: [DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Posted by Fokko Driesprong <fo...@apache.org>.
Hey everyone,
In the beginning of PyIceberg, one of the goals was to keep PyIceberg pure
Python. At some point, we've added a Cython Avro decoder because of
performance reasons, but we still have a pure Python fallback. Today you
can still do metadata operating using s3fs without any native code. Once
you start reading/writing data, you'll need to pull in PyArrow.
I think it would be a win-win situation where we can combine the momentum
of PyIceberg and push the things that Python doesn't do well (mostly
multithreading and heavy lifting) to Iceberg-Rust, and also have
Iceberg-Rust as a library for Rust query engines to interact with Iceberg
tables directly.
My main question here is about sequencing. Would pushing the bucket
transforms into Rust a good exercise to get the scaffolding in place?
Kind regards,
Fokko
Op vr 2 aug 2024 om 16:38 schreef Renjie Liu <li...@gmail.com>:
> Hi:
>
> Thanks Sung for raising this. Just as Ryan said, I'm also +1 for using
> more iceberg-rust in pyiceberg.
>
> Is it that we would introduce a hard dependency on iceberg-rust?
>
>
> I think so.
>
> Is that a risk that could make PyIceberg unusable for some people? I don't
>> think that would be a problem since we already have a requirement for
>> pyarrow for these cases.
>
>
> +1.
>
> In fact, what I think more about is making iceberg-rust the backend of
> pyiceberg, and this is what me, xuanwo and Fokko had talked about in
> brainstorming several times. Not only bucket transform, but also not only
> FileIO in another thread initiated by xuanwo. I think these are all
> intermediate steps to our final goal.
>
>
> On Fri, Aug 2, 2024 at 3:21 AM Ryan Blue <bl...@databricks.com.invalid>
> wrote:
>
>> In general, I think the idea of using iceberg-rust more from PyIceberg is
>> great. I think it will be a good path to pushing more things down to native
>> code.
>>
>> What are the trade-offs of doing it this way? Is it that we would
>> introduce a hard dependency on iceberg-rust? Is that a risk that could make
>> PyIceberg unusable for some people? I don't think that would be a problem
>> since we already have a requirement for pyarrow for these cases.
>>
>> On Thu, Aug 1, 2024 at 6:39 AM Sung Yun <su...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> This is something I've been mulling about for a while and I thought this
>>> would be the right forum to discuss this topic as a follow up to a similar
>>> topic discussion thread on using python bindings from iceberg-rust to
>>> support pyiceberg.
>>>
>>> As soon as we released 0.7.0 which supports writes into tables with
>>> TimeTransform partitions
>>> <https://github.com/apache/iceberg-python/pull/784/files>, our
>>> prospective users started asking about the support for Bucket Transform
>>> partitions.
>>>
>>> Iceberg has a custom logic for Bucket partitions (Thanks for the link
>>> <https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I
>>> took a look into the Java code
>>> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
>>> and I think it looks somewhat like:
>>>
>>> * mmh3_hash(val) mod (num_buckets)
>>>
>>> And has field type specific logic so that each type is hashed
>>> appropriately.
>>>
>>> Unfortunately there is no existing pyarrow compute function that does
>>> this, so I'd like to propose that we write the function in iceberg-rust
>>> that takes an Arrow Array reference and the bucket number as the input,
>>> that returns a new Arrow Array reference with the bucket values evaluated
>>> that corresponds to the input Arrow Array in the same order.
>>>
>>> When iceberg-rust becomes more mature, I believe that the same
>>> underlying transform function can be reused for bucket partitions within
>>> this repository, and in the interim we could support writes into Bucket
>>> partitioned tables on PyIceberg by exposing this function as a Python
>>> binding that we import into PyIceberg.
>>>
>>> I'd love to hear how folks feel about this idea!
>>>
>>>
>>> Cross posted Discussion on iceberg-rust: #514
>>> <https://github.com/apache/iceberg-rust/discussions/514>
>>>
>>>
>>> Sung
>>>
>>
>>
>> --
>> Ryan Blue
>> Databricks
>>
>
Re: [DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Posted by Renjie Liu <li...@gmail.com>.
Hi:
Thanks Sung for raising this. Just as Ryan said, I'm also +1 for using more
iceberg-rust in pyiceberg.
Is it that we would introduce a hard dependency on iceberg-rust?
I think so.
Is that a risk that could make PyIceberg unusable for some people? I don't
> think that would be a problem since we already have a requirement for
> pyarrow for these cases.
+1.
In fact, what I think more about is making iceberg-rust the backend of
pyiceberg, and this is what me, xuanwo and Fokko had talked about in
brainstorming several times. Not only bucket transform, but also not only
FileIO in another thread initiated by xuanwo. I think these are all
intermediate steps to our final goal.
On Fri, Aug 2, 2024 at 3:21 AM Ryan Blue <bl...@databricks.com.invalid>
wrote:
> In general, I think the idea of using iceberg-rust more from PyIceberg is
> great. I think it will be a good path to pushing more things down to native
> code.
>
> What are the trade-offs of doing it this way? Is it that we would
> introduce a hard dependency on iceberg-rust? Is that a risk that could make
> PyIceberg unusable for some people? I don't think that would be a problem
> since we already have a requirement for pyarrow for these cases.
>
> On Thu, Aug 1, 2024 at 6:39 AM Sung Yun <su...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> This is something I've been mulling about for a while and I thought this
>> would be the right forum to discuss this topic as a follow up to a similar
>> topic discussion thread on using python bindings from iceberg-rust to
>> support pyiceberg.
>>
>> As soon as we released 0.7.0 which supports writes into tables with
>> TimeTransform partitions
>> <https://github.com/apache/iceberg-python/pull/784/files>, our
>> prospective users started asking about the support for Bucket Transform
>> partitions.
>>
>> Iceberg has a custom logic for Bucket partitions (Thanks for the link
>> <https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I
>> took a look into the Java code
>> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
>> and I think it looks somewhat like:
>>
>> * mmh3_hash(val) mod (num_buckets)
>>
>> And has field type specific logic so that each type is hashed
>> appropriately.
>>
>> Unfortunately there is no existing pyarrow compute function that does
>> this, so I'd like to propose that we write the function in iceberg-rust
>> that takes an Arrow Array reference and the bucket number as the input,
>> that returns a new Arrow Array reference with the bucket values evaluated
>> that corresponds to the input Arrow Array in the same order.
>>
>> When iceberg-rust becomes more mature, I believe that the same underlying
>> transform function can be reused for bucket partitions within this
>> repository, and in the interim we could support writes into Bucket
>> partitioned tables on PyIceberg by exposing this function as a Python
>> binding that we import into PyIceberg.
>>
>> I'd love to hear how folks feel about this idea!
>>
>>
>> Cross posted Discussion on iceberg-rust: #514
>> <https://github.com/apache/iceberg-rust/discussions/514>
>>
>>
>> Sung
>>
>
>
> --
> Ryan Blue
> Databricks
>
Re: [DISCUSS] Use iceberg-rust for PyIceberg Bucket Transform
Posted by Ryan Blue <bl...@databricks.com.INVALID>.
In general, I think the idea of using iceberg-rust more from PyIceberg is
great. I think it will be a good path to pushing more things down to native
code.
What are the trade-offs of doing it this way? Is it that we would introduce
a hard dependency on iceberg-rust? Is that a risk that could make PyIceberg
unusable for some people? I don't think that would be a problem since we
already have a requirement for pyarrow for these cases.
On Thu, Aug 1, 2024 at 6:39 AM Sung Yun <su...@gmail.com> wrote:
> Hi everyone,
>
> This is something I've been mulling about for a while and I thought this
> would be the right forum to discuss this topic as a follow up to a similar
> topic discussion thread on using python bindings from iceberg-rust to
> support pyiceberg.
>
> As soon as we released 0.7.0 which supports writes into tables with
> TimeTransform partitions
> <https://github.com/apache/iceberg-python/pull/784/files>, our
> prospective users started asking about the support for Bucket Transform
> partitions.
>
> Iceberg has a custom logic for Bucket partitions (Thanks for the link
> <https://iceberg.apache.org/spec/#bucket-transform-details> Fokko). I
> took a look into the Java code
> <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/transforms/Bucket.java#L99>
> and I think it looks somewhat like:
>
> * mmh3_hash(val) mod (num_buckets)
>
> And has field type specific logic so that each type is hashed
> appropriately.
>
> Unfortunately there is no existing pyarrow compute function that does
> this, so I'd like to propose that we write the function in iceberg-rust
> that takes an Arrow Array reference and the bucket number as the input,
> that returns a new Arrow Array reference with the bucket values evaluated
> that corresponds to the input Arrow Array in the same order.
>
> When iceberg-rust becomes more mature, I believe that the same underlying
> transform function can be reused for bucket partitions within this
> repository, and in the interim we could support writes into Bucket
> partitioned tables on PyIceberg by exposing this function as a Python
> binding that we import into PyIceberg.
>
> I'd love to hear how folks feel about this idea!
>
>
> Cross posted Discussion on iceberg-rust: #514
> <https://github.com/apache/iceberg-rust/discussions/514>
>
>
> Sung
>
--
Ryan Blue
Databricks