You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Anton Okolnychyi <ao...@gmail.com> on 2024/08/21 21:28:40 UTC
[DISCUSS] Improving Position Deletes in V3
Hey folks,
As discussed during the sync, I've been working on a proposal to improve
the handling of position deletes in V3. It builds on lessons learned from
deploying the current approach at scale and addresses all unresolved
questions from past community discussions and proposals around this topic.
In particular, the proposal attempts to address the following shortcomings
we observe today:
- Choosing between fewer delete files on disk or targeted deletes.
- Dependence on external maintenance for consistent write and read
performance.
- Writing and reading overhead as in-memory and on-disk representations
differ.
Please, take a look at the doc [1] and let me know what you think. Any
feedback is highly appreciated!
- Anton
[1] -
https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
Re: [DISCUSS] Improving Position Deletes in V3
Posted by Bryan Keller <br...@gmail.com>.
Thanks for the doc Anton, I reviewed it and it looks good to me. Let me know if I can help with anything, it is an area of interest for me.
-Bryan
> On Aug 21, 2024, at 2:28 PM, Anton Okolnychyi <ao...@gmail.com> wrote:
>
> Hey folks,
>
> As discussed during the sync, I've been working on a proposal to improve the handling of position deletes in V3. It builds on lessons learned from deploying the current approach at scale and addresses all unresolved questions from past community discussions and proposals around this topic.
>
> In particular, the proposal attempts to address the following shortcomings we observe today:
> Choosing between fewer delete files on disk or targeted deletes.
> Dependence on external maintenance for consistent write and read performance.
> Writing and reading overhead as in-memory and on-disk representations differ.
> Please, take a look at the doc [1] and let me know what you think. Any feedback is highly appreciated!
>
> - Anton
>
> [1] - https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
>
>
Re: [DISCUSS] Improving Position Deletes in V3
Posted by Russell Spitzer <ru...@gmail.com>.
+1 I am on board, all of my doubts are resolved
On Thu, Sep 5, 2024 at 3:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Anton
>
> Sorry for the late reply on this proposal.
> I like it ! It looks good to me (I have a few minor comments).
>
> It would be great to include this spec update in V3.
> Please let me know if I can help on this !
>
> Regards
> JB
>
> On Wed, Aug 21, 2024 at 11:28 PM Anton Okolnychyi <ao...@gmail.com>
> wrote:
> >
> > Hey folks,
> >
> > As discussed during the sync, I've been working on a proposal to improve
> the handling of position deletes in V3. It builds on lessons learned from
> deploying the current approach at scale and addresses all unresolved
> questions from past community discussions and proposals around this topic.
> >
> > In particular, the proposal attempts to address the following
> shortcomings we observe today:
> >
> > Choosing between fewer delete files on disk or targeted deletes.
> > Dependence on external maintenance for consistent write and read
> performance.
> > Writing and reading overhead as in-memory and on-disk representations
> differ.
> >
> > Please, take a look at the doc [1] and let me know what you think. Any
> feedback is highly appreciated!
> >
> > - Anton
> >
> > [1] -
> https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
> >
> >
>
Re: [DISCUSS] Improving Position Deletes in V3
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Anton
Sorry for the late reply on this proposal.
I like it ! It looks good to me (I have a few minor comments).
It would be great to include this spec update in V3.
Please let me know if I can help on this !
Regards
JB
On Wed, Aug 21, 2024 at 11:28 PM Anton Okolnychyi <ao...@gmail.com> wrote:
>
> Hey folks,
>
> As discussed during the sync, I've been working on a proposal to improve the handling of position deletes in V3. It builds on lessons learned from deploying the current approach at scale and addresses all unresolved questions from past community discussions and proposals around this topic.
>
> In particular, the proposal attempts to address the following shortcomings we observe today:
>
> Choosing between fewer delete files on disk or targeted deletes.
> Dependence on external maintenance for consistent write and read performance.
> Writing and reading overhead as in-memory and on-disk representations differ.
>
> Please, take a look at the doc [1] and let me know what you think. Any feedback is highly appreciated!
>
> - Anton
>
> [1] - https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
>
>
Re: [DISCUSS] Improving Position Deletes in V3
Posted by Manu Zhang <ow...@gmail.com>.
Anton,
Thanks for the write-up. It's very entertaining to read. I left some minor
comments on the doc. +1 to the proposal.
Regards,
Manu
On Tue, Aug 27, 2024 at 7:21 AM Steven Wu <st...@gmail.com> wrote:
> Anton,
>
> Thanks a lot for the improvement proposal and great write-up with
> quantitative supporting arguments.
>
> +1 from my side.
>
> Thanks,
> Steven
>
>
> On Wed, Aug 21, 2024 at 2:29 PM Anton Okolnychyi <ao...@gmail.com>
> wrote:
>
>> Hey folks,
>>
>> As discussed during the sync, I've been working on a proposal to improve
>> the handling of position deletes in V3. It builds on lessons learned from
>> deploying the current approach at scale and addresses all unresolved
>> questions from past community discussions and proposals around this topic.
>>
>> In particular, the proposal attempts to address the following
>> shortcomings we observe today:
>>
>> - Choosing between fewer delete files on disk or targeted deletes.
>> - Dependence on external maintenance for consistent write and read
>> performance.
>> - Writing and reading overhead as in-memory and on-disk
>> representations differ.
>>
>> Please, take a look at the doc [1] and let me know what you think. Any
>> feedback is highly appreciated!
>>
>> - Anton
>>
>> [1] -
>> https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
>>
>>
>>
Re: [DISCUSS] Improving Position Deletes in V3
Posted by Steven Wu <st...@gmail.com>.
Anton,
Thanks a lot for the improvement proposal and great write-up with
quantitative supporting arguments.
+1 from my side.
Thanks,
Steven
On Wed, Aug 21, 2024 at 2:29 PM Anton Okolnychyi <ao...@gmail.com>
wrote:
> Hey folks,
>
> As discussed during the sync, I've been working on a proposal to improve
> the handling of position deletes in V3. It builds on lessons learned from
> deploying the current approach at scale and addresses all unresolved
> questions from past community discussions and proposals around this topic.
>
> In particular, the proposal attempts to address the following shortcomings
> we observe today:
>
> - Choosing between fewer delete files on disk or targeted deletes.
> - Dependence on external maintenance for consistent write and read
> performance.
> - Writing and reading overhead as in-memory and on-disk
> representations differ.
>
> Please, take a look at the doc [1] and let me know what you think. Any
> feedback is highly appreciated!
>
> - Anton
>
> [1] -
> https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM
>
>
>