You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by wang <zh...@gmail.com> on 2026/06/22 06:00:00 UTC
[DISCUSS] Support Deletion Vectors for DataEvolution Tables
Hi Paimon community, I would like to start a discussion about supporting
Deletion Vectors (DVs) for DataEvolution tables. In many AI scenarios,
users frequently need to delete data (e.g., removing low-quality samples,
deduplication, or excluding biased training data). The existing file-level
DV in AppendTable is incompatible with DataEvolution, and rewriting files
for random deletions causes small-file explosion and index invalidation. I
prepared a short design document proposing a range-based DV approach, along
with solutions for Merge Into, Compaction, and Vector Index adaptation:
https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing
Could you please take a look and share your thoughts? Best, wang
Re: [DISCUSS] Support Deletion Vectors for DataEvolution Tables
Posted by Jingsong Li <ji...@gmail.com>.
Thanks wang,
Sounds good to me.
Best,
Jingsong
On Mon, Jun 22, 2026 at 2:01 PM wang <zh...@gmail.com> wrote:
>
> Hi Paimon community, I would like to start a discussion about supporting
> Deletion Vectors (DVs) for DataEvolution tables. In many AI scenarios,
> users frequently need to delete data (e.g., removing low-quality samples,
> deduplication, or excluding biased training data). The existing file-level
> DV in AppendTable is incompatible with DataEvolution, and rewriting files
> for random deletions causes small-file explosion and index invalidation. I
> prepared a short design document proposing a range-based DV approach, along
> with solutions for Merge Into, Compaction, and Vector Index adaptation:
> https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing
> Could you please take a look and share your thoughts? Best, wang
Re: [DISCUSS] Support Deletion Vectors for DataEvolution Tables
Posted by JUNHAO YE <ye...@gmail.com>.
+1 for this～
Thanks, wang.
Best,
Junhao
> 2026年6月22日 14:00，wang <zh...@gmail.com> 写道：
>
> Hi Paimon community, I would like to start a discussion about supporting
> Deletion Vectors (DVs) for DataEvolution tables. In many AI scenarios,
> users frequently need to delete data (e.g., removing low-quality samples,
> deduplication, or excluding biased training data). The existing file-level
> DV in AppendTable is incompatible with DataEvolution, and rewriting files
> for random deletions causes small-file explosion and index invalidation. I
> prepared a short design document proposing a range-based DV approach, along
> with solutions for Merge Into, Compaction, and Vector Index adaptation:
> https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing
> Could you please take a look and share your thoughts? Best, wang