You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhongyujiang (via GitHub)" <gi...@apache.org> on 2024/12/09 13:13:10 UTC

[I] [Bug] Aggregation table retract issue [paimon]

zhongyujiang opened a new issue, #4670:
URL: https://github.com/apache/paimon/issues/4670

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   master
   
   ### Compute Engine
   
   core
   
   ### Minimal reproduce step
   
   Have writen some unit tests to demonstrate this issue, see https://github.com/apache/paimon/compare/master...zhongyujiang:incubator-paimon:gh/retract-issue
   
   ### What doesn't meet your expectations?
   
   The retraction result of the agg table might be incorrect:
   - the early aggregation before write will merge retract records with the same key into a single +I record
   - during compaction, retract records are eliminated, even if there are still records with the same key at lower levels in the LSM tree that are supposed to be retracted
   
   These two points will cause retraction on agg table to fail, please see the unit tests I wrote in the above link. 
   
   It seems we need to handle retracts in a similar way to how we handle delete tombstones in LSM, but that would result in both an insert and a delete record for the same key existing within a single sorted run, which might be challenging to handle I guess?
   
   cc @JingsongLi @tsreaper could you please take a look? Thanks
   
   
   
   
   
   
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Aggregation table retract issue [paimon]

Posted by "xiangyuf (via GitHub)" <gi...@apache.org>.
xiangyuf commented on issue #4670:
URL: https://github.com/apache/paimon/issues/4670#issuecomment-2568924290

   I second with you. The early arrived retract record should not be considered as a single +I record.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org