You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhongyujiang (via GitHub)" <gi...@apache.org> on 2024/12/09 13:13:10 UTC
[I] [Bug] Aggregation table retract issue [paimon]
zhongyujiang opened a new issue, #4670:
URL: https://github.com/apache/paimon/issues/4670
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
### Compute Engine
core
### Minimal reproduce step
Have writen some unit tests to demonstrate this issue, see https://github.com/apache/paimon/compare/master...zhongyujiang:incubator-paimon:gh/retract-issue
### What doesn't meet your expectations?
The retraction result of the agg table might be incorrect:
- the early aggregation before write will merge retract records with the same key into a single +I record
- during compaction, retract records are eliminated, even if there are still records with the same key at lower levels in the LSM tree that are supposed to be retracted
These two points will cause retraction on agg table to fail, please see the unit tests I wrote in the above link.
It seems we need to handle retracts in a similar way to how we handle delete tombstones in LSM, but that would result in both an insert and a delete record for the same key existing within a single sorted run, which might be challenging to handle I guess?
cc @JingsongLi @tsreaper could you please take a look? Thanks
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Aggregation table retract issue [paimon]
Posted by "xiangyuf (via GitHub)" <gi...@apache.org>.
xiangyuf commented on issue #4670:
URL: https://github.com/apache/paimon/issues/4670#issuecomment-2568924290
I second with you. The early arrived retract record should not be considered as a single +I record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org