Re: [PR] [core] Add async bucket refresh for dynamic bucket mode [paimon]

["irodriguezclaveria (via GitHub)" <gi...@apache.org>]

irodriguezclaveria commented on PR #6426:
URL: https://github.com/apache/paimon/pull/6426#issuecomment-5159517771

   Hi @JingsongLi,
   
   I completely agree with your assessment! The test failures and bugs you just mentioned are exactly why I realized the current PR implementation (using async refreshes and the HASH index) is fundamentally flawed and should not be merged.
   
   That is why my previous message was proposing a completely new design to replace this PR. The new plan (tracking live keys entirely in memory via RowKind insert/deletes) completely eliminates async refreshes, disk scans, and HASH index dependencies. By removing those mechanics, it naturally prevents the double-counting and vanished-bucket issues you ran into during regression testing.
   
   I haven't written the code for this new design yet. Could you take a look at the conceptual plan in my previous comment? I'd love to get your thoughts on that new direction before I start implementing it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org