[GitHub] [iceberg] kbendick edited a comment on pull request #2680: Core: Add RocksDBStructLikeMap

[GitBox <gi...@apache.org>]

kbendick edited a comment on pull request #2680:
URL: https://github.com/apache/iceberg/pull/2680#issuecomment-987585529


   > Are we still considering this feature? I wonder what would be the performance of this versus in memory vectorized read + compaction. If the later is good enough, we can probably avoid introducing this complication.
   
   Discussion came back up, but I'm not sure.
   
   Besides other perceived possible benefits from RocksDB (which admittedly might not be enough to introduce the added complexity for some use cases), the motivating use case is really the potentially much larger than typical memory usage required to process the initial table snapshot for CDC, which is a common use case. To take a snapshot or begin a CDC stream from MySQL for example, a full table copy is required before the initial checkpoint can be taken iiuc. So this historically has lead users to have to majorly increase their initial memory / resource allocation and then tune down after the job had successfully checkpointed.
   
   I'll follow up with openinx and steven about the status of this feature and state of the problem in general.
   
   Without that issue, I'd lean towards keeping as few elements in state as possible to avoid having to worry about further serrialized forms. But I think the issue is still outstanding, as a full snapshot is needed before any deltas can be written iiuc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org