[DISCUSS][RIP-66] Support KV(Rocksdb) Storage

[闽 <fu...@qq.com.INVALID>]

Hi RocketMQ Community: 



We have encountered and been plagued by some probelms in the actual production environment. In some actual production scenarios, millions of topics and subsriptions will be created, and both of them may be deleted and created frequently. In current architecture, both topics and subcriptions are persisted in real time, which means each request of topics and subsciptions updating will trigger the persist interface, and the persistence of such metadata is written in full rather than append-only. 



In the scenario of millions of topics, frequent persistence generates the large memory object jsonString of the topicConfigTable. When the memory is tight, the large memory object jsonString will be directly allocated to the old generation, resulting in frequent Full GC.



In addition, millions of topics will inevitably bring millions of indexed ConsumeQueue small files, a large number of indexed small files will destroy the advantages of rocketmq sequential writing, and the random writing of ConsumeQueue will make the performance drop sharply.



We have already done part of the work. Our proposals are provided at the links below：

https://docs.google.com/document/d/e/2PACX-1vT3QJ8XDbKi8HQKaIDodOD4vIV0-xlERFtg8Ed3FsYUi7IfRs5JVKjElGaxpk-0ErH0xdWT0Vv-a1u5/pub



Please welcome to reply to this email or comment on the proposal if you have any questions or suggestions.





Thanks, 

FujianZhao




left
2573259572@qq.com