You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "LetLetMe (via GitHub)" <gi...@apache.org> on 2024/08/28 02:37:22 UTC
[I] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
LetLetMe opened a new issue, #8589:
URL: https://github.com/apache/rocketmq/issues/8589
### Before Creating the Enhancement Request
- [X] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
### Summary
支持cq文件和Offset从json版本升级到rocksdb版本
Support Upgrade of CQ Files and Offset from JSON Version to RocksDB Version
### Motivation
我们百万队列引入了rocksdb来存储cq和Offset,但是现在没有从json版原地升级到rocksdb版的方案,此方案就是为了解决这个问题
We have introduced RocksDB to store CQ and Offset in our million queues, but currently, there is no solution for in-place upgrading from the JSON version to the RocksDB version. This proposal aims to address this issue.
### Describe the Solution You'd Like
对于Offset,我们会在transferOffsetJsonToRocksdb打开时,broker启动时候做转换。
对于cq,我们提供了俩种格式cq双写的模式,通过rocksdbCQWriteEnable来控制是否同时Dispatch俩种格式的cq,同时我们还提供了检测双写进度的工具,方便寻找合适的时间以切流。
For Offset, we will perform the conversion when the broker starts if transferOffsetJsonToRocksdb is enabled. For CQ, we offer a dual-write mode for two formats, controlled by rocksdbCQWriteEnable to determine whether to dispatch both formats of CQ simultaneously. Additionally, we provide a tool to monitor the dual-write progress, making it easier to find an appropriate time to switch over.
### Describe Alternatives You've Considered
nothing
### Additional Context
nothing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on issue #8589:
URL: https://github.com/apache/rocketmq/issues/8589#issuecomment-2367894420
> 在第二阶段时,当 RocketDB 开始写入 cq 位点,但在一些场景下,内存中的 cq(RocksDBConsumeQueueStore#bufferDRList) 数据没有完成持久化。重启时,原来文件版cq 是正常的,但 rocketdb 中的 cq 有些缺失,也就说如何保障原来 cq 和 RocketDB 中的 cq 事务性
>
> In the second phase, when RocketDB starts writing the cq(RocksDBConsumeQueueStore#bufferDRList) offset, there are scenarios where the cq data in memory hasn't been fully persisted. Upon reboot, the original file-based cq is intact, but there are some losses in the cq within RocketDB. This raises the question of how to ensure the transactional consistency between the original cq and the cq in RocketDB
1. 如果双写过程中出现kvCq重启导致落后,那我们什么都不需要做,多双写一段时间,等待再次对齐就好了
2. 如果是kv模式下重启,那么会走正常的revover逻辑
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins closed issue #8589: [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB
URL: https://github.com/apache/rocketmq/issues/8589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "fuyou001 (via GitHub)" <gi...@apache.org>.
fuyou001 commented on issue #8589:
URL: https://github.com/apache/rocketmq/issues/8589#issuecomment-2324575243
在第二阶段时,当 RocketDB 开始写入 cq 位点,但在一些场景下,内存中的 cq(RocksDBConsumeQueueStore#bufferDRList) 数据没有完成持久化。重启时,原来的 文件版cq 是正常的,但 rocketdb 中的 cq 有些缺失,也就说如何保障原来 cq 和 RocketDB 中的 cq 事务性
In the second phase, when RocketDB starts writing the cq(RocksDBConsumeQueueStore#bufferDRList) offset, there are scenarios where the cq data in memory hasn't been fully persisted. Upon reboot, the original file-based cq is intact, but there are some losses in the cq within RocketDB. This raises the question of how to ensure the transactional consistency between the original cq and the cq in RocketDB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org