You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "LetLetMe (via GitHub)" <gi...@apache.org> on 2024/08/29 07:05:09 UTC
[PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
LetLetMe opened a new pull request, #8600:
URL: https://github.com/apache/rocketmq/pull/8600
<!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
### Which Issue(s) This PR Fixes
<!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
Fixes #8589
### Brief Description
<!-- Write a brief description for your pull request to help the maintainer understand the reasons behind your changes. -->
### How Did You Test This Change?
<!-- In order to ensure the code quality of Apache RocketMQ, we expect every pull request to have undergone thorough testing. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758905265
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java:
##########
@@ -373,6 +374,25 @@ public void setDataVersion(DataVersion dataVersion) {
this.dataVersion = dataVersion;
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ ConsumerOffsetManager obj = RemotingSerializable.fromJson(jsonString, ConsumerOffsetManager.class);
+ if (obj != null) {
+ this.dataVersion = obj.dataVersion;
+ }
+ LOG.info("load consumer offset dataVersion success, " + fileName + " " + jsonString);
Review Comment:
There are many other places in this PR having this issue, plz fix them too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe closed pull request #8600: [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB
URL: https://github.com/apache/rocketmq/pull/8600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1751635805
##########
broker/src/test/java/org/apache/rocketmq/broker/offset/RocksdbTransferOffsetAndCqTest.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.broker.offset;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.commons.collections.MapUtils;
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.MixAll;
+import org.apache.rocketmq.common.Pair;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.RocksDBMessageStore;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.ConsumeQueueStoreInterface;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.rocksdb.RocksDBException;
+
+@RunWith(MockitoJUnitRunner.class)
+public class RocksdbTransferOffsetAndCqTest {
+
+ private final String basePath = Paths.get(System.getProperty("user.home"),
+ "unit-test-store", UUID.randomUUID().toString().substring(0, 16).toUpperCase()).toString();
+
+ private final String topic = "topic";
+ private final String group = "group";
+ private final String clientHost = "clientHost";
+ private final int queueId = 1;
+
+ private RocksDBConsumerOffsetManager rocksdbConsumerOffsetManager;
+
+ private ConsumerOffsetManager consumerOffsetManager;
+
+ private DefaultMessageStore defaultMessageStore;
+
+ @Mock
+ private BrokerController brokerController;
+
+ @Before
+ public void init() throws IOException {
+ if (notToBeExecuted()) {
+ return;
+ }
+ BrokerConfig brokerConfig = new BrokerConfig();
+ brokerConfig.setConsumerOffsetUpdateVersionStep(10);
+ MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+ messageStoreConfig.setStorePathRootDir(basePath);
+ messageStoreConfig.setTransferOffsetJsonToRocksdb(true);
+ messageStoreConfig.setRocksdbCQDoubleWriteEnable(true);
+ Mockito.lenient().when(brokerController.getBrokerConfig()).thenReturn(brokerConfig);
+ Mockito.lenient().when(brokerController.getMessageStoreConfig()).thenReturn(messageStoreConfig);
+
+ defaultMessageStore = new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("aaa", true), null,
+ brokerConfig, new ConcurrentHashMap<String, TopicConfig>());
+ defaultMessageStore.enableRocksdbCQWrite();
+ defaultMessageStore.loadCheckPoint();
+
+ consumerOffsetManager = new ConsumerOffsetManager(brokerController);
+ consumerOffsetManager.load();
+
+ rocksdbConsumerOffsetManager = new RocksDBConsumerOffsetManager(brokerController);
+ }
+
+ @Test
+ public void testTransferOffset() {
+ if (notToBeExecuted()) {
+ return;
+ }
+
+ for (int i = 0; i < 200; i++) {
+ consumerOffsetManager.commitOffset(clientHost, group, topic, queueId, i);
+ }
+
+ ConcurrentMap<String, ConcurrentMap<Integer, Long>> offsetTable = consumerOffsetManager.getOffsetTable();
+ ConcurrentMap<Integer, Long> map = offsetTable.get(topic + "@" + group);
+ Assert.assertTrue(MapUtils.isNotEmpty(map));
+
+ Long offset = map.get(queueId);
+ Assert.assertEquals(199L, (long) offset);
+
+ long offsetDataVersion = consumerOffsetManager.getDataVersion().getCounter().get();
+ Assert.assertEquals(20L, offsetDataVersion);
+
+ consumerOffsetManager.persist();
+
+ boolean loadResult = rocksdbConsumerOffsetManager.load();
+ Assert.assertTrue(loadResult);
+
+ ConcurrentMap<String, ConcurrentMap<Integer, Long>> rocksdbOffsetTable = rocksdbConsumerOffsetManager.getOffsetTable();
+
+ ConcurrentMap<Integer, Long> rocksdbMap = rocksdbOffsetTable.get(topic + "@" + group);
+ Assert.assertTrue(MapUtils.isNotEmpty(rocksdbMap));
+
+ Long aLong1 = rocksdbMap.get(queueId);
+ Assert.assertEquals(199L, (long) aLong1);
+
+ long rocksdbOffset = rocksdbConsumerOffsetManager.getDataVersion().getCounter().get();
+ Assert.assertEquals(21L, rocksdbOffset);
+ }
+
+ @Test
+ public void testRocksdbCqWrite() throws RocksDBException {
+ if (notToBeExecuted()) {
+ return;
+ }
+ RocksDBMessageStore kvStore = defaultMessageStore.getRocksDBMessageStore();
+ ConsumeQueueStoreInterface store = kvStore.getConsumeQueueStore();
+ ConsumeQueueInterface rocksdbCq = defaultMessageStore.getRocksDBMessageStore().findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface fileCq = defaultMessageStore.findConsumeQueue(topic, queueId);
+ for (int i = 0; i < 200; i++) {
+ DispatchRequest request = new DispatchRequest(topic, queueId, i, 200, 0, System.currentTimeMillis(), i, "", "", 0, 0, new HashMap<>());
+ fileCq.putMessagePositionInfoWrapper(request);
+ store.putMessagePositionInfoWrapper(request);
+ }
+ Pair<CqUnit, Long> unit = rocksdbCq.getCqUnitAndStoreTime(100);
+ Pair<CqUnit, Long> unit1 = fileCq.getCqUnitAndStoreTime(100);
+ Assert.assertTrue(unit.getObject1().getPos() == unit1.getObject1().getPos());
+ }
+
+ private boolean notToBeExecuted() {
+ return MixAll.isMac();
Review Comment:
这个有什么问题吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737993975
##########
broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java:
##########
@@ -637,6 +637,26 @@ public String encode() {
return encode(false);
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ TopicConfigSerializeWrapper topicConfigSerializeWrapper =
+ TopicConfigSerializeWrapper.fromJson(jsonString, TopicConfigSerializeWrapper.class);
+ if (topicConfigSerializeWrapper != null) {
+ this.dataVersion.assignNewOne(topicConfigSerializeWrapper.getDataVersion());
+ }
+ log.info("load topic metadata dataVersion success" + fileName + " " + topicConfigSerializeWrapper.getDataVersion());
Review Comment:
log should use template interpolation. aka, log.info("bla...{}", value);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737983778
##########
tools/src/main/java/org/apache/rocketmq/tools/command/export/ExportMetadataInRocksDBCommand.java:
##########
@@ -38,6 +41,17 @@ public class ExportMetadataInRocksDBCommand implements SubCommand {
private static final String TOPICS_JSON_CONFIG = "topics";
private static final String SUBSCRIPTION_GROUP_JSON_CONFIG = "subscriptionGroups";
+ public static void main(String[] args) throws SubCommandException {
Review Comment:
Should be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759648623
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/RocksDBConsumerOffsetManager.java:
##########
@@ -16,36 +16,79 @@
*/
package org.apache.rocketmq.broker.offset;
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.serializer.SerializerFeature;
import java.io.File;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentMap;
-
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.RocksDBConfigManager;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.utils.DataConverter;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.remoting.protocol.DataVersion;
import org.rocksdb.WriteBatch;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.serializer.SerializerFeature;
-
public class RocksDBConsumerOffsetManager extends ConsumerOffsetManager {
+ protected static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+
protected RocksDBConfigManager rocksDBConfigManager;
public RocksDBConsumerOffsetManager(BrokerController brokerController) {
super(brokerController);
- this.rocksDBConfigManager = new RocksDBConfigManager(configFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
+ this.rocksDBConfigManager = new RocksDBConfigManager(rocksdbConfigFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
}
@Override
public boolean load() {
if (!rocksDBConfigManager.init()) {
return false;
}
- return this.rocksDBConfigManager.loadData(this::decodeOffset);
+ if (!loadDataVersion() || !loadConsumerOffset()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ public boolean loadConsumerOffset() {
+ return this.rocksDBConfigManager.loadData(this::decodeOffset) && merge();
+ }
+
+ private boolean merge() {
+ if (!brokerController.getMessageStoreConfig().isTransferOffsetJsonToRocksdb()) {
+ log.info("the switch transferOffsetJsonToRocksdb is off, no merge offset operation is needed.");
+ return true;
+ }
+ if (!UtilAll.isPathExists(this.configFilePath()) && !UtilAll.isPathExists(this.configFilePath() + ".bak")) {
+ log.info("consumerOffset json file does not exist, so skip merge");
+ return true;
+ }
+ if (!super.loadDataVersion()) {
+ log.error("load json consumerOffset dataVersion error, startup will exit");
+ return false;
+ }
+
+ final DataVersion dataVersion = super.getDataVersion();
+ final DataVersion kvDataVersion = this.getDataVersion();
+ if (dataVersion.getCounter().get() > kvDataVersion.getCounter().get()) {
+ if (!super.load()) {
+ log.error("load json consumerOffset info failed, startup will exit");
+ return false;
+ }
+ this.persist();
+ this.getDataVersion().assignNewOne(dataVersion);
+ updateDataVersion();
Review Comment:
这里不原子应该也没问题,只要updateDataVersion在persist之后就可以,最差的情况是原模原样重新导入一次
It should be fine even if this is not atomic, as long as updateDataVersion occurs after persist. In the worst-case scenario, it would just require re-importing in the same state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1751294150
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -458,6 +464,79 @@ private RemotingCommand updateAndGetGroupForbidden(ChannelHandlerContext ctx, Re
return response;
}
+ private RemotingCommand diffConsumeQueue(ChannelHandlerContext ctx, RemotingCommand request) {
+ final RemotingCommand response = RemotingCommand.createResponseCommand(null);
+ response.setCode(ResponseCode.SUCCESS);
+ response.setBody(JSON.toJSONBytes(ImmutableMap.of("diffResult", "diff success, very good!")));
+
+ DefaultMessageStore messageStore = (DefaultMessageStore) brokerController.getMessageStore();
+ RocksDBMessageStore rocksDBMessageStore = messageStore.getRocksDBMessageStore();
+
+ if (!messageStore.getMessageStoreConfig().isRocksdbCQWriteEnable()) {
+ response.setBody(JSON.toJSONBytes(ImmutableMap.of("diffResult", "RocksdbCQWriteEnable is false, diffConsumeQueue is invalid")));
+ return response;
+ }
+
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> cqTable = messageStore.getConsumeQueueTable();
+ Random random = new Random();
+ for (Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> topicToCqListEntry : cqTable.entrySet()) {
+ String topic = topicToCqListEntry.getKey();
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueIdToCqMap = topicToCqListEntry.getValue();
+ for (Map.Entry<Integer, ConsumeQueueInterface> queueIdToCqEntry : queueIdToCqMap.entrySet()) {
+ Integer queueId = queueIdToCqEntry.getKey();
+
+ ConsumeQueueInterface jsonCq = queueIdToCqEntry.getValue();
+ ConsumeQueueInterface kvCq = rocksDBMessageStore.getConsumeQueue(topic, queueId);
+
+ CqUnit kvCqEarliestUnit = kvCq.getEarliestUnit();
+ CqUnit kvCqLatestUnit = kvCq.getLatestUnit();
+ CqUnit jsonCqEarliestUnit = jsonCq.getEarliestUnit();
+ CqUnit jsonCqLatestUnit = jsonCq.getLatestUnit();
+ LOGGER.info("diffConsumeQueue topic:{}, queue:{}, kvCqEarliestUnit:{}, jsonCqEarliestUnit:{}, kvCqLatestUnit:{}, jsonCqLatestUnit:{}",
+ topic, queueId, kvCqEarliestUnit, jsonCqEarliestUnit, kvCqLatestUnit, jsonCqLatestUnit);
+
+ long jsonOffset = jsonCq.getMaxOffsetInQueue() - 1;
+ int sampleCount = 10;
+
+ Set<Long> sampledOffsets = new HashSet<>();
+
+ if (jsonOffset > 100) {
Review Comment:
已改,现在是全量对比
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1740760674
##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -424,6 +424,26 @@ public class MessageStoreConfig {
private boolean putConsumeQueueDataByFileChannel = true;
+ private boolean transferOffsetJsonToRocksdb = false;
+
+ private boolean rocksdbCQWriteEnable = false;
Review Comment:
The naming need to be clearer, for example, by emphasizing double writing and adding comments, otherwise users may confuse it with the parameters for opening rocksdb.
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -163,11 +163,13 @@ public class DefaultMessageStore implements MessageStore {
private volatile boolean shutdown = true;
protected boolean notifyMessageArriveInBatch = false;
- private StoreCheckpoint storeCheckpoint;
+ protected StoreCheckpoint storeCheckpoint;
private TimerMessageStore timerMessageStore;
private final LinkedList<CommitLogDispatcher> dispatcherList;
+ private RocksDBMessageStore rocksDBMessageStore;
+
Review Comment:
Creating a new class, such as RocksDBConsumeQueueDoubleWriteMessageStore, and then inheriting from DefaultMessageStore.java, and overriding some methods instead of modifying the original DefaultMessageStore, would be better?
##########
tools/src/main/java/org/apache/rocketmq/tools/command/queue/DiffConsumeQueueCommand.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.tools.command.queue;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.remoting.RPCHook;
+import org.apache.rocketmq.remoting.protocol.body.ClusterInfo;
+import org.apache.rocketmq.remoting.protocol.body.DiffConsumeQueueResponseBody;
+import org.apache.rocketmq.remoting.protocol.route.BrokerData;
+import org.apache.rocketmq.srvutil.ServerUtil;
+import org.apache.rocketmq.tools.admin.DefaultMQAdminExt;
+import org.apache.rocketmq.tools.command.SubCommand;
+
+public class DiffConsumeQueueCommand implements SubCommand {
Review Comment:
Does this command and its related functions need to be here? Or would it be better to place them in the commercial?"
##########
tools/src/main/java/org/apache/rocketmq/tools/command/export/ExportMetadataInRocksDBCommand.java:
##########
@@ -86,7 +101,7 @@ public void execute(CommandLine commandLine, Options options, RPCHook rpcHook) t
ConfigRocksDBStorage kvStore = new ConfigRocksDBStorage(path, true /* readOnly */);
if (!kvStore.start()) {
- System.out.print("RocksDB load error, path=" + path + "\n");
+ System.out.printf("RocksDB load error, path=" + path + "\n");
Review Comment:
No need to concatenate strings when using printf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737994434
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -339,6 +343,8 @@ public RemotingCommand processRequest(ChannelHandlerContext ctx,
return fetchAllConsumeStatsInBroker(ctx, request);
case RequestCode.QUERY_CONSUME_QUEUE:
return queryConsumeQueue(ctx, request);
+ case RequestCode.DIFF_CONSUME_QUEUE:
Review Comment:
How about change code to DIFF_CONSUME_QUEUE_FOR_ROCKSDB, means this admin tool is used to diff consumerQueue Content in file and rocksDB
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/RocksDBConsumerOffsetManager.java:
##########
@@ -103,4 +147,22 @@ private void putWriteBatch(final WriteBatch writeBatch, final String topicGroupN
byte[] valueBytes = JSON.toJSONBytes(wrapper, SerializerFeature.BrowserCompatible);
writeBatch.put(keyBytes, valueBytes);
}
+
+ @Override
+ public boolean loadDataVersion() {
+ return this.rocksDBConfigManager.loadDataVersion();
+ }
+
+ @Override
+ public DataVersion getDataVersion() {
+ return rocksDBConfigManager.getKvDataVersion();
+ }
+
+ public void updateDataVersion() {
+ try {
+ rocksDBConfigManager.updateKvDataVersion();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
Review Comment:
Add log here.
##########
store/src/main/java/org/apache/rocketmq/store/RocksDBMessageStore.java:
##########
@@ -178,4 +180,30 @@ public void initMetrics(Meter meter, Supplier<AttributesBuilder> attributesBuild
// Also add some metrics for rocksdb's monitoring.
RocksDBStoreMetricsManager.init(meter, attributesBuilderSupplier, this);
}
+
+ public CommitLogDispatcherBuildRocksdbConsumeQueue getDispatcherBuildRocksdbConsumeQueue() {
+ return dispatcherBuildRocksdbConsumeQueue;
+ }
+
+ class CommitLogDispatcherBuildRocksdbConsumeQueue implements CommitLogDispatcher {
+ @Override
+ public void dispatch(DispatchRequest request) throws RocksDBException {
+ putMessagePositionInfo(request);
+ }
+ }
+
+ public void loadAndStartConsumerServiceOnly() {
+ try {
+ this.dispatcherBuildRocksdbConsumeQueue = new CommitLogDispatcherBuildRocksdbConsumeQueue();
+ boolean loadResult = this.consumeQueueStore.load();
+ if (!loadResult) {
+ throw new RuntimeException("load consume queue failed");
+ }
+ super.loadCheckPoint();
+ this.consumeQueueStore.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
Review Comment:
Should add log here.
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -458,6 +464,79 @@ private RemotingCommand updateAndGetGroupForbidden(ChannelHandlerContext ctx, Re
return response;
}
+ private RemotingCommand diffConsumeQueue(ChannelHandlerContext ctx, RemotingCommand request) {
+ final RemotingCommand response = RemotingCommand.createResponseCommand(null);
+ response.setCode(ResponseCode.SUCCESS);
+ response.setBody(JSON.toJSONBytes(ImmutableMap.of("diffResult", "diff success, very good!")));
+
+ DefaultMessageStore messageStore = (DefaultMessageStore) brokerController.getMessageStore();
+ RocksDBMessageStore rocksDBMessageStore = messageStore.getRocksDBMessageStore();
+
+ if (!messageStore.getMessageStoreConfig().isRocksdbCQWriteEnable()) {
+ response.setBody(JSON.toJSONBytes(ImmutableMap.of("diffResult", "RocksdbCQWriteEnable is false, diffConsumeQueue is invalid")));
+ return response;
+ }
+
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> cqTable = messageStore.getConsumeQueueTable();
+ Random random = new Random();
+ for (Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> topicToCqListEntry : cqTable.entrySet()) {
+ String topic = topicToCqListEntry.getKey();
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueIdToCqMap = topicToCqListEntry.getValue();
+ for (Map.Entry<Integer, ConsumeQueueInterface> queueIdToCqEntry : queueIdToCqMap.entrySet()) {
+ Integer queueId = queueIdToCqEntry.getKey();
+
+ ConsumeQueueInterface jsonCq = queueIdToCqEntry.getValue();
+ ConsumeQueueInterface kvCq = rocksDBMessageStore.getConsumeQueue(topic, queueId);
+
+ CqUnit kvCqEarliestUnit = kvCq.getEarliestUnit();
+ CqUnit kvCqLatestUnit = kvCq.getLatestUnit();
+ CqUnit jsonCqEarliestUnit = jsonCq.getEarliestUnit();
+ CqUnit jsonCqLatestUnit = jsonCq.getLatestUnit();
+ LOGGER.info("diffConsumeQueue topic:{}, queue:{}, kvCqEarliestUnit:{}, jsonCqEarliestUnit:{}, kvCqLatestUnit:{}, jsonCqLatestUnit:{}",
+ topic, queueId, kvCqEarliestUnit, jsonCqEarliestUnit, kvCqLatestUnit, jsonCqLatestUnit);
+
+ long jsonOffset = jsonCq.getMaxOffsetInQueue() - 1;
+ int sampleCount = 10;
+
+ Set<Long> sampledOffsets = new HashSet<>();
+
+ if (jsonOffset > 100) {
Review Comment:
if jsonCq.getMaxOffsetInQueue() here is 100, meantime jsonCq.getMinOffsetInQueue() is 100 too, you may not find 10 messages here.
##########
common/src/main/java/org/apache/rocketmq/common/config/AbstractRocksDBStorage.java:
##########
@@ -496,6 +496,7 @@ public void statRocksdb(Logger logger) {
logger.info("MemUsage. blockCache: {}, indexesAndFilterBlock: {}, memtable: {}, blocksPinnedByIterator: {}",
blockCacheMemUsage, indexesAndFilterBlockMemUsage, memTableMemUsage, blocksPinnedByIteratorMemUsage);
} catch (Exception ignored) {
+ throw new RuntimeException(ignored);
Review Comment:
Change ignored to e
##########
tools/src/main/java/org/apache/rocketmq/tools/command/queue/DiffConsumeQueueCommand.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.tools.command.queue;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.remoting.RPCHook;
+import org.apache.rocketmq.remoting.protocol.body.ClusterInfo;
+import org.apache.rocketmq.remoting.protocol.body.DiffConsumeQueueResponseBody;
+import org.apache.rocketmq.remoting.protocol.route.BrokerData;
+import org.apache.rocketmq.srvutil.ServerUtil;
+import org.apache.rocketmq.tools.admin.DefaultMQAdminExt;
+import org.apache.rocketmq.tools.command.SubCommand;
+
+public class DiffConsumeQueueCommand implements SubCommand {
+
+ public static void main(String[] args) {
+ DiffConsumeQueueCommand cmd = new DiffConsumeQueueCommand();
+
+ Options options = ServerUtil.buildCommandlineOptions(new Options());
+ String[] subargs = new String[] {"-c LetLetMe", "-n xxxx:9876"};
Review Comment:
Means DefaultCluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759649104
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -381,6 +378,14 @@ public boolean load() {
return result;
}
+ public void loadCheckPoint() throws IOException {
Review Comment:
是这样的,需要按照issuse里边的配置来进行原地升级,如果直接改storeType,旧的消息将消费不到了,cq升级有俩种方案,一种是broker启动的时候做转换,一种是双写一段时间然后切流,我们选择了后者,这里的原因在issue里边有写,为了稳定性和不阻塞broker启动考虑的
Yes, it is necessary to perform an in-place upgrade according to the configuration in the issue. If you directly change the storeType, old messages will not be consumable. There are two solutions for upgrading the consume queue (CQ): one is to perform the conversion when the broker starts, and the other is to double-write for a period of time and then switch the flow. We chose the latter, as explained in the issue, for reasons of stability and to avoid blocking the broker startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759647553
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueueStore.java:
##########
@@ -164,12 +166,12 @@ private boolean shutdownInner() {
@Override
public void putMessagePositionInfoWrapper(DispatchRequest request) throws RocksDBException {
- if (request == null || this.bufferDRList.size() >= BATCH_SIZE) {
- putMessagePosition();
- }
if (request != null) {
this.bufferDRList.add(request);
}
+ if (request == null || !enableBatchWriteKvCq || this.bufferDRList.size() >= BATCH_SIZE) {
Review Comment:
其实是的,刚发现这个问题时我有点不太确定,为了防止对原有逻辑产生影响,这里加了一个开关。我先在预发测一下看看,没问题的话这里的batch可以直接去掉
Actually, yes, when I first discovered this issue, I was a bit unsure. To prevent any impact on the original logic, I added a switch here. I'll test it in the pre-release environment first, and if there are no issues, the batch here can be removed directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737993975
##########
broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java:
##########
@@ -637,6 +637,26 @@ public String encode() {
return encode(false);
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ TopicConfigSerializeWrapper topicConfigSerializeWrapper =
+ TopicConfigSerializeWrapper.fromJson(jsonString, TopicConfigSerializeWrapper.class);
+ if (topicConfigSerializeWrapper != null) {
+ this.dataVersion.assignNewOne(topicConfigSerializeWrapper.getDataVersion());
+ }
+ log.info("load topic metadata dataVersion success" + fileName + " " + topicConfigSerializeWrapper.getDataVersion());
Review Comment:
log should use template interpolation instead of direct string concatenation. aka, log.info("bla...{}", value);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737989136
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -3305,4 +3384,23 @@ private boolean validateBlackListConfigExist(Properties properties) {
}
return false;
}
+
+ private boolean checkCqUnitEqual(CqUnit cqUnit1, CqUnit cqUnit2) {
Review Comment:
Why not use idiomatic equals override approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1740109195
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueue.java:
##########
@@ -311,7 +311,7 @@ public CqUnit getEarliestUnit() {
public CqUnit getLatestUnit() {
try {
long maxOffset = this.messageStore.getQueueStore().getMaxOffsetInQueue(topic, queueId);
- return get(maxOffset);
+ return get(maxOffset > 0 ? maxOffset - 1 : maxOffset);
Review Comment:
The reason should be explained
##########
broker/src/main/java/org/apache/rocketmq/broker/subscription/SubscriptionGroupManager.java:
##########
@@ -334,6 +334,26 @@ public DataVersion getDataVersion() {
return dataVersion;
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ SubscriptionGroupManager obj = RemotingSerializable.fromJson(jsonString, SubscriptionGroupManager.class);
+ if (obj != null) {
+ this.dataVersion.assignNewOne(obj.dataVersion);
+ this.printLoadDataWhenFirstBoot(obj);
+ }
+ log.info("load subGroup dataVersion success " + fileName + " " + obj.dataVersion);
Review Comment:
should use {} in logger content
##########
tools/src/main/java/org/apache/rocketmq/tools/command/export/ExportMetadataInRocksDBCommand.java:
##########
@@ -38,6 +41,17 @@ public class ExportMetadataInRocksDBCommand implements SubCommand {
private static final String TOPICS_JSON_CONFIG = "topics";
private static final String SUBSCRIPTION_GROUP_JSON_CONFIG = "subscriptionGroups";
+ public static void main(String[] args) throws SubCommandException {
+ ExportMetadataInRocksDBCommand cmd = new ExportMetadataInRocksDBCommand();
+
+ Options options = ServerUtil.buildCommandlineOptions(new Options());
+ String[] subargs = new String[] {"-p /Users/huyitao/store/config", "-t subscriptionGroups", "-j true"};
Review Comment:
should use "user.home"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe closed pull request #8600: [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB
URL: https://github.com/apache/rocketmq/pull/8600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758511392
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -3305,4 +3384,23 @@ private boolean validateBlackListConfigExist(Properties properties) {
}
return false;
}
+
+ private boolean checkCqUnitEqual(CqUnit cqUnit1, CqUnit cqUnit2) {
Review Comment:
降低什么影响, 有什么风险, 说说清楚
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1751295391
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -3305,4 +3384,23 @@ private boolean validateBlackListConfigExist(Properties properties) {
}
return false;
}
+
+ private boolean checkCqUnitEqual(CqUnit cqUnit1, CqUnit cqUnit2) {
Review Comment:
尽量减少影响,降低风险,重写CqUnit的equea影响比较大
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1741674389
##########
tools/src/main/java/org/apache/rocketmq/tools/command/queue/DiffConsumeQueueCommand.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.tools.command.queue;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.remoting.RPCHook;
+import org.apache.rocketmq.remoting.protocol.body.ClusterInfo;
+import org.apache.rocketmq.remoting.protocol.body.DiffConsumeQueueResponseBody;
+import org.apache.rocketmq.remoting.protocol.route.BrokerData;
+import org.apache.rocketmq.srvutil.ServerUtil;
+import org.apache.rocketmq.tools.admin.DefaultMQAdminExt;
+import org.apache.rocketmq.tools.command.SubCommand;
+
+public class DiffConsumeQueueCommand implements SubCommand {
Review Comment:
我们把这个特性写在开源版里边,因为开源版也需要这个功能,目前开源版式没法json原地升级成rocksdb版的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1741668627
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -163,11 +163,13 @@ public class DefaultMessageStore implements MessageStore {
private volatile boolean shutdown = true;
protected boolean notifyMessageArriveInBatch = false;
- private StoreCheckpoint storeCheckpoint;
+ protected StoreCheckpoint storeCheckpoint;
private TimerMessageStore timerMessageStore;
private final LinkedList<CommitLogDispatcher> dispatcherList;
+ private RocksDBMessageStore rocksDBMessageStore;
+
Review Comment:
把双写理解为DefaultMessageStore的一个小功能,而不是一个独立的MessageStore,这样更简单一点
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2349001534
> > Moving offset management to RocksDB is very good to have. but decisions and rational should be explained.
>
> 我们的rocksdb方案 offset已经是rocksdb存储的了,不是这里引入的
It's about introducing RocksDB, it's about decisions how to migrate from file-based one to the KV counterpart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2350892352
> Got another issue, if rocksdbCQDoubleWriteEnable is on, meaning users still have chance to rollback to file-based CQ and offset management, offsets should also write in dual manner.
>
> This PR currently only write to KV.
#8699
#8698
这俩个额外的issue会被用来关联这些问题,我会在接下来光速修复掉
These two additional issues will be used to associate these problems, and I will fix them at lightning speed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "fuyou001 (via GitHub)" <gi...@apache.org>.
fuyou001 commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1740778775
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java:
##########
@@ -373,6 +374,25 @@ public void setDataVersion(DataVersion dataVersion) {
this.dataVersion = dataVersion;
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ ConsumerOffsetManager obj = RemotingSerializable.fromJson(jsonString, ConsumerOffsetManager.class);
+ if (obj != null) {
+ this.dataVersion = obj.dataVersion;
Review Comment:
when dataVersion update? the dataVersion only changes during the HA switch
##########
broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java:
##########
@@ -789,6 +788,9 @@ public boolean initializeMessageStore() {
defaultMessageStore = new RocksDBMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig, topicConfigManager.getTopicConfigTable());
} else {
defaultMessageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig, topicConfigManager.getTopicConfigTable());
+ if (messageStoreConfig.isRocksdbCQWriteEnable()) {
Review Comment:
must be isEnableRocksDBStore is true ,This feature can only be enabled when isEnableRocksDBStore is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758517916
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java:
##########
@@ -373,6 +374,25 @@ public void setDataVersion(DataVersion dataVersion) {
this.dataVersion = dataVersion;
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ ConsumerOffsetManager obj = RemotingSerializable.fromJson(jsonString, ConsumerOffsetManager.class);
+ if (obj != null) {
+ this.dataVersion = obj.dataVersion;
+ }
+ LOG.info("load consumer offset dataVersion success, " + fileName + " " + jsonString);
Review Comment:
Log with formatter, not string concatenation: See https://medium.com/@mohanakrishnakavali/logging-efficiency-in-java-string-concatenation-vs-string-format-efa02055ddae
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758832686
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueue.java:
##########
@@ -311,7 +310,7 @@ public CqUnit getEarliestUnit() {
public CqUnit getLatestUnit() {
try {
long maxOffset = this.messageStore.getQueueStore().getMaxOffsetInQueue(topic, queueId);
- return get(maxOffset);
+ return get(maxOffset > 0 ? maxOffset - 1 : maxOffset);
Review Comment:
Need a comment, explaining why maxOffset -1 makes sense here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2350797677
> > > Moving offset management to RocksDB is very good to have. but decisions and rational should be explained.
> >
> >
> > 我们的rocksdb方案 offset已经是rocksdb存储的了,不是这里引入的
>
> It's NOT about introducing RocksDB, it's about decisions how to migrate from file-based one to the KV counterpart.
和元数据(topic,group)一样,在启动的时候会校验DataVersion,然后Transfer一次
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737992306
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -458,6 +464,79 @@ private RemotingCommand updateAndGetGroupForbidden(ChannelHandlerContext ctx, Re
return response;
}
+ private RemotingCommand diffConsumeQueue(ChannelHandlerContext ctx, RemotingCommand request) {
Review Comment:
What is the rational maintaining two offset managers at the same time and offering diff feature here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1741637402
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java:
##########
@@ -458,6 +464,79 @@ private RemotingCommand updateAndGetGroupForbidden(ChannelHandlerContext ctx, Re
return response;
}
+ private RemotingCommand diffConsumeQueue(ChannelHandlerContext ctx, RemotingCommand request) {
Review Comment:
查看双写进度,决定什么时候切流到rocksdbcq只读
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2316938493
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/8600?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `23.95437%` with `200 lines` in your changes missing coverage. Please review.
> Project coverage is 47.19%. Comparing base [(`a1ea1eb`)](https://app.codecov.io/gh/apache/rocketmq/commit/a1ea1eb1f2b8187166b1b0e1f646e6476f27dda5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`5b255dc`)](https://app.codecov.io/gh/apache/rocketmq/commit/5b255dc3afd8dddd9af3546cca631e04787a604e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 2 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/8600?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ocketmq/broker/processor/AdminBrokerProcessor.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fprocessor%2FAdminBrokerProcessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0FkbWluQnJva2VyUHJvY2Vzc29yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...q/tools/command/queue/DiffConsumeQueueCommand.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=tools%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Ftools%2Fcommand%2Fqueue%2FDiffConsumeQueueCommand.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-dG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Rvb2xzL2NvbW1hbmQvcXVldWUvRGlmZkNvbnN1bWVRdWV1ZUNvbW1hbmQuamF2YQ==) | 0.00% | [39 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...mq/broker/offset/RocksDBConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Foffset%2FRocksDBConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L1JvY2tzREJDb25zdW1lck9mZnNldE1hbmFnZXIuamF2YQ==) | 54.54% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/rocketmq/store/RocksDBMessageStore.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2FRocksDBMessageStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL1JvY2tzREJNZXNzYWdlU3RvcmUuamF2YQ==) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/rocketmq/store/DefaultMessageStore.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2FDefaultMessageStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0RlZmF1bHRNZXNzYWdlU3RvcmUuamF2YQ==) | 33.33% | [11 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...command/export/ExportMetadataInRocksDBCommand.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=tools%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Ftools%2Fcommand%2Fexport%2FExportMetadataInRocksDBCommand.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-dG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3Rvb2xzL2NvbW1hbmQvZXhwb3J0L0V4cG9ydE1ldGFkYXRhSW5Sb2Nrc0RCQ29tbWFuZC5qYXZh) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/rocketmq/client/impl/MQClientAPIImpl.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fclient%2Fimpl%2FMQClientAPIImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9NUUNsaWVudEFQSUltcGwuamF2YQ==) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../rocketmq/broker/offset/ConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Foffset%2FConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0NvbnN1bWVyT2Zmc2V0TWFuYWdlci5qYXZh) | 58.33% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../subscription/RocksDBSubscriptionGroupManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fsubscription%2FRocksDBSubscriptionGroupManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc3Vic2NyaXB0aW9uL1JvY2tzREJTdWJzY3JpcHRpb25Hcm91cE1hbmFnZXIuamF2YQ==) | 58.33% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../broker/subscription/SubscriptionGroupManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fsubscription%2FSubscriptionGroupManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc3Vic2NyaXB0aW9uL1N1YnNjcmlwdGlvbkdyb3VwTWFuYWdlci5qYXZh) | 61.53% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [9 more](https://app.codecov.io/gh/apache/rocketmq/pull/8600?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #8600 +/- ##
=============================================
- Coverage 47.27% 47.19% -0.08%
- Complexity 11467 11479 +12
=============================================
Files 1277 1279 +2
Lines 89342 89687 +345
Branches 11480 11553 +73
=============================================
+ Hits 42234 42331 +97
- Misses 41918 42127 +209
- Partials 5190 5229 +39
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/8600?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1735751626
##########
broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java:
##########
@@ -815,6 +818,8 @@ public boolean initializeMessageStore() {
} catch (IOException e) {
result = false;
LOG.error("BrokerController#initialize: unexpected error occurs", e);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
Review Comment:
Should follow the original way, return false here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1741677067
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java:
##########
@@ -373,6 +374,25 @@ public void setDataVersion(DataVersion dataVersion) {
this.dataVersion = dataVersion;
}
+ public boolean loadDataVersion() {
+ String fileName = null;
+ try {
+ fileName = this.configFilePath();
+ String jsonString = MixAll.file2String(fileName);
+ if (jsonString != null) {
+ ConsumerOffsetManager obj = RemotingSerializable.fromJson(jsonString, ConsumerOffsetManager.class);
+ if (obj != null) {
+ this.dataVersion = obj.dataVersion;
Review Comment:
没有的,offset的刷新有个步长,满足步长也会刷新的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759649805
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -381,6 +378,14 @@ public boolean load() {
return result;
}
+ public void loadCheckPoint() throws IOException {
Review Comment:
Then it's necessary to add a check and abort logic to guard against this incident from happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2348996890
As previously mentioned, overall it is good to have PR, but there are many issues to resolve yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759647553
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueueStore.java:
##########
@@ -164,12 +166,12 @@ private boolean shutdownInner() {
@Override
public void putMessagePositionInfoWrapper(DispatchRequest request) throws RocksDBException {
- if (request == null || this.bufferDRList.size() >= BATCH_SIZE) {
- putMessagePosition();
- }
if (request != null) {
this.bufferDRList.add(request);
}
+ if (request == null || !enableBatchWriteKvCq || this.bufferDRList.size() >= BATCH_SIZE) {
Review Comment:
其实是的,刚发现这个问题时我有点不太确定,为了防止对原有逻辑产生影响,这里加了一个开关。我先在预发测一下看看,没问题的话这里的batch可以直接去掉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759647088
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueue.java:
##########
@@ -311,7 +310,7 @@ public CqUnit getEarliestUnit() {
public CqUnit getLatestUnit() {
try {
long maxOffset = this.messageStore.getQueueStore().getMaxOffsetInQueue(topic, queueId);
- return get(maxOffset);
+ return get(maxOffset > 0 ? maxOffset - 1 : maxOffset);
Review Comment:
这里应该是之前写错了,不-1拿不到最新的cqUnit,这块和别的ConsumeQueue实现保持一致了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758839628
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueueStore.java:
##########
@@ -164,12 +166,12 @@ private boolean shutdownInner() {
@Override
public void putMessagePositionInfoWrapper(DispatchRequest request) throws RocksDBException {
- if (request == null || this.bufferDRList.size() >= BATCH_SIZE) {
- putMessagePosition();
- }
if (request != null) {
this.bufferDRList.add(request);
}
+ if (request == null || !enableBatchWriteKvCq || this.bufferDRList.size() >= BATCH_SIZE) {
Review Comment:
Actually, batching here does NOT improve performance at all. Building consume queue in RocksDB has WAL disabled with atomic flush feature on. Put KV into RocksDB actually means insert into MemTable without incurring any IO. Given that all ops are memory-only, batching here is not necessary at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1759650333
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/RocksDBConsumerOffsetManager.java:
##########
@@ -16,36 +16,79 @@
*/
package org.apache.rocketmq.broker.offset;
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.serializer.SerializerFeature;
import java.io.File;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentMap;
-
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.RocksDBConfigManager;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.utils.DataConverter;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.remoting.protocol.DataVersion;
import org.rocksdb.WriteBatch;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.serializer.SerializerFeature;
-
public class RocksDBConsumerOffsetManager extends ConsumerOffsetManager {
+ protected static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+
protected RocksDBConfigManager rocksDBConfigManager;
public RocksDBConsumerOffsetManager(BrokerController brokerController) {
super(brokerController);
- this.rocksDBConfigManager = new RocksDBConfigManager(configFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
+ this.rocksDBConfigManager = new RocksDBConfigManager(rocksdbConfigFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
}
@Override
public boolean load() {
if (!rocksDBConfigManager.init()) {
return false;
}
- return this.rocksDBConfigManager.loadData(this::decodeOffset);
+ if (!loadDataVersion() || !loadConsumerOffset()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ public boolean loadConsumerOffset() {
+ return this.rocksDBConfigManager.loadData(this::decodeOffset) && merge();
+ }
+
+ private boolean merge() {
+ if (!brokerController.getMessageStoreConfig().isTransferOffsetJsonToRocksdb()) {
+ log.info("the switch transferOffsetJsonToRocksdb is off, no merge offset operation is needed.");
+ return true;
+ }
+ if (!UtilAll.isPathExists(this.configFilePath()) && !UtilAll.isPathExists(this.configFilePath() + ".bak")) {
+ log.info("consumerOffset json file does not exist, so skip merge");
+ return true;
+ }
+ if (!super.loadDataVersion()) {
+ log.error("load json consumerOffset dataVersion error, startup will exit");
+ return false;
+ }
+
+ final DataVersion dataVersion = super.getDataVersion();
+ final DataVersion kvDataVersion = this.getDataVersion();
+ if (dataVersion.getCounter().get() > kvDataVersion.getCounter().get()) {
+ if (!super.load()) {
+ log.error("load json consumerOffset info failed, startup will exit");
+ return false;
+ }
+ this.persist();
+ this.getDataVersion().assignNewOne(dataVersion);
+ updateDataVersion();
Review Comment:
I said not mandatory, but keeping data and version consistent is really good to have, especially when you are trouble shooting bugs online.
IMO, it will be a minor change to achieve this goal. Why not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737989124
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/RocksDBConsumerOffsetManager.java:
##########
@@ -23,29 +23,74 @@
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.RocksDBConfigManager;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.utils.DataConverter;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.remoting.protocol.DataVersion;
import org.rocksdb.WriteBatch;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.serializer.SerializerFeature;
public class RocksDBConsumerOffsetManager extends ConsumerOffsetManager {
+ protected static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+
protected RocksDBConfigManager rocksDBConfigManager;
public RocksDBConsumerOffsetManager(BrokerController brokerController) {
super(brokerController);
- this.rocksDBConfigManager = new RocksDBConfigManager(configFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
+ this.rocksDBConfigManager = new RocksDBConfigManager(rocksdbConfigFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
}
@Override
public boolean load() {
if (!rocksDBConfigManager.init()) {
return false;
}
- return this.rocksDBConfigManager.loadData(this::decodeOffset);
+ if (!loadDataVersion() || !loadConsumerOffset()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ public boolean loadConsumerOffset() {
+ return this.rocksDBConfigManager.loadData(this::decodeOffset) && merge();
+ }
+
+ private boolean merge() {
+ if (!brokerController.getMessageStoreConfig().isTransferOffsetJsonToRocksdb()) {
+ log.info("the switch transferOffsetJsonToRocksdb is off, no merge offset operation is needed.");
+ return true;
+ }
+ if (!UtilAll.isPathExists(this.configFilePath()) && !UtilAll.isPathExists(this.configFilePath() + ".bak")) {
+ log.info("consumerOffset json file is not exist, so skip merge");
+ return true;
+ }
+ if (!super.loadDataVersion()) {
+ log.error("load json consumerOffset dataVersion error, startup will exit");
+ return false;
+ }
+
+ final DataVersion dataVersion = super.getDataVersion();
+ final DataVersion kvDataVersion = this.getDataVersion();
+ if (dataVersion.getCounter().get() > kvDataVersion.getCounter().get()) {
+ if (!super.load()) {
+ log.error("load json consumerOffset info failed, startup will exit");
+ return false;
+ }
+ this.persist();
+ this.getDataVersion().assignNewOne(dataVersion);
+ updateDataVersion();
Review Comment:
Add log here to clarify where you have loaded the final consumerOffset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #8589] Support file format CQ and json format offset in-place upgrade to rocksdb management [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins merged PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1741676200
##########
broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java:
##########
@@ -789,6 +788,9 @@ public boolean initializeMessageStore() {
defaultMessageStore = new RocksDBMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig, topicConfigManager.getTopicConfigTable());
} else {
defaultMessageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig, topicConfigManager.getTopicConfigTable());
+ if (messageStoreConfig.isRocksdbCQWriteEnable()) {
Review Comment:
是的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2325954397
> Moving offset management to RocksDB is very good to have. but decisions and rational should be explained.
我们rocksdb方案中,topic,group,offset,cq都是存储在rocksdb中,前三者我们认为都是元数据
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758834952
##########
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueueStore.java:
##########
@@ -164,12 +166,12 @@ private boolean shutdownInner() {
@Override
public void putMessagePositionInfoWrapper(DispatchRequest request) throws RocksDBException {
- if (request == null || this.bufferDRList.size() >= BATCH_SIZE) {
- putMessagePosition();
- }
if (request != null) {
this.bufferDRList.add(request);
}
+ if (request == null || !enableBatchWriteKvCq || this.bufferDRList.size() >= BATCH_SIZE) {
Review Comment:
Batching without timeout is an obvious bug. Instead of add a configuration to walk around this bug, please fix it directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2348994885
There is another issue to address: file backed ConsumeQueue now unconditionally preserve the last file in the file queue. If configured to RocksDB only, need clean them up once the file-based CQ segments are totally expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1751642576
##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -424,6 +424,37 @@ public class MessageStoreConfig {
private boolean putConsumeQueueDataByFileChannel = true;
+ private boolean transferOffsetJsonToRocksdb = false;
+
+ private boolean rocksdbCQDoubleWriteEnable = false;
+
+ private boolean enableBatchWriteKvCq = false;
Review Comment:
这里和 cq type 配置项的关系是什么呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2348438712
> Moving offset management to RocksDB is very good to have. but decisions and rational should be explained.
我们的rocksdb方案 offset已经是rocksdb存储的了,不是这里引入的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758851660
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -381,6 +378,14 @@ public boolean load() {
return result;
}
+ public void loadCheckPoint() throws IOException {
Review Comment:
This implies this PR won't build CQ records into RocksDB CQ store for existing messages in commit log if they are previously indexed by file-based CQ store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758855947
##########
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java:
##########
@@ -381,6 +378,14 @@ public boolean load() {
return result;
}
+ public void loadCheckPoint() throws IOException {
Review Comment:
Config will be error prune. Once config of existing classic nodes are changed inappropriately, things will be totally messed up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "LetLetMe (via GitHub)" <gi...@apache.org>.
LetLetMe commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758236042
##########
broker/src/test/java/org/apache/rocketmq/broker/offset/RocksdbTransferOffsetAndCqTest.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.broker.offset;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.commons.collections.MapUtils;
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.MixAll;
+import org.apache.rocketmq.common.Pair;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.RocksDBMessageStore;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.ConsumeQueueStoreInterface;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.rocksdb.RocksDBException;
+
+@RunWith(MockitoJUnitRunner.class)
+public class RocksdbTransferOffsetAndCqTest {
+
+ private final String basePath = Paths.get(System.getProperty("user.home"),
+ "unit-test-store", UUID.randomUUID().toString().substring(0, 16).toUpperCase()).toString();
+
+ private final String topic = "topic";
+ private final String group = "group";
+ private final String clientHost = "clientHost";
+ private final int queueId = 1;
+
+ private RocksDBConsumerOffsetManager rocksdbConsumerOffsetManager;
+
+ private ConsumerOffsetManager consumerOffsetManager;
+
+ private DefaultMessageStore defaultMessageStore;
+
+ @Mock
+ private BrokerController brokerController;
+
+ @Before
+ public void init() throws IOException {
+ if (notToBeExecuted()) {
+ return;
+ }
+ BrokerConfig brokerConfig = new BrokerConfig();
+ brokerConfig.setConsumerOffsetUpdateVersionStep(10);
+ MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+ messageStoreConfig.setStorePathRootDir(basePath);
+ messageStoreConfig.setTransferOffsetJsonToRocksdb(true);
+ messageStoreConfig.setRocksdbCQDoubleWriteEnable(true);
+ Mockito.lenient().when(brokerController.getBrokerConfig()).thenReturn(brokerConfig);
+ Mockito.lenient().when(brokerController.getMessageStoreConfig()).thenReturn(messageStoreConfig);
+
+ defaultMessageStore = new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("aaa", true), null,
+ brokerConfig, new ConcurrentHashMap<String, TopicConfig>());
+ defaultMessageStore.enableRocksdbCQWrite();
+ defaultMessageStore.loadCheckPoint();
+
+ consumerOffsetManager = new ConsumerOffsetManager(brokerController);
+ consumerOffsetManager.load();
+
+ rocksdbConsumerOffsetManager = new RocksDBConsumerOffsetManager(brokerController);
+ }
+
+ @Test
+ public void testTransferOffset() {
+ if (notToBeExecuted()) {
+ return;
+ }
+
+ for (int i = 0; i < 200; i++) {
+ consumerOffsetManager.commitOffset(clientHost, group, topic, queueId, i);
+ }
+
+ ConcurrentMap<String, ConcurrentMap<Integer, Long>> offsetTable = consumerOffsetManager.getOffsetTable();
+ ConcurrentMap<Integer, Long> map = offsetTable.get(topic + "@" + group);
+ Assert.assertTrue(MapUtils.isNotEmpty(map));
+
+ Long offset = map.get(queueId);
+ Assert.assertEquals(199L, (long) offset);
+
+ long offsetDataVersion = consumerOffsetManager.getDataVersion().getCounter().get();
+ Assert.assertEquals(20L, offsetDataVersion);
+
+ consumerOffsetManager.persist();
+
+ boolean loadResult = rocksdbConsumerOffsetManager.load();
+ Assert.assertTrue(loadResult);
+
+ ConcurrentMap<String, ConcurrentMap<Integer, Long>> rocksdbOffsetTable = rocksdbConsumerOffsetManager.getOffsetTable();
+
+ ConcurrentMap<Integer, Long> rocksdbMap = rocksdbOffsetTable.get(topic + "@" + group);
+ Assert.assertTrue(MapUtils.isNotEmpty(rocksdbMap));
+
+ Long aLong1 = rocksdbMap.get(queueId);
+ Assert.assertEquals(199L, (long) aLong1);
+
+ long rocksdbOffset = rocksdbConsumerOffsetManager.getDataVersion().getCounter().get();
+ Assert.assertEquals(21L, rocksdbOffset);
+ }
+
+ @Test
+ public void testRocksdbCqWrite() throws RocksDBException {
+ if (notToBeExecuted()) {
+ return;
+ }
+ RocksDBMessageStore kvStore = defaultMessageStore.getRocksDBMessageStore();
+ ConsumeQueueStoreInterface store = kvStore.getConsumeQueueStore();
+ ConsumeQueueInterface rocksdbCq = defaultMessageStore.getRocksDBMessageStore().findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface fileCq = defaultMessageStore.findConsumeQueue(topic, queueId);
+ for (int i = 0; i < 200; i++) {
+ DispatchRequest request = new DispatchRequest(topic, queueId, i, 200, 0, System.currentTimeMillis(), i, "", "", 0, 0, new HashMap<>());
+ fileCq.putMessagePositionInfoWrapper(request);
+ store.putMessagePositionInfoWrapper(request);
+ }
+ Pair<CqUnit, Long> unit = rocksdbCq.getCqUnitAndStoreTime(100);
+ Pair<CqUnit, Long> unit1 = fileCq.getCqUnitAndStoreTime(100);
+ Assert.assertTrue(unit.getObject1().getPos() == unit1.getObject1().getPos());
+ }
+
+ private boolean notToBeExecuted() {
+ return MixAll.isMac();
Review Comment:
rocksdb相关的测试在mac都跳过了,github提供的mac环境有问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#issuecomment-2350799954
Got another issue, if rocksdbCQDoubleWriteEnable is on, meaning users still have chance to rollback to file-based CQ and offset management, offsets should also write in dual manner.
This PR currently only write to KV.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1758884572
##########
broker/src/main/java/org/apache/rocketmq/broker/offset/RocksDBConsumerOffsetManager.java:
##########
@@ -16,36 +16,79 @@
*/
package org.apache.rocketmq.broker.offset;
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.serializer.SerializerFeature;
import java.io.File;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentMap;
-
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.RocksDBConfigManager;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.utils.DataConverter;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.remoting.protocol.DataVersion;
import org.rocksdb.WriteBatch;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.serializer.SerializerFeature;
-
public class RocksDBConsumerOffsetManager extends ConsumerOffsetManager {
+ protected static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+
protected RocksDBConfigManager rocksDBConfigManager;
public RocksDBConsumerOffsetManager(BrokerController brokerController) {
super(brokerController);
- this.rocksDBConfigManager = new RocksDBConfigManager(configFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
+ this.rocksDBConfigManager = new RocksDBConfigManager(rocksdbConfigFilePath(), brokerController.getMessageStoreConfig().getMemTableFlushIntervalMs());
}
@Override
public boolean load() {
if (!rocksDBConfigManager.init()) {
return false;
}
- return this.rocksDBConfigManager.loadData(this::decodeOffset);
+ if (!loadDataVersion() || !loadConsumerOffset()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ public boolean loadConsumerOffset() {
+ return this.rocksDBConfigManager.loadData(this::decodeOffset) && merge();
+ }
+
+ private boolean merge() {
+ if (!brokerController.getMessageStoreConfig().isTransferOffsetJsonToRocksdb()) {
+ log.info("the switch transferOffsetJsonToRocksdb is off, no merge offset operation is needed.");
+ return true;
+ }
+ if (!UtilAll.isPathExists(this.configFilePath()) && !UtilAll.isPathExists(this.configFilePath() + ".bak")) {
+ log.info("consumerOffset json file does not exist, so skip merge");
+ return true;
+ }
+ if (!super.loadDataVersion()) {
+ log.error("load json consumerOffset dataVersion error, startup will exit");
+ return false;
+ }
+
+ final DataVersion dataVersion = super.getDataVersion();
+ final DataVersion kvDataVersion = this.getDataVersion();
+ if (dataVersion.getCounter().get() > kvDataVersion.getCounter().get()) {
+ if (!super.load()) {
+ log.error("load json consumerOffset info failed, startup will exit");
+ return false;
+ }
+ this.persist();
+ this.getDataVersion().assignNewOne(dataVersion);
+ updateDataVersion();
Review Comment:
persistOffset and updateDataVersion should be in the same writeBatch, ensuring change is atomic. It looks not mandatory now, but it is idiomatic to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Enhancement] Support JSON CQ Files and offset In-place Upgrade to RocksDB [rocketmq]
Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #8600:
URL: https://github.com/apache/rocketmq/pull/8600#discussion_r1737992796
##########
broker/src/main/java/org/apache/rocketmq/broker/subscription/RocksDBSubscriptionGroupManager.java:
##########
@@ -79,28 +79,30 @@ public boolean loadForbidden(BiConsumer<byte[], byte[]> biConsumer) {
private boolean merge() {
if (!brokerController.getMessageStoreConfig().isTransferMetadataJsonToRocksdb()) {
- log.info("The switch is off, no merge operation is needed.");
+ log.info("the switch transferMetadataJsonToRocksdb is off, no merge subGroup operation is needed.");
return true;
}
if (!UtilAll.isPathExists(this.configFilePath()) && !UtilAll.isPathExists(this.configFilePath() + ".bak")) {
- log.info("json file and json back file not exist, so skip merge");
+ log.info("subGroup json file is not exist, so skip merge");
Review Comment:
is not exist --> does not exist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org