You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by ji...@apache.org on 2022/02/08 02:43:43 UTC
[rocketmq] branch 5.0.0-alpha updated (5254805 -> 0b4adb4)
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a change to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git.
omit 5254805 [Assignment] Fix the risk of memory overflow caused by excessive popShareQueueNum.
omit 1b09702 change the level to warn when the ack message failed (#3756)
omit 03526d8 [ISSUE #3498] Make messages in reviveTopic more evenly written to different queues #3499
omit 7be401e [ISSUE #3708] add CorrectLogicOffsetService to periodically correct min logic offset (#3722)
omit 0d62194 [ISSUE #3708] Refactor CQ and BCQ loading process and Fix some unit-tests issue. (#3713)
omit e923a7e [ISSUE #3708] Both CQ and BCQ need to be supported in DefaultMessageStore. (#3712)
omit 48a2beb fix check style (#3703)
omit 16128db Fix check style
omit ba53f93 Fix test for consumer offset
omit add737f Convert the consumer offset too
omit d5701ae [ISSUE #3679] Support topic attributes (#3698)
omit 31deb47 [acl] Modify unit test to solve the problem of lack of license
omit ca206b3 [benchmark] Add clientRebalanceEnable
omit d559016 [tools] Fix parameter conflicts
omit 6928c5d [RIP-19] Pop Consuming (tools)
omit f7c63fe Merge remote-tracking branch 'origin/5.0.0-alpha' into 5.0.0-alpha-static-topic
omit 16a3195 Merge pull request #3671 from Erik1288/5.0.0-alpha-bcq
omit 1afe169 support batch consume-queue.
omit 7f7cf1b Add test for topicStats and consumeStats
omit 61fcf43 Finish the topic stats and consume stats in admin client
omit 47622e6 Polish doc, add todo list
omit 3a14786 Fix checkstyle
omit d4a656c Fix unit test
omit 1c69b64 Fix check style
omit ba2c768 Fix check stype
omit 6a3eac9 Add the scope concept to logic queue
omit 5ebc327 Polish doc
omit f51143f Polish doc
omit 8aea6bf Polish doc, introduce geo-recovery cluster
omit 1a204a1 Polish doc, scope of static topic
omit a165b4a Add the scope to Static topic
omit ad90cc1 Finish the slave sync logic for topic queue mapping
omit a9addc3 Add tests for command
omit 48db31b Finish the test for topicStats
omit 8b747f9 Try using the new style to handble get min offset
omit 1d7807b Finish the test for topic queue mapping clean serice
omit ad993e3 Use 3 brokers for IT test
omit 1ca218f Add TopicQueueMappingCleanService
omit c06564f Try polishing the clear logic, need more polishment
omit 9cae8c1 Add decode encode test for topic queue manager
omit 12915b8 Add clean item logic for topic queue mapping
omit 527382e Clean the items more than second gen
omit d677d85 Polish doc
omit aa505d2 Polish doc
omit 3a621e6 Add code reading guide to doc
omit a71ee3b Add test for logicOffset = -1
omit 4f96f72 Fix the max offset logic for logicOffset = -1
omit 2dd6594 Refactor the admin code, reduce exposing apis
omit 5b77793 Polish the resetZero logic for logic queue
omit 74565fd Fix the computePullFromWhereWithException
omit fad17e1 Finish the logic for double-read-check
omit ae7b675 Finish the logic for RETRY message of logic queue
omit 085f239 Polish newline for doc
omit e002c74 Polish doc for static topic
omit e952dd6 Finish the produce and consume test for remapped static topic
omit f7f32e7 Add the test for remapping static topic
omit c2c56ea Fix the command name for remapping
omit fe8b7de Delete the unused code from old implementation
omit 6f2b120 Finish the basic create,produce,consume of the static topic
omit ed2d626 Add a basic produce test for static topic
omit c1d4b8b Polish the concept for static topic doc
omit cc40d34 Polish the doc for static topic
omit 0c4ef24 Add the cn doc for Static Topic Logic Queue
omit 2ee5897 Fix the stability of remapping
omit 5ce093e Add test for remapping static topic
omit a172c76 Finish the test for utils of createStaticTopic
omit 1dacb66 Test utils for static topic
omit 1074ef7 Polish the variable name in StaticTopicIT
omit cb7e032 Finish the test for createStaticTopic
omit a493ca4 Add the allocator test
omit 8962581 Fix serialize problem
omit a8ef92e Fix the serialize probelm
omit 6c64028 Polish the json problem
omit fa39815 Catch the exception
omit e93536a Add IT Test for static topic
omit 2f4bf29 Polish the code structure for static topic command
omit 3d68590 Polish the code
omit daf4749 Polish the use of route data
omit 4018435 Polish the static topic command
omit cf8b846 Polish the lock and persist for updateTopicQueueMapping
omit d18d787 Add the check logic for admin process update-and-create static topic
omit 4e9b097 Check the correctness of logic items
omit f05cfb9 Polish the static topic commands
omit f225fd0 Polish the code for command
omit 165e133 Enable to run from file for createStaticTopic command
omit 82364fc Polish the tools
omit ffcc548 Refactor the code package for static topic
omit cabc383 Finish the remapping command
omit e9cafe9 Polish the remapping logic
omit 25a588b Init the remapping command
omit 45946d4 Ignore the existed queueId in static topi creation
omit c673cb7 Use timestamp as the epoch to prevent some unknown problem
omit a5af2cf Finish the UpdateStaticTopicSubCommand
omit 295a6bd Polish the update utils
omit f12eceb Add the UpdateStaticTopicSubCommand
omit 0c6ee5c Add epoch, dirty to the topic mapping detail
omit 215c0e4 Add the adminExt
omit 86c9f10 Fix code to RpcRequest
omit 4070da8 Finish typo
omit 6381e6e Rename the rpc header
omit 86b3ff7 Finish the processor
omit da12c9e Correct the invoke for PullMessageProcessor
omit 077cf09 Add request header builder
omit 912613d Abstract the rpc layer
omit fdbc7ff Polish typo
omit 5274294 Finish the rewrite logic for AdminBrokerProcessor
omit 507553b Finish the rewrite logic for SEARCH_OFFSET_BY_TIMESTAMP
omit 8d49c16 Polish the rpc usage for PullProcessor
omit 311d5f4 Try to abstract the rpc layer for BrokerOuterAPI
omit 32f58c7 Polish the topic queue mapping context, and process the conext for ConsumerManager
omit f5285b0 Polish and add some test
omit 5e04a27 Finish the rewrite topic for pull and send process
omit 36a82fa Polish the rewrite logic for SendProcessor
omit 71af9c7 Finish the logic for SendProcessor
omit 1f89733 Polish compile errrors
omit 6133887 Fix
omit 08618d5 Finish the findBrokerAddr for admin publish subscribe
omit f308cd3 Convert mq to broker name
omit c0ffc5b Finish the producer to get the real broker addr
omit 40d8626 Finish the topicRoute2endPoints for static topic
omit a64ad01 Add some notes, and revert the id map, set the globalId first
omit 7707275 Finish the register process in namesrv
omit 7bbc7dd Add the register logic for mapping topic
omit 3e2c920 Add update_static_topic code
omit e862ac8 Add definition for logic queue
new b6ff649 feat(all):new feature for static topic
new 821b91f support batch consume-queue.
new 6ba9344 [RIP-19] Pop Consuming (tools)
new a014330 [tools] Fix parameter conflicts
new c998f3f [benchmark] Add clientRebalanceEnable
new 8bb98c9 [acl] Modify unit test to solve the problem of lack of license
new 18010a4 [ISSUE #3679] Support topic attributes (#3698)
new b3f9fbd Convert the consumer offset too
new a767cc1 Fix test for consumer offset
new 372e42f Fix check style
new 5ce1b88 fix check style (#3703)
new e8cf133 [ISSUE #3708] Both CQ and BCQ need to be supported in DefaultMessageStore. (#3712)
new 6172ad3 [ISSUE #3708] Refactor CQ and BCQ loading process and Fix some unit-tests issue. (#3713)
new 78a3ed7 [ISSUE #3708] add CorrectLogicOffsetService to periodically correct min logic offset (#3722)
new fe88fb3 [ISSUE #3498] Make messages in reviveTopic more evenly written to different queues #3499
new 5d15c3e change the level to warn when the ack message failed (#3756)
new 0b4adb4 [Assignment] Fix the risk of memory overflow caused by excessive popShareQueueNum.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5254805)
\
N -- N -- N refs/heads/5.0.0-alpha (0b4adb4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 17 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
[rocketmq] 01/17: feat(all):new feature for static topic
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit b6ff649291453d3dc2d702fbba1548d991b610ac
Author: dongeforever <do...@apache.org>
AuthorDate: Sat Nov 6 11:54:21 2021 +0800
feat(all):new feature for static topic
---
.../apache/rocketmq/broker/BrokerController.java | 168 ++-
.../rocketmq/broker/BrokerPathConfigHelper.java | 4 +
.../client/rebalance/RebalanceLockManager.java | 9 +-
.../broker/domain/LogicalQueuesInfoInBroker.java | 116 --
.../apache/rocketmq/broker/out/BrokerOuterAPI.java | 57 +-
.../processor/AbstractSendMessageProcessor.java | 174 +--
.../broker/processor/AdminBrokerProcessor.java | 1013 ++++++-----------
.../broker/processor/ConsumerManageProcessor.java | 101 +-
.../broker/processor/PullMessageProcessor.java | 369 ++++--
.../broker/processor/SendMessageProcessor.java | 101 +-
.../rocketmq/broker/slave/SlaveSynchronize.java | 23 +-
.../rocketmq/broker/topic/TopicConfigManager.java | 115 +-
.../topic/TopicQueueMappingCleanService.java | 332 ++++++
.../broker/topic/TopicQueueMappingManager.java | 259 +++++
.../apache/rocketmq/broker/BrokerOuterAPITest.java | 2 +-
.../apache/rocketmq/broker/BrokerStartupTest.java | 20 +-
.../broker/processor/AdminBrokerProcessorTest.java | 298 +----
.../broker/processor/PullMessageProcessorTest.java | 111 +-
.../broker/processor/SendMessageProcessorTest.java | 73 +-
.../broker/topic/TopicConfigManagerTest.java | 138 ---
.../broker/topic/TopicQueueMappingManagerTest.java | 112 ++
.../client/consumer/DefaultMQPushConsumer.java | 2 +-
.../rocketmq/client/consumer/PullResult.java | 1 +
.../consumer/store/RemoteBrokerOffsetStore.java | 30 +-
.../client/exception/OffsetNotFoundException.java | 20 +-
.../apache/rocketmq/client/impl/MQAdminImpl.java | 192 +---
.../rocketmq/client/impl/MQClientAPIImpl.java | 239 +---
.../ConsumeMessageConcurrentlyService.java | 2 +-
.../impl/consumer/DefaultLitePullConsumerImpl.java | 42 +-
.../impl/consumer/DefaultMQPullConsumerImpl.java | 10 +-
.../impl/consumer/DefaultMQPushConsumerImpl.java | 85 +-
.../client/impl/consumer/PullAPIWrapper.java | 339 +-----
.../client/impl/consumer/PullResultExt.java | 12 +
.../impl/consumer/PullResultWithLogicalQueues.java | 96 --
.../client/impl/consumer/RebalanceImpl.java | 7 +-
.../client/impl/consumer/RebalancePushImpl.java | 15 +-
.../client/impl/factory/MQClientInstance.java | 169 ++-
.../impl/producer/DefaultMQProducerImpl.java | 236 +---
.../client/producer/SendResultForLogicalQueue.java | 46 -
.../DefaultMQPullConsumerLogicalQueueTest.java | 248 -----
.../store/RemoteBrokerOffsetStoreTest.java | 10 +-
.../DefaultMQProducerLogicalQueueTest.java | 311 ------
.../client/producer/DefaultMQProducerTest.java | 28 +-
.../DefaultMQProducerWithOpenTracingTest.java | 14 +-
.../trace/DefaultMQProducerWithTraceTest.java | 26 +-
.../TransactionMQProducerWithOpenTracingTest.java | 16 +-
.../trace/TransactionMQProducerWithTraceTest.java | 40 +-
.../java/org/apache/rocketmq/common/MixAll.java | 20 +-
.../org/apache/rocketmq/common/TopicConfig.java | 6 +
.../apache/rocketmq/common/admin/TopicOffset.java | 9 +
.../rocketmq/common/protocol/RequestCode.java | 11 +-
.../rocketmq/common/protocol/ResponseCode.java | 9 +
...eateMessageQueueForLogicalQueueRequestBody.java | 50 -
.../protocol/body/MigrateLogicalQueueBody.java | 42 -
.../common/protocol/body/RegisterBrokerBody.java | 32 +-
.../body/ReuseTopicLogicalQueueRequestBody.java | 59 -
.../body/SealTopicLogicalQueueRequestBody.java | 49 -
.../TopicConfigAndMappingSerializeWrapper.java | 68 ++
.../protocol/body/TopicConfigSerializeWrapper.java | 11 -
...java => TopicQueueMappingSerializeWrapper.java} | 34 +-
.../protocol/header/CreateTopicRequestHeader.java | 12 +
.../GetEarliestMsgStoretimeRequestHeader.java | 8 +-
.../protocol/header/GetMaxOffsetRequestHeader.java | 25 +-
.../protocol/header/GetMinOffsetRequestHeader.java | 8 +-
.../header/GetTopicConfigRequestHeader.java | 5 +-
.../header/GetTopicStatsInfoRequestHeader.java | 4 +-
.../protocol/header/PullMessageRequestHeader.java | 8 +-
.../protocol/header/PullMessageResponseHeader.java | 11 +
.../header/QueryConsumerOffsetRequestHeader.java | 18 +-
.../protocol/header/SearchOffsetRequestHeader.java | 8 +-
.../protocol/header/SendMessageRequestHeader.java | 8 +-
.../header/UpdateConsumerOffsetRequestHeader.java | 8 +-
.../header/namesrv/GetRouteInfoRequestHeader.java | 20 -
.../protocol/route/LogicalQueueRouteData.java | 309 ------
.../common/protocol/route/LogicalQueuesInfo.java | 87 --
.../protocol/route/LogicalQueuesInfoUnordered.java | 108 --
.../common/protocol/route/TopicRouteData.java | 28 +-
.../protocol/route/TopicRouteDataNameSrv.java | 64 --
.../apache/rocketmq/common/rpc/ClientMetadata.java | 176 +++
.../apache/rocketmq/common/rpc/RequestBuilder.java | 81 ++
.../RpcClient.java} | 36 +-
.../RpcClientHook.java} | 24 +-
.../apache/rocketmq/common/rpc/RpcClientImpl.java | 317 ++++++
.../apache/rocketmq/common/rpc/RpcClientUtils.java | 58 +
.../RpcException.java} | 29 +-
.../RpcRequest.java} | 28 +-
.../RpcRequestHeader.java} | 54 +-
.../apache/rocketmq/common/rpc/RpcResponse.java | 70 ++
.../TopicQueueRequestHeader.java} | 23 +-
.../TopicRequestHeader.java} | 27 +-
.../common/statictopic/LogicQueueMappingItem.java | 212 ++++
.../statictopic/TopicConfigAndQueueMapping.java | 63 ++
.../statictopic/TopicQueueMappingContext.java | 99 ++
.../statictopic/TopicQueueMappingDetail.java | 144 +++
.../common/statictopic/TopicQueueMappingInfo.java | 161 +++
.../common/statictopic/TopicQueueMappingOne.java | 88 ++
.../common/statictopic/TopicQueueMappingUtils.java | 694 ++++++++++++
.../statictopic/TopicRemappingDetailWrapper.java | 104 ++
.../rocketmq/common/sysflag/MessageSysFlag.java | 1 -
.../rocketmq/common/sysflag/PullSysFlag.java | 4 +
.../apache/rocketmq/common/ConfigManagerTest.java | 5 +-
.../rocketmq/common/RegisterBrokerBodyTest.java | 3 +-
.../org/apache/rocketmq/common/UtilAllTest.java | 3 +
.../GenericMapSuperclassDeserializerTest.java | 57 -
.../common/statictopic/TopicQueueMappingTest.java | 63 ++
.../statictopic/TopicQueueMappingUtilsTest.java | 303 +++++
..._Topic_Logic_Queue_\350\256\276\350\256\241.md" | 498 +++++++++
docs/cn/statictopic/The_Scope_Of_Static_Topic.md | 116 ++
.../processor/ClusterTestRequestProcessor.java | 2 +-
.../namesrv/processor/DefaultRequestProcessor.java | 19 +-
.../namesrv/routeinfo/RouteInfoManager.java | 76 +-
.../processor/DefaultRequestProcessorTest.java | 21 +-
.../remoting/netty/NettyRemotingClient.java | 1 +
.../remoting/protocol/RemotingCommand.java | 47 +-
.../remoting/protocol/RemotingCommandTest.java | 101 +-
.../test/client/rmq/RMQNormalConsumer.java | 5 +
.../test/client/rmq/RMQNormalProducer.java | 7 +
.../test/clientinterface/AbstractMQProducer.java | 1 +
.../org/apache/rocketmq/test/util/MQAdmin.java | 166 ---
.../rocketmq/test/util/MQAdminTestUtils.java | 274 +++++
.../org/apache/rocketmq/test/base/BaseConf.java | 38 +-
.../rocketmq/test/base/IntegrationTestBase.java | 7 +-
.../apache/rocketmq/test/smoke/LogicalQueueIT.java | 1170 --------------------
.../rocketmq/test/statictopic/StaticTopicIT.java | 465 ++++++++
.../rocketmq/tools/admin/DefaultMQAdminExt.java | 75 +-
.../tools/admin/DefaultMQAdminExtImpl.java | 126 +--
.../apache/rocketmq/tools/admin/MQAdminExt.java | 43 +-
.../apache/rocketmq/tools/admin/MQAdminUtils.java | 343 ++++++
.../rocketmq/tools/command/MQAdminStartup.java | 24 +-
.../DeleteTopicLogicalQueueMappingCommand.java | 91 --
.../MigrateTopicLogicalQueueCommand.java | 210 ----
.../QueryTopicLogicalQueueMappingCommand.java | 123 --
.../UpdateTopicLogicalQueueMappingCommand.java | 159 ---
.../UpdateTopicLogicalQueueNumCommand.java | 285 -----
.../topic/RemappingStaticTopicSubCommand.java | 207 ++++
.../command/topic/UpdateStaticTopicSubCommand.java | 208 ++++
.../tools/command/topic/UpdateTopicSubCommand.java | 20 -
.../tools/admin/DefaultMQAdminExtTest.java | 31 +-
.../command/message/ConsumeMessageCommandTest.java | 4 +-
139 files changed, 7453 insertions(+), 7407 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
index 5eb9169..76c8007 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
@@ -17,29 +17,6 @@
package org.apache.rocketmq.broker;
import com.google.common.collect.Maps;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.AbstractMap;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.function.Function;
-import java.util.stream.Collectors;
import org.apache.rocketmq.acl.AccessValidator;
import org.apache.rocketmq.broker.client.ClientHousekeepingService;
import org.apache.rocketmq.broker.client.ConsumerIdsChangeListener;
@@ -49,7 +26,6 @@ import org.apache.rocketmq.broker.client.ProducerManager;
import org.apache.rocketmq.broker.client.net.Broker2Client;
import org.apache.rocketmq.broker.client.rebalance.RebalanceLockManager;
import org.apache.rocketmq.broker.dledger.DLedgerRoleChangeHandler;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.filter.CommitLogDispatcherCalcBitMap;
import org.apache.rocketmq.broker.filter.ConsumerFilterManager;
import org.apache.rocketmq.broker.filtersrv.FilterServerManager;
@@ -80,6 +56,8 @@ import org.apache.rocketmq.broker.processor.SendMessageProcessor;
import org.apache.rocketmq.broker.slave.SlaveSynchronize;
import org.apache.rocketmq.broker.subscription.SubscriptionGroupManager;
import org.apache.rocketmq.broker.topic.TopicConfigManager;
+import org.apache.rocketmq.broker.topic.TopicQueueMappingCleanService;
+import org.apache.rocketmq.broker.topic.TopicQueueMappingManager;
import org.apache.rocketmq.broker.transaction.AbstractTransactionalMessageCheckListener;
import org.apache.rocketmq.broker.transaction.TransactionalMessageCheckService;
import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
@@ -87,11 +65,9 @@ import org.apache.rocketmq.broker.transaction.queue.DefaultTransactionalMessageC
import org.apache.rocketmq.broker.transaction.queue.TransactionalMessageBridge;
import org.apache.rocketmq.broker.transaction.queue.TransactionalMessageServiceImpl;
import org.apache.rocketmq.broker.util.ServiceProvider;
-import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.Configuration;
import org.apache.rocketmq.common.DataVersion;
-import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.ThreadFactoryImpl;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
@@ -99,18 +75,16 @@ import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.namesrv.RegisterBrokerResult;
import org.apache.rocketmq.common.protocol.RequestCode;
-import org.apache.rocketmq.common.protocol.body.ClusterInfo;
+import org.apache.rocketmq.common.protocol.body.TopicConfigAndMappingSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingInfo;
import org.apache.rocketmq.common.stats.MomentStatsItem;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.RPCHook;
import org.apache.rocketmq.remoting.RemotingServer;
import org.apache.rocketmq.remoting.common.TlsMode;
-import org.apache.rocketmq.remoting.exception.RemotingConnectException;
-import org.apache.rocketmq.remoting.exception.RemotingSendRequestException;
-import org.apache.rocketmq.remoting.exception.RemotingTimeoutException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyRemotingServer;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
@@ -128,6 +102,30 @@ import org.apache.rocketmq.store.dledger.DLedgerCommitLog;
import org.apache.rocketmq.store.stats.BrokerStats;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
public class BrokerController {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private static final InternalLogger LOG_PROTECTION = InternalLoggerFactory.getLogger(LoggerName.PROTECTION_LOGGER_NAME);
@@ -179,6 +177,7 @@ public class BrokerController {
private RemotingServer remotingServer;
private RemotingServer fastRemotingServer;
private TopicConfigManager topicConfigManager;
+ private TopicQueueMappingManager topicQueueMappingManager;
private ExecutorService sendMessageExecutor;
private ExecutorService pullMessageExecutor;
private ExecutorService ackMessageExecutor;
@@ -195,12 +194,13 @@ public class BrokerController {
private InetSocketAddress storeHost;
private BrokerFastFailure brokerFastFailure;
private Configuration configuration;
+ private TopicQueueMappingCleanService topicQueueMappingCleanService;
private FileWatchService fileWatchService;
private TransactionalMessageCheckService transactionalMessageCheckService;
private TransactionalMessageService transactionalMessageService;
private AbstractTransactionalMessageCheckListener transactionalMessageCheckListener;
private Future<?> slaveSyncFuture;
- private Map<Class,AccessValidator> accessValidatorMap = new HashMap<Class, AccessValidator>();
+ private Map<Class, AccessValidator> accessValidatorMap = new HashMap<Class, AccessValidator>();
private long shouldStartTime;
public BrokerController(
@@ -215,6 +215,7 @@ public class BrokerController {
this.messageStoreConfig = messageStoreConfig;
this.consumerOffsetManager = new ConsumerOffsetManager(this);
this.topicConfigManager = new TopicConfigManager(this);
+ this.topicQueueMappingManager = new TopicQueueMappingManager(this);
this.pullMessageProcessor = new PullMessageProcessor(this);
this.pullRequestHoldService = new PullRequestHoldService(this);
this.popMessageProcessor = new PopMessageProcessor(this);
@@ -231,7 +232,7 @@ public class BrokerController {
this.clientHousekeepingService = new ClientHousekeepingService(this);
this.broker2Client = new Broker2Client(this);
this.subscriptionGroupManager = new SubscriptionGroupManager(this);
- this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig);
+ this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig, this);
this.filterServerManager = new FilterServerManager(this);
this.assignmentManager = new AssignmentManager(this);
@@ -287,18 +288,20 @@ public class BrokerController {
public boolean initialize() throws CloneNotSupportedException {
boolean result = this.topicConfigManager.load();
+ result = result && this.topicQueueMappingManager.load();
+
result = result && this.consumerOffsetManager.load();
result = result && this.subscriptionGroupManager.load();
result = result && this.consumerFilterManager.load();
if (result) {
try {
- DefaultMessageStore messageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
- messageStore.registerCleanFileHook(topicConfigManager.getLogicalQueueCleanHook());
- this.messageStore = messageStore;
+ this.messageStore =
+ new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener,
+ this.brokerConfig);
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
- ((DLedgerCommitLog)((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
+ ((DLedgerCommitLog) ((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
}
this.brokerStats = new BrokerStats((DefaultMessageStore) this.messageStore);
//load plugin
@@ -342,7 +345,6 @@ public class BrokerController {
this.ackThreadPoolQueue,
new ThreadFactoryImpl("AckMessageThread_"));
-
this.replyMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
@@ -485,11 +487,11 @@ public class BrokerController {
this.scheduledExecutorService.scheduleAtFixedRate(() -> {
try {
- BrokerController.this.refreshBrokerNameMapping();
+ BrokerController.this.brokerOuterAPI.refreshMetadata();
} catch (Exception e) {
- log.error("ScheduledTask examineBrokerClusterInfo exception", e);
+ log.error("ScheduledTask refresh metadata exception", e);
}
- }, 10, 10, TimeUnit.SECONDS);
+ }, 1, 5, TimeUnit.SECONDS);
if (!messageStoreConfig.isEnableDLegerCommitLog()) {
if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
@@ -513,6 +515,8 @@ public class BrokerController {
}
}
+ this.topicQueueMappingCleanService = new TopicQueueMappingCleanService(this);
+
if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
// Register a listener to reload SslContext
try {
@@ -587,9 +591,9 @@ public class BrokerController {
return;
}
- for (AccessValidator accessValidator: accessValidators) {
+ for (AccessValidator accessValidator : accessValidators) {
final AccessValidator validator = accessValidator;
- accessValidatorMap.put(validator.getClass(),validator);
+ accessValidatorMap.put(validator.getClass(), validator);
this.registerServerRPCHook(new RPCHook() {
@Override
@@ -605,26 +609,17 @@ public class BrokerController {
}
}
-
private void initialRpcHooks() {
List<RPCHook> rpcHooks = ServiceProvider.load(ServiceProvider.RPC_HOOK_ID, RPCHook.class);
if (rpcHooks == null || rpcHooks.isEmpty()) {
return;
}
- for (RPCHook rpcHook: rpcHooks) {
+ for (RPCHook rpcHook : rpcHooks) {
this.registerServerRPCHook(rpcHook);
}
}
- private void refreshBrokerNameMapping() throws InterruptedException, MQBrokerException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- ClusterInfo brokerClusterInfo = this.brokerOuterAPI.getBrokerClusterInfo();
- brokerClusterInfo.getBrokerAddrTable().forEach((brokerName, data) -> {
- String masterBrokerAddr = data.getBrokerAddrs().get(MixAll.MASTER_ID);
- this.brokerName2AddrMap.put(brokerName, masterBrokerAddr);
- });
- }
-
public String getBrokerAddrByName(String brokerName) {
return this.brokerName2AddrMap.get(brokerName);
}
@@ -878,6 +873,10 @@ public class BrokerController {
this.fastRemotingServer.shutdown();
}
+ if (this.topicQueueMappingCleanService != null) {
+ this.topicQueueMappingCleanService.shutdown();
+ }
+
if (this.fileWatchService != null) {
this.fileWatchService.shutdown();
}
@@ -995,6 +994,10 @@ public class BrokerController {
this.fastRemotingServer.start();
}
+ if (this.topicQueueMappingCleanService != null) {
+ this.topicQueueMappingCleanService.start();
+ }
+
if (this.fileWatchService != null) {
this.fileWatchService.start();
}
@@ -1041,7 +1044,6 @@ public class BrokerController {
this.brokerFastFailure.start();
}
-
}
public synchronized void registerIncrementBrokerData(TopicConfig topicConfig, DataVersion dataVersion) {
@@ -1053,7 +1055,7 @@ public class BrokerController {
return;
}
- TopicConfigSerializeWrapper topicConfigSerializeWrapper = new TopicConfigSerializeWrapper();
+ TopicConfigAndMappingSerializeWrapper topicConfigSerializeWrapper = new TopicConfigAndMappingSerializeWrapper();
topicConfigSerializeWrapper.setDataVersion(dataVersion);
ConcurrentMap<String, TopicConfig> topicConfigTable = topicConfigList.stream()
@@ -1074,49 +1076,41 @@ public class BrokerController {
.collect(Collectors.toConcurrentMap(TopicConfig::getTopicName, Function.identity()));
topicConfigSerializeWrapper.setTopicConfigTable(topicConfigTable);
- String brokerName = this.brokerConfig.getBrokerName();
- Map<String, LogicalQueuesInfo> logicalQueuesInfoMap = topicConfigList.stream()
+ Map<String, TopicQueueMappingInfo> topicQueueMappingInfoMap = topicConfigList.stream()
.map(TopicConfig::getTopicName)
- .map(topicName -> Optional.ofNullable(this.topicConfigManager.selectLogicalQueuesInfo(topicName))
- .map(info -> {
- info.readLock().lock();
- try {
- return new AbstractMap.SimpleImmutableEntry<>(topicName, new LogicalQueuesInfoInBroker(info, data -> Objects.equals(data.getBrokerName(), brokerName)));
- } finally {
- info.readLock().unlock();
- }
- })
+ .map(topicName -> Optional.ofNullable(this.topicQueueMappingManager.getTopicQueueMapping(topicName))
+ .map(info -> new AbstractMap.SimpleImmutableEntry<>(topicName, TopicQueueMappingDetail.cloneAsMappingInfo(info)))
.orElse(null))
.filter(Objects::nonNull)
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- if (!logicalQueuesInfoMap.isEmpty()) {
- topicConfigSerializeWrapper.setLogicalQueuesInfoMap(logicalQueuesInfoMap);
+ if (!topicQueueMappingInfoMap.isEmpty()) {
+ topicConfigSerializeWrapper.setTopicQueueMappingInfoMap(topicQueueMappingInfoMap);
}
doRegisterBrokerAll(true, false, topicConfigSerializeWrapper);
}
public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
- TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();
+
+ TopicConfigAndMappingSerializeWrapper topicConfigWrapper = new TopicConfigAndMappingSerializeWrapper();
+
+ topicConfigWrapper.setDataVersion(this.getTopicConfigManager().getDataVersion());
+ topicConfigWrapper.setTopicConfigTable(this.getTopicConfigManager().getTopicConfigTable());
+
+ topicConfigWrapper.setTopicQueueMappingInfoMap(this.getTopicQueueMappingManager().getTopicQueueMappingTable().entrySet().stream().map(
+ entry -> new AbstractMap.SimpleImmutableEntry<>(entry.getKey(), TopicQueueMappingDetail.cloneAsMappingInfo(entry.getValue()))
+ ).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)));
if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
|| !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
- Map<String, LogicalQueuesInfo> logicalQueuesInfoMap = Maps.newHashMapWithExpectedSize(topicConfigWrapper.getTopicConfigTable().size());
for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
- String topicName = topicConfig.getTopicName();
TopicConfig tmp =
- new TopicConfig(topicName, topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
+ new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
this.brokerConfig.getBrokerPermission());
- topicConfigTable.put(topicName, tmp);
- LogicalQueuesInfoInBroker logicalQueuesInfo = this.topicConfigManager.selectLogicalQueuesInfo(topicName);
- if (logicalQueuesInfo != null) {
- String brokerName = this.brokerConfig.getBrokerName();
- logicalQueuesInfoMap.put(topicName, new LogicalQueuesInfoInBroker(logicalQueuesInfo, data -> Objects.equals(data.getBrokerName(), brokerName)));
- }
+ topicConfigTable.put(topicConfig.getTopicName(), tmp);
}
topicConfigWrapper.setTopicConfigTable(topicConfigTable);
- topicConfigWrapper.setLogicalQueuesInfoMap(logicalQueuesInfoMap);
}
if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
@@ -1129,7 +1123,7 @@ public class BrokerController {
}
private void doRegisterBrokerAll(boolean checkOrderConfig, boolean oneway,
- TopicConfigSerializeWrapper topicConfigWrapper) {
+ TopicConfigAndMappingSerializeWrapper topicConfigWrapper) {
List<RegisterBrokerResult> registerBrokerResultList = this.brokerOuterAPI.registerBrokerAll(
this.brokerConfig.getBrokerClusterName(),
this.getBrokerAddr(),
@@ -1184,6 +1178,10 @@ public class BrokerController {
this.topicConfigManager = topicConfigManager;
}
+ public TopicQueueMappingManager getTopicQueueMappingManager() {
+ return topicQueueMappingManager;
+ }
+
public String getHAServerAddr() {
return this.brokerConfig.getBrokerIP2() + ":" + this.messageStoreConfig.getHaListenPort();
}
@@ -1297,7 +1295,6 @@ public class BrokerController {
this.transactionalMessageCheckListener = transactionalMessageCheckListener;
}
-
public BlockingQueue<Runnable> getEndTransactionThreadPoolQueue() {
return endTransactionThreadPoolQueue;
@@ -1318,8 +1315,7 @@ public class BrokerController {
public void run() {
try {
BrokerController.this.slaveSynchronize.syncAll();
- }
- catch (Throwable e) {
+ } catch (Throwable e) {
log.error("ScheduledTask SlaveSynchronize syncAll error.", e);
}
}
@@ -1365,8 +1361,6 @@ public class BrokerController {
log.info("Finish to change to slave brokerName={} brokerId={}", brokerConfig.getBrokerName(), brokerId);
}
-
-
public void changeToMaster(BrokerRole role) {
if (role == BrokerRole.SLAVE) {
return;
@@ -1435,4 +1429,8 @@ public class BrokerController {
public QueryAssignmentProcessor getQueryAssignmentProcessor() {
return queryAssignmentProcessor;
}
+
+ public TopicQueueMappingCleanService getTopicQueueMappingCleanService() {
+ return topicQueueMappingCleanService;
+ }
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/BrokerPathConfigHelper.java b/broker/src/main/java/org/apache/rocketmq/broker/BrokerPathConfigHelper.java
index 43a9946..6360d54 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/BrokerPathConfigHelper.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/BrokerPathConfigHelper.java
@@ -35,6 +35,10 @@ public class BrokerPathConfigHelper {
return rootDir + File.separator + "config" + File.separator + "topics.json";
}
+ public static String getTopicQueueMappingPath(final String rootDir) {
+ return rootDir + File.separator + "config" + File.separator + "topicQueueMapping.json";
+ }
+
public static String getConsumerOffsetPath(final String rootDir) {
return rootDir + File.separator + "config" + File.separator + "consumerOffset.json";
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/client/rebalance/RebalanceLockManager.java b/broker/src/main/java/org/apache/rocketmq/broker/client/rebalance/RebalanceLockManager.java
index 678b1f5..9056998 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/client/rebalance/RebalanceLockManager.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/client/rebalance/RebalanceLockManager.java
@@ -16,16 +16,17 @@
*/
package org.apache.rocketmq.broker.client.rebalance;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
-import org.apache.rocketmq.common.constant.LoggerName;
-import org.apache.rocketmq.logging.InternalLogger;
-import org.apache.rocketmq.logging.InternalLoggerFactory;
-import org.apache.rocketmq.common.message.MessageQueue;
public class RebalanceLockManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.REBALANCE_LOCK_LOGGER_NAME);
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/domain/LogicalQueuesInfoInBroker.java b/broker/src/main/java/org/apache/rocketmq/broker/domain/LogicalQueuesInfoInBroker.java
deleted file mode 100644
index a6728c8..0000000
--- a/broker/src/main/java/org/apache/rocketmq/broker/domain/LogicalQueuesInfoInBroker.java
+++ /dev/null
@@ -1,116 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.rocketmq.broker.domain;
-
-import com.alibaba.fastjson.parser.ParserConfig;
-import com.google.common.collect.Maps;
-import java.util.Collections;
-import java.util.List;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ConcurrentNavigableMap;
-import java.util.concurrent.ConcurrentSkipListMap;
-import java.util.function.Predicate;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
-import org.apache.rocketmq.common.fastjson.GenericMapSuperclassDeserializer;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.srvutil.ConcurrentHashMapUtil;
-
-import static java.util.Optional.ofNullable;
-
-public class LogicalQueuesInfoInBroker extends LogicalQueuesInfo {
- private final ConcurrentMap<Integer, ConcurrentNavigableMap<Long, LogicalQueueRouteData>> queueId2LogicalQueueMap = Maps.newConcurrentMap();
-
- public LogicalQueuesInfoInBroker() {
- }
-
- public LogicalQueuesInfoInBroker(LogicalQueuesInfoInBroker other) {
- this(other, null);
- }
-
- // deep copy
- public LogicalQueuesInfoInBroker(LogicalQueuesInfoInBroker other, Predicate<LogicalQueueRouteData> predicate) {
- other.readLock().lock();
- try {
- for (Entry<Integer, List<LogicalQueueRouteData>> entry : other.entrySet()) {
- Stream<LogicalQueueRouteData> stream = entry.getValue().stream();
- if (predicate != null) {
- stream = stream.filter(predicate);
- }
- this.put(entry.getKey(), stream.map(LogicalQueueRouteData::new).collect(Collectors.toList()));
- }
- } finally {
- other.readLock().unlock();
- }
- }
-
- public void updateQueueRouteDataByQueueId(int queueId, LogicalQueueRouteData queueRouteData) {
- if (queueRouteData == null) {
- return;
- }
- ConcurrentHashMapUtil.computeIfAbsent(queueId2LogicalQueueMap, queueId, k -> new ConcurrentSkipListMap<>()).put(queueRouteData.getOffsetDelta(), queueRouteData);
- }
-
- /**
- * find logical queue route data for message queues owned by this broker
- */
- public LogicalQueueRouteData queryQueueRouteDataByQueueId(int queueId, long offset) {
- ConcurrentNavigableMap<Long, LogicalQueueRouteData> m = this.queueId2LogicalQueueMap.get(queueId);
- if (m == null || m.isEmpty()) {
- return null;
- }
- Entry<Long, LogicalQueueRouteData> entry = m.floorEntry(offset);
- if (entry == null) {
- return null;
- }
- return entry.getValue();
- }
-
- public void deleteQueueRouteData(LogicalQueueRouteData logicalQueueRouteData) {
- ConcurrentNavigableMap<Long, LogicalQueueRouteData> m = this.queueId2LogicalQueueMap.get(logicalQueueRouteData.getQueueId());
- if (m != null) {
- m.remove(logicalQueueRouteData.getOffsetDelta(), logicalQueueRouteData);
- }
- }
-
- public LogicalQueueRouteData nextAvailableLogicalRouteData(LogicalQueueRouteData queueRouteData,
- Predicate<LogicalQueueRouteData> predicate) {
- this.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = ofNullable(this.get(queueRouteData.getLogicalQueueIndex())).orElse(Collections.emptyList());
- int idx = Collections.binarySearch(queueRouteDataList, queueRouteData);
- if (idx >= 0) {
- for (int i = idx + 1, size = queueRouteDataList.size(); i < size; i++) {
- LogicalQueueRouteData tmp = queueRouteDataList.get(i);
- if (predicate.test(tmp)) {
- return tmp;
- }
- }
- }
- } finally {
- this.readLock().unlock();
- }
- return null;
- }
-
- static {
- // workaround https://github.com/alibaba/fastjson/issues/3730
- ParserConfig.getGlobalInstance().putDeserializer(LogicalQueuesInfoInBroker.class, GenericMapSuperclassDeserializer.INSTANCE);
- }
-}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/out/BrokerOuterAPI.java b/broker/src/main/java/org/apache/rocketmq/broker/out/BrokerOuterAPI.java
index 4d33663..8a2093a 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/out/BrokerOuterAPI.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/out/BrokerOuterAPI.java
@@ -16,13 +16,7 @@
*/
package org.apache.rocketmq.broker.out;
-import java.io.UnsupportedEncodingException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.ArrayBlockingQueue;
-import java.util.concurrent.CopyOnWriteArrayList;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
+import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.latency.BrokerFixedThreadPoolExecutor;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.common.DataVersion;
@@ -39,6 +33,7 @@ import org.apache.rocketmq.common.protocol.body.ConsumerOffsetSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.KVTable;
import org.apache.rocketmq.common.protocol.body.RegisterBrokerBody;
import org.apache.rocketmq.common.protocol.body.SubscriptionGroupWrapper;
+import org.apache.rocketmq.common.protocol.body.TopicConfigAndMappingSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
import org.apache.rocketmq.common.protocol.header.namesrv.GetRouteInfoRequestHeader;
import org.apache.rocketmq.common.protocol.header.namesrv.QueryDataVersionRequestHeader;
@@ -47,6 +42,9 @@ import org.apache.rocketmq.common.protocol.header.namesrv.RegisterBrokerRequestH
import org.apache.rocketmq.common.protocol.header.namesrv.RegisterBrokerResponseHeader;
import org.apache.rocketmq.common.protocol.header.namesrv.UnRegisterBrokerRequestHeader;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
+import org.apache.rocketmq.common.rpc.ClientMetadata;
+import org.apache.rocketmq.common.rpc.RpcClient;
+import org.apache.rocketmq.common.rpc.RpcClientImpl;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.InvokeCallback;
@@ -62,21 +60,39 @@ import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyRemotingClient;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
public class BrokerOuterAPI {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private final RemotingClient remotingClient;
+ private final BrokerController brokerController;
private final TopAddressing topAddressing = new TopAddressing(MixAll.getWSAddr());
private String nameSrvAddr = null;
+ private final String currBrokerName;
private BrokerFixedThreadPoolExecutor brokerOuterExecutor = new BrokerFixedThreadPoolExecutor(4, 10, 1, TimeUnit.MINUTES,
new ArrayBlockingQueue<Runnable>(32), new ThreadFactoryImpl("brokerOutApi_thread_", true));
- public BrokerOuterAPI(final NettyClientConfig nettyClientConfig) {
- this(nettyClientConfig, null);
+ private ClientMetadata clientMetadata;
+ private RpcClient rpcClient;
+
+ public BrokerOuterAPI(final NettyClientConfig nettyClientConfig, final BrokerController brokerController) {
+
+ this(nettyClientConfig, null, brokerController, new ClientMetadata());
}
- public BrokerOuterAPI(final NettyClientConfig nettyClientConfig, RPCHook rpcHook) {
+ private BrokerOuterAPI(final NettyClientConfig nettyClientConfig, RPCHook rpcHook, final BrokerController brokerController, ClientMetadata clientMetadata) {
this.remotingClient = new NettyRemotingClient(nettyClientConfig);
+ this.clientMetadata = clientMetadata;
this.remotingClient.registerRPCHook(rpcHook);
+ this.brokerController = brokerController;
+ this.currBrokerName = brokerController.getBrokerConfig().getBrokerName();
+ this.rpcClient = new RpcClientImpl(this.clientMetadata, this.remotingClient);
}
public void start() {
@@ -140,7 +156,7 @@ public class BrokerOuterAPI {
requestHeader.setCompressed(compressed);
RegisterBrokerBody requestBody = new RegisterBrokerBody();
- requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
+ requestBody.setTopicConfigSerializeWrapper(TopicConfigAndMappingSerializeWrapper.from(topicConfigWrapper));
requestBody.setFilterServerList(filterServerList);
final byte[] body = requestBody.encode(compressed);
final int bodyCrc32 = UtilAll.crc32(body);
@@ -327,7 +343,7 @@ public class BrokerOuterAPI {
return changedList;
}
- public TopicConfigSerializeWrapper getAllTopicConfig(
+ public TopicConfigAndMappingSerializeWrapper getAllTopicConfig(
final String addr) throws RemotingConnectException, RemotingSendRequestException,
RemotingTimeoutException, InterruptedException, MQBrokerException {
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
@@ -336,7 +352,7 @@ public class BrokerOuterAPI {
assert response != null;
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
- return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
+ return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigAndMappingSerializeWrapper.class);
}
default:
break;
@@ -453,4 +469,19 @@ public class BrokerOuterAPI {
public void forwardRequest(String brokerAddr, RemotingCommand request, long timeoutMillis, InvokeCallback invokeCallback) throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException, RemotingTooMuchRequestException, RemotingConnectException {
this.remotingClient.invokeAsync(brokerAddr, request, timeoutMillis, invokeCallback);
}
+
+
+ public void refreshMetadata() throws Exception {
+ ClusterInfo brokerClusterInfo = getBrokerClusterInfo();
+ clientMetadata.refreshClusterInfo(brokerClusterInfo);
+ }
+
+
+ public ClientMetadata getClientMetadata() {
+ return clientMetadata;
+ }
+
+ public RpcClient getRpcClient() {
+ return rpcClient;
+ }
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AbstractSendMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AbstractSendMessageProcessor.java
index af14b5b..3303d70 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AbstractSendMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AbstractSendMessageProcessor.java
@@ -16,31 +16,21 @@
*/
package org.apache.rocketmq.broker.processor;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.Maps;
import io.netty.channel.ChannelHandlerContext;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
-
import java.util.concurrent.ThreadLocalRandom;
-import java.util.Optional;
-import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.atomic.LongAdder;
import org.apache.rocketmq.broker.BrokerController;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.mqtrace.SendMessageContext;
import org.apache.rocketmq.broker.mqtrace.SendMessageHook;
-import org.apache.rocketmq.broker.topic.TopicConfigManager;
+import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.TopicFilterType;
-import org.apache.rocketmq.common.TopicQueueId;
import org.apache.rocketmq.common.constant.DBMsgConstants;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.constant.PermName;
@@ -54,23 +44,17 @@ import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeaderV2;
import org.apache.rocketmq.common.protocol.header.SendMessageResponseHeader;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
-import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.common.utils.ChannelUtil;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.AsyncNettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-import org.apache.rocketmq.srvutil.ConcurrentHashMapUtil;
import org.apache.rocketmq.store.MessageExtBrokerInner;
-import org.apache.rocketmq.store.PutMessageResult;
public abstract class AbstractSendMessageProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
protected static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
@@ -80,8 +64,6 @@ public abstract class AbstractSendMessageProcessor extends AsyncNettyRequestProc
protected final SocketAddress storeHost;
private List<SendMessageHook> sendMessageHookList;
- private final ConcurrentMap<TopicQueueId, LongAdder> inFlyWritingCounterMap = Maps.newConcurrentMap();
-
public AbstractSendMessageProcessor(final BrokerController brokerController) {
this.brokerController = brokerController;
this.storeHost =
@@ -419,158 +401,4 @@ public abstract class AbstractSendMessageProcessor extends AsyncNettyRequestProc
public boolean rejectRequest() {
return false;
}
-
-
- public ConcurrentMap<TopicQueueId, LongAdder> getInFlyWritingCounterMap() {
- return inFlyWritingCounterMap;
- }
-
- protected LogicalQueueContext buildLogicalQueueContext(String topic, int queueId,
- RemotingCommand response) {
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- return noopLogicalQueueContext;
- }
- // writable route data will has largest offset
- LogicalQueueRouteData curQueueRouteData = logicalQueuesInfo.queryQueueRouteDataByQueueId(queueId, Long.MAX_VALUE);
- if (curQueueRouteData == null) {
- // topic enabled logical queue, but some message queues are not converted or being converted
- String msg = String.format(Locale.ENGLISH, "queueId %d not included in logical queue", queueId);
- log.debug("buildLogicalQueueContext unexpected error, topic {} {}", topic, msg);
- response.setCode(ResponseCode.SYSTEM_ERROR);
- response.setRemark(msg);
- return noopLogicalQueueContext;
- }
- LongAdder inFlyWritingCounter = ConcurrentHashMapUtil.computeIfAbsent(inFlyWritingCounterMap, new TopicQueueId(topic, queueId), ignore -> new LongAdder());
- return new LogicalQueueContext(topic, queueId, logicalQueuesInfo, curQueueRouteData, inFlyWritingCounter);
- }
-
- protected class LogicalQueueContext {
- private final String topic;
- private final int queueId;
- private final LogicalQueuesInfoInBroker logicalQueuesInfo;
- private final LogicalQueueRouteData curQueueRouteData;
- private final LongAdder inFlyWritingCounter;
-
- public LogicalQueueContext(String topic, int queueId,
- LogicalQueuesInfoInBroker logicalQueuesInfo,
- LogicalQueueRouteData curQueueRouteData, LongAdder inFlyWritingCounter) {
- this.topic = topic;
- this.queueId = queueId;
- this.logicalQueuesInfo = logicalQueuesInfo;
- this.curQueueRouteData = curQueueRouteData;
- this.inFlyWritingCounter = inFlyWritingCounter;
- }
-
- public CompletableFuture<RemotingCommand> hookBeforePut(ChannelHandlerContext ctx, SendMessageRequestHeader requestHeader,
- RemotingCommand request, RemotingCommand response) {
- if (curQueueRouteData.isWritable()) {
- this.inFlyWritingCounter.increment();
- return null;
- }
- int logicalQueueIdx = curQueueRouteData.getLogicalQueueIndex();
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(logicalQueueIdx);
- LogicalQueueRouteData writableQueueRouteData = null;
- for (int i = queueRouteDataList.size() - 1; i >= 0; i--) {
- LogicalQueueRouteData queueRouteData = queueRouteDataList.get(i);
- if (queueRouteData.isWritable()) {
- writableQueueRouteData = queueRouteData;
- break;
- }
- }
- if (writableQueueRouteData == null) {
- response.setCode(ResponseCode.NO_PERMISSION);
- response.setRemark(String.format(Locale.ENGLISH, "broker[%s] topic[%s] queueId[%d] logicalQueueIdx[%d] not writable", AbstractSendMessageProcessor.this.brokerController.getBrokerConfig().getBrokerIP1(), topic, queueId, logicalQueueIdx));
- return CompletableFuture.completedFuture(response);
- }
- if ((Optional.ofNullable(requestHeader.getSysFlag()).orElse(0) & MessageSysFlag.LOGICAL_QUEUE_FLAG) > 0) {
- // new client, use redirect
- response.setCode(ResponseCode.NO_PERMISSION);
- response.addExtField(MessageConst.PROPERTY_REDIRECT, "1");
- response.setBody(RemotingSerializable.encode(ImmutableList.of(curQueueRouteData, writableQueueRouteData)));
- return CompletableFuture.completedFuture(response);
- } else {
- // old client, use forward
- this.logicalQueueHookForward(ctx, writableQueueRouteData, requestHeader, request, response);
- }
- if (response.getCode() != -1) {
- return CompletableFuture.completedFuture(response);
- } else if (response.getCode() == ResponseCode.ASYNC_AND_RETURN_NULL) {
- return CompletableFuture.completedFuture(null);
- }
- return null;
- }
-
- private void logicalQueueHookForward(ChannelHandlerContext ctx,
- LogicalQueueRouteData writableQueueRouteData,
- SendMessageRequestHeader requestHeader, RemotingCommand request,
- RemotingCommand response) {
- response.setCode(ResponseCode.SUCCESS);
- requestHeader.setQueueId(writableQueueRouteData.getQueueId());
- request.writeCustomHeader(requestHeader);
- String brokerName = writableQueueRouteData.getBrokerName();
- BrokerController brokerController = AbstractSendMessageProcessor.this.brokerController;
- String brokerAddr = brokerController.getBrokerAddrByName(brokerName);
- if (brokerAddr == null) {
- log.warn("getBrokerAddrByName brokerName={} got null, fallback to queueRouteData.getBrokerAddr()", brokerName);
- brokerAddr = writableQueueRouteData.getBrokerAddr();
- }
- if (brokerAddr == null) {
- response.setCode(ResponseCode.SYSTEM_ERROR);
- String msg = String.format(Locale.ENGLISH, "unknown brokerName %s", brokerName);
- response.setRemark(msg);
- log.warn("logicalQueueHookForward can not look up brokerName={}: {}", brokerName, requestHeader);
- return;
- }
- try {
- String finalBrokerAddr = brokerAddr;
- brokerController.getBrokerOuterAPI().forwardRequest(brokerAddr, request, brokerController.getBrokerConfig().getForwardTimeout(), responseFuture -> {
- RemotingCommand forwardResponse = responseFuture.getResponseCommand();
- if (forwardResponse == null) {
- forwardResponse = response;
- forwardResponse.setCode(ResponseCode.SYSTEM_ERROR);
- if (!responseFuture.isSendRequestOK()) {
- forwardResponse.setRemark(String.format(Locale.ENGLISH, "send request failed to %s: %s", finalBrokerAddr, responseFuture.getCause()));
- } else if (responseFuture.isTimeout()) {
- forwardResponse.setRemark(String.format(Locale.ENGLISH, "wait response from %s timeout: %dms", finalBrokerAddr, responseFuture.getTimeoutMillis()));
- } else {
- forwardResponse.setRemark(String.format(Locale.ENGLISH, "unknown reason. addr: %s, timeoutMillis: %d: %s", finalBrokerAddr, responseFuture.getTimeoutMillis(), responseFuture.getCause()));
- }
- } else {
- CommandCustomHeader customHeader = forwardResponse.readCustomHeader();
- if (customHeader instanceof SendMessageResponseHeader) {
- SendMessageResponseHeader responseHeader = (SendMessageResponseHeader) customHeader;
- Integer forwardQueueId = responseHeader.getQueueId();
- forwardResponse.addExtField(MessageConst.PROPERTY_FORWARD_QUEUE_ID, forwardQueueId != null ? Integer.toString(forwardQueueId) : "null");
- responseHeader.setQueueId(requestHeader.getQueueId());
- // queueOffset should not be changed since forwarded broker will add delta to it.
- }
- }
- AbstractSendMessageProcessor.this.doResponse(ctx, request, forwardResponse);
- });
- response.setCode(ResponseCode.ASYNC_AND_RETURN_NULL);
- } catch (Exception e) {
- response.setCode(ResponseCode.SYSTEM_ERROR);
- response.setRemark("forward error");
- log.warn(String.format(Locale.ENGLISH, "logicalQueueHookForward to %s error", brokerAddr), e);
- }
- }
-
- public void hookAfterPut(CompletableFuture<PutMessageResult> putMessageResult) {
- Optional.ofNullable(putMessageResult).orElse(CompletableFuture.completedFuture(null)).whenComplete((result, throwable) -> {
- this.inFlyWritingCounter.decrement();
- });
- }
- }
-
- private final LogicalQueueContext noopLogicalQueueContext = new LogicalQueueContext(null, 0, null, null, null) {
- @Override public CompletableFuture<RemotingCommand> hookBeforePut(ChannelHandlerContext ctx, SendMessageRequestHeader requestHeader,
- RemotingCommand request, RemotingCommand response) {
- return null;
- }
-
- @Override public void hookAfterPut(CompletableFuture<PutMessageResult> putMessageResult) {
- }
- };
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index e7b7949..2891baf 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -18,48 +18,22 @@ package org.apache.rocketmq.broker.processor;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
-import java.io.UnsupportedEncodingException;
-import java.net.UnknownHostException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.Properties;
-import java.util.Set;
-import java.util.TreeMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.TimeoutException;
-import java.util.concurrent.atomic.LongAdder;
-import java.util.stream.Collectors;
import org.apache.rocketmq.acl.AccessValidator;
import org.apache.rocketmq.acl.plain.PlainAccessValidator;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ClientChannelInfo;
import org.apache.rocketmq.broker.client.ConsumerGroupInfo;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.filter.ConsumerFilterData;
import org.apache.rocketmq.broker.filter.ExpressionMessageFilter;
-import org.apache.rocketmq.broker.topic.TopicConfigManager;
import org.apache.rocketmq.broker.transaction.queue.TransactionalMessageUtil;
import org.apache.rocketmq.common.AclConfig;
+import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
import org.apache.rocketmq.common.TopicConfig;
-import org.apache.rocketmq.common.TopicQueueId;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.admin.ConsumeStats;
import org.apache.rocketmq.common.admin.OffsetWrapper;
@@ -80,29 +54,24 @@ import org.apache.rocketmq.common.protocol.body.Connection;
import org.apache.rocketmq.common.protocol.body.ConsumeQueueData;
import org.apache.rocketmq.common.protocol.body.ConsumeStatsList;
import org.apache.rocketmq.common.protocol.body.ConsumerConnection;
-import org.apache.rocketmq.common.protocol.body.CreateMessageQueueForLogicalQueueRequestBody;
import org.apache.rocketmq.common.protocol.body.GroupList;
import org.apache.rocketmq.common.protocol.body.KVTable;
import org.apache.rocketmq.common.protocol.body.LockBatchRequestBody;
import org.apache.rocketmq.common.protocol.body.LockBatchResponseBody;
-import org.apache.rocketmq.common.protocol.body.MigrateLogicalQueueBody;
import org.apache.rocketmq.common.protocol.body.ProducerConnection;
import org.apache.rocketmq.common.protocol.body.QueryConsumeQueueResponseBody;
import org.apache.rocketmq.common.protocol.body.QueryConsumeTimeSpanBody;
import org.apache.rocketmq.common.protocol.body.QueryCorrectionOffsetBody;
import org.apache.rocketmq.common.protocol.body.QueueTimeSpan;
-import org.apache.rocketmq.common.protocol.body.ReuseTopicLogicalQueueRequestBody;
-import org.apache.rocketmq.common.protocol.body.SealTopicLogicalQueueRequestBody;
+import org.apache.rocketmq.common.protocol.body.TopicConfigAndMappingSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.TopicList;
import org.apache.rocketmq.common.protocol.body.UnlockBatchRequestBody;
-import org.apache.rocketmq.common.protocol.body.UpdateTopicLogicalQueueMappingRequestBody;
import org.apache.rocketmq.common.protocol.header.CloneGroupOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.ConsumeMessageDirectlyResultRequestHeader;
import org.apache.rocketmq.common.protocol.header.CreateAccessConfigRequestHeader;
import org.apache.rocketmq.common.protocol.header.CreateTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteAccessConfigRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteSubscriptionGroupRequestHeader;
-import org.apache.rocketmq.common.protocol.header.DeleteTopicLogicalQueueRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.GetAllTopicConfigResponseHeader;
import org.apache.rocketmq.common.protocol.header.GetBrokerAclConfigResponseHeader;
@@ -127,7 +96,6 @@ import org.apache.rocketmq.common.protocol.header.QueryConsumeQueueRequestHeader
import org.apache.rocketmq.common.protocol.header.QueryConsumeTimeSpanRequestHeader;
import org.apache.rocketmq.common.protocol.header.QueryCorrectionOffsetHeader;
import org.apache.rocketmq.common.protocol.header.QueryTopicConsumeByWhoRequestHeader;
-import org.apache.rocketmq.common.protocol.header.QueryTopicLogicalQueueMappingRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResetOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResumeCheckHalfMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.SearchOffsetRequestHeader;
@@ -137,8 +105,16 @@ import org.apache.rocketmq.common.protocol.header.ViewBrokerStatsDataRequestHead
import org.apache.rocketmq.common.protocol.header.filtersrv.RegisterFilterServerRequestHeader;
import org.apache.rocketmq.common.protocol.header.filtersrv.RegisterFilterServerResponseHeader;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
+import org.apache.rocketmq.common.rpc.RpcClient;
+import org.apache.rocketmq.common.rpc.RpcClientUtils;
+import org.apache.rocketmq.common.rpc.RpcException;
+import org.apache.rocketmq.common.rpc.RpcRequest;
+import org.apache.rocketmq.common.rpc.RpcResponse;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
+import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
import org.apache.rocketmq.common.stats.StatsItem;
import org.apache.rocketmq.common.stats.StatsSnapshot;
import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
@@ -155,7 +131,6 @@ import org.apache.rocketmq.remoting.protocol.LanguageCode;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
import org.apache.rocketmq.remoting.protocol.RemotingSysResponseCode;
-import org.apache.rocketmq.srvutil.ConcurrentHashMapUtil;
import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.ConsumeQueueExt;
import org.apache.rocketmq.store.DefaultMessageStore;
@@ -166,12 +141,31 @@ import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
+import java.io.UnsupportedEncodingException;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentMap;
+
+import static org.apache.rocketmq.remoting.protocol.RemotingCommand.buildErrorResponse;
+
public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private final BrokerController brokerController;
+ private final RpcClient rpcClient;
+ private final BrokerConfig brokerConfig;
public AdminBrokerProcessor(final BrokerController brokerController) {
this.brokerController = brokerController;
+ this.brokerConfig = brokerController.getBrokerConfig();
+ this.rpcClient = brokerController.getBrokerOuterAPI().getRpcClient();
}
@Override
@@ -264,24 +258,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return getBrokerClusterAclConfig(ctx, request);
case RequestCode.GET_TOPIC_CONFIG:
return getTopicConfig(ctx, request);
- case RequestCode.UPDATE_TOPIC_LOGICAL_QUEUE_MAPPING:
- return updateTopicLogicalQueueMapping(ctx, request);
- case RequestCode.DELETE_TOPIC_LOGICAL_QUEUE_MAPPING:
- return deleteTopicLogicalQueueMapping(ctx, request);
- case RequestCode.QUERY_TOPIC_LOGICAL_QUEUE_MAPPING:
- return queryTopicLogicalQueueMapping(ctx, request);
- case RequestCode.SEAL_TOPIC_LOGICAL_QUEUE:
- return sealTopicLogicalQueue(ctx, request);
- case RequestCode.REUSE_TOPIC_LOGICAL_QUEUE:
- return reuseTopicLogicalQueue(ctx, request);
- case RequestCode.CREATE_MESSAGE_QUEUE_FOR_LOGICAL_QUEUE:
- return createMessageQueueForLogicalQueue(ctx, request);
- case RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_PREPARE:
- return migrateTopicLogicalQueuePrepare(ctx, request);
- case RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_COMMIT:
- return migrateTopicLogicalQueueCommit(ctx, request);
- case RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_NOTIFY:
- return migrateTopicLogicalQueueNotify(ctx, request);
+ case RequestCode.UPDATE_AND_CREATE_STATIC_TOPIC:
+ return this.updateAndCreateStaticTopic(ctx, request);
default:
return getUnknownCmdResponse(ctx, request);
}
@@ -323,6 +301,51 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+ private synchronized RemotingCommand updateAndCreateStaticTopic(ChannelHandlerContext ctx,
+ RemotingCommand request) throws RemotingCommandException {
+ final RemotingCommand response = RemotingCommand.createResponseCommand(null);
+ final CreateTopicRequestHeader requestHeader =
+ (CreateTopicRequestHeader) request.decodeCommandCustomHeader(CreateTopicRequestHeader.class);
+ log.info("updateAndCreateTopic called by {}", RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
+
+ final TopicQueueMappingDetail topicQueueMappingDetail = RemotingSerializable.decode(request.getBody(), TopicQueueMappingDetail.class);
+
+ String topic = requestHeader.getTopic();
+
+ if (!TopicValidator.validateTopic(topic, response)) {
+ return response;
+ }
+ if (TopicValidator.isSystemTopic(topic, response)) {
+ return response;
+ }
+ boolean force = false;
+ if (requestHeader.getForce() != null && requestHeader.getForce()) {
+ force = true;
+ }
+
+ TopicConfig topicConfig = new TopicConfig(topic);
+ topicConfig.setReadQueueNums(requestHeader.getReadQueueNums());
+ topicConfig.setWriteQueueNums(requestHeader.getWriteQueueNums());
+ topicConfig.setTopicFilterType(requestHeader.getTopicFilterTypeEnum());
+ topicConfig.setPerm(requestHeader.getPerm());
+ topicConfig.setTopicSysFlag(requestHeader.getTopicSysFlag() == null ? 0 : requestHeader.getTopicSysFlag());
+
+ try {
+ this.brokerController.getTopicConfigManager().updateTopicConfig(topicConfig);
+
+ this.brokerController.getTopicQueueMappingManager().updateTopicQueueMapping(topicQueueMappingDetail, force, false, true);
+
+ this.brokerController.registerIncrementBrokerData(topicConfig, this.brokerController.getTopicConfigManager().getDataVersion());
+ response.setCode(ResponseCode.SUCCESS);
+ } catch (Exception e) {
+ log.error("Update static topic failed for [{}]", request, e);
+ response.setCode(ResponseCode.SYSTEM_ERROR);
+ response.setRemark(e.getMessage());
+ }
+ return response;
+ }
+
+
private synchronized RemotingCommand deleteTopic(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
@@ -340,12 +363,14 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
}
this.brokerController.getTopicConfigManager().deleteTopicConfig(topic);
+ this.brokerController.getTopicQueueMappingManager().delete(topic);
+
this.brokerController.getMessageStore()
.cleanUnusedTopic(this.brokerController.getTopicConfigManager().getTopicConfigTable().keySet());
if (this.brokerController.getBrokerConfig().isAutoDeleteUnusedStats()) {
this.brokerController.getBrokerStatsManager().onTopicDeleted(requestHeader.getTopic());
}
- this.brokerController.getTopicConfigManager().deleteQueueRouteData(requestHeader.getTopic());
+
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
@@ -518,7 +543,16 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
// final GetAllTopicConfigResponseHeader responseHeader =
// (GetAllTopicConfigResponseHeader) response.readCustomHeader();
- String content = this.brokerController.getTopicConfigManager().encode();
+ TopicConfigAndMappingSerializeWrapper topicConfigAndMappingSerializeWrapper = new TopicConfigAndMappingSerializeWrapper();
+
+ topicConfigAndMappingSerializeWrapper.setDataVersion(this.brokerController.getTopicConfigManager().getDataVersion());
+ topicConfigAndMappingSerializeWrapper.setTopicConfigTable(this.brokerController.getTopicConfigManager().getTopicConfigTable());
+
+ topicConfigAndMappingSerializeWrapper.setMappingDataVersion(this.brokerController.getTopicQueueMappingManager().getDataVersion());
+ topicConfigAndMappingSerializeWrapper.setTopicQueueMappingDetailMap(this.brokerController.getTopicQueueMappingManager().getTopicQueueMappingTable());
+
+
+ String content = topicConfigAndMappingSerializeWrapper.toJson();
if (content != null && content.length() > 0) {
try {
response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
@@ -603,6 +637,62 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+ private RemotingCommand rewriteRequestForStaticTopic(SearchOffsetRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ List<LogicQueueMappingItem> mappingItems = mappingContext.getMappingItemList();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname()));
+ }
+ //TO DO should make sure the timestampOfOffset is equal or bigger than the searched timestamp
+ Long timestamp = requestHeader.getTimestamp();
+ long offset = -1;
+ for (int i = 0; i < mappingItems.size(); i++) {
+ LogicQueueMappingItem item = mappingItems.get(i);
+ if (!item.checkIfLogicoffsetDecided()) {
+ continue;
+ }
+ if (mappingDetail.getBname().equals(item.getBname())) {
+ offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(mappingContext.getTopic(), item.getQueueId(), timestamp);
+ if (offset > 0) {
+ offset = item.computeStaticQueueOffsetStrictly(offset);
+ break;
+ }
+ } else {
+ requestHeader.setLo(false);
+ requestHeader.setTimestamp(timestamp);
+ requestHeader.setQueueId(item.getQueueId());
+ requestHeader.setBname(item.getBname());
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.SEARCH_OFFSET_BY_TIMESTAMP, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ SearchOffsetResponseHeader offsetResponseHeader = (SearchOffsetResponseHeader) rpcResponse.getHeader();
+ if (offsetResponseHeader.getOffset() < 0
+ || (item.checkIfEndOffsetDecided() && offsetResponseHeader.getOffset() >= item.getEndOffset())) {
+ continue;
+ } else {
+ offset = item.computeStaticQueueOffsetStrictly(offsetResponseHeader.getOffset());
+ }
+
+ }
+ }
+ final RemotingCommand response = RemotingCommand.createResponseCommand(SearchOffsetResponseHeader.class);
+ final SearchOffsetResponseHeader responseHeader = (SearchOffsetResponseHeader) response.readCustomHeader();
+ responseHeader.setOffset(offset);
+ response.setCode(ResponseCode.SUCCESS);
+ response.setRemark(null);
+ return response;
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
private RemotingCommand searchOffsetByTimestamp(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(SearchOffsetResponseHeader.class);
@@ -610,6 +700,13 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
final SearchOffsetRequestHeader requestHeader =
(SearchOffsetRequestHeader) request.decodeCommandCustomHeader(SearchOffsetRequestHeader.class);
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader);
+
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+
long offset = this.brokerController.getMessageStore().getOffsetInQueueByTime(requestHeader.getTopic(), requestHeader.getQueueId(),
requestHeader.getTimestamp());
@@ -620,6 +717,50 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+ private RemotingCommand rewriteRequestForStaticTopic(GetMaxOffsetRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ LogicQueueMappingItem mappingItem = mappingContext.getLeaderItem();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname()));
+ }
+
+ try {
+ LogicQueueMappingItem maxItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), Long.MAX_VALUE, true);
+ assert maxItem != null;
+ assert maxItem.getLogicOffset() >= 0;
+ requestHeader.setBname(maxItem.getBname());
+ requestHeader.setLo(false);
+ requestHeader.setQueueId(mappingItem.getQueueId());
+
+ long maxPhysicalOffset = Long.MAX_VALUE;
+ if (maxItem.getBname().equals(mappingDetail.getBname())) {
+ //current broker
+ maxPhysicalOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(mappingContext.getTopic(), mappingItem.getQueueId());
+ } else {
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.GET_MAX_OFFSET, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ GetMaxOffsetResponseHeader offsetResponseHeader = (GetMaxOffsetResponseHeader) rpcResponse.getHeader();
+ maxPhysicalOffset = offsetResponseHeader.getOffset();
+ }
+
+ final RemotingCommand response = RemotingCommand.createResponseCommand(GetMaxOffsetResponseHeader.class);
+ final GetMaxOffsetResponseHeader responseHeader = (GetMaxOffsetResponseHeader) response.readCustomHeader();
+ responseHeader.setOffset(maxItem.computeStaticQueueOffsetStrictly(maxPhysicalOffset));
+ response.setCode(ResponseCode.SUCCESS);
+ response.setRemark(null);
+ return response;
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
private RemotingCommand getMaxOffset(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(GetMaxOffsetResponseHeader.class);
@@ -627,33 +768,13 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
final GetMaxOffsetRequestHeader requestHeader =
(GetMaxOffsetRequestHeader) request.decodeCommandCustomHeader(GetMaxOffsetRequestHeader.class);
- String topic = requestHeader.getTopic();
- int queueId = requestHeader.getQueueId();
-
- if (requestHeader.getLogicalQueue()) {
- LogicalQueuesInfoInBroker logicalQueuesInfo = this.brokerController.getTopicConfigManager().selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo != null) {
- // max offset must be in the queue route with largest offset
- LogicalQueueRouteData requestLogicalQueueRouteData = logicalQueuesInfo.queryQueueRouteDataByQueueId(queueId, Long.MAX_VALUE);
- if (requestLogicalQueueRouteData != null) {
- logicalQueuesInfo.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(requestLogicalQueueRouteData.getLogicalQueueIndex());
- if (queueRouteDataList != null && !queueRouteDataList.isEmpty()) {
- LogicalQueueRouteData selectedLogicalQueueRouteData = queueRouteDataList.get(queueRouteDataList.size() - 1);
- if (!Objects.equals(selectedLogicalQueueRouteData.getMessageQueue(), new MessageQueue(topic, this.brokerController.getBrokerConfig().getBrokerName(), queueId))) {
- log.info("getMaxOffset topic={} queueId={} not latest, redirect: {}", topic, queueId, selectedLogicalQueueRouteData);
- response.addExtField(MessageConst.PROPERTY_REDIRECT, "1");
- }
- }
- } finally {
- logicalQueuesInfo.readLock().unlock();
- }
- }
- }
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader);
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
}
- long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId, requestHeader.isCommitted());
+ long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId());
responseHeader.setOffset(offset);
@@ -662,19 +783,105 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+ private CompletableFuture<RpcResponse> handleGetMinOffsetForStaticTopic(RpcRequest request, TopicQueueMappingContext mappingContext) {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ if (!mappingContext.isLeader()) {
+ //this may not
+ return CompletableFuture.completedFuture(new RpcResponse(new RpcException(ResponseCode.NOT_LEADER_FOR_QUEUE,
+ String.format("%s-%d is not leader in broker %s, request code %d", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname(), request.getCode()))));
+ }
+ GetMinOffsetRequestHeader requestHeader = (GetMinOffsetRequestHeader) request.getHeader();
+ LogicQueueMappingItem mappingItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), 0L, true);
+ assert mappingItem != null;
+ try {
+ requestHeader.setBname(mappingItem.getBname());
+ requestHeader.setLo(false);
+ requestHeader.setQueueId(mappingItem.getQueueId());
+ long physicalOffset;
+ //run in local
+ if (mappingItem.getBname().equals(mappingDetail.getBname())) {
+ physicalOffset = this.brokerController.getMessageStore().getMinOffsetInQueue(mappingDetail.getTopic(), mappingItem.getQueueId());
+ } else {
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.GET_MIN_OFFSET, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ GetMinOffsetResponseHeader offsetResponseHeader = (GetMinOffsetResponseHeader) rpcResponse.getHeader();
+ physicalOffset = offsetResponseHeader.getOffset();
+ }
+ long offset = mappingItem.computeStaticQueueOffsetLoosely(physicalOffset);
+
+ final GetMinOffsetResponseHeader responseHeader = new GetMinOffsetResponseHeader();
+ responseHeader.setOffset(offset);
+ return CompletableFuture.completedFuture(new RpcResponse(ResponseCode.SUCCESS, responseHeader, null));
+ } catch (Throwable t) {
+ log.error("rewriteRequestForStaticTopic failed", t);
+ return CompletableFuture.completedFuture(new RpcResponse(new RpcException(ResponseCode.SYSTEM_ERROR, t.getMessage(), t)));
+ }
+ }
+
+ private CompletableFuture<RpcResponse> handleGetMinOffset(RpcRequest request) {
+ assert request.getCode() == RequestCode.GET_MIN_OFFSET;
+ GetMinOffsetRequestHeader requestHeader = (GetMinOffsetRequestHeader) request.getHeader();
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader, false);
+ CompletableFuture<RpcResponse> rewriteResult = handleGetMinOffsetForStaticTopic(request, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+ final GetMinOffsetResponseHeader responseHeader = new GetMinOffsetResponseHeader();
+ long offset = this.brokerController.getMessageStore().getMinOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId());
+ responseHeader.setOffset(offset);
+ return CompletableFuture.completedFuture(new RpcResponse(ResponseCode.SUCCESS, responseHeader, null));
+ }
+
private RemotingCommand getMinOffset(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
- final RemotingCommand response = RemotingCommand.createResponseCommand(GetMinOffsetResponseHeader.class);
- final GetMinOffsetResponseHeader responseHeader = (GetMinOffsetResponseHeader) response.readCustomHeader();
final GetMinOffsetRequestHeader requestHeader =
(GetMinOffsetRequestHeader) request.decodeCommandCustomHeader(GetMinOffsetRequestHeader.class);
+ try {
+ CompletableFuture<RpcResponse> responseFuture = handleGetMinOffset(new RpcRequest(RequestCode.GET_MIN_OFFSET, requestHeader, null));
+ RpcResponse rpcResponse = responseFuture.get();
+ return RpcClientUtils.createCommandForRpcResponse(rpcResponse);
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
- long offset = this.brokerController.getMessageStore().getMinOffsetInQueue(requestHeader.getTopic(), requestHeader.getQueueId());
+ private RemotingCommand rewriteRequestForStaticTopic(GetEarliestMsgStoretimeRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
- responseHeader.setOffset(offset);
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark(null);
- return response;
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname()));
+ }
+ LogicQueueMappingItem mappingItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), 0L, true);
+ assert mappingItem != null;
+ try {
+ requestHeader.setBname(mappingItem.getBname());
+ requestHeader.setLo(false);
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.GET_EARLIEST_MSG_STORETIME, requestHeader, null);
+ //TO DO check if it is in current broker
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ GetEarliestMsgStoretimeResponseHeader offsetResponseHeader = (GetEarliestMsgStoretimeResponseHeader) rpcResponse.getHeader();
+
+ final RemotingCommand response = RemotingCommand.createResponseCommand(GetEarliestMsgStoretimeResponseHeader.class);
+ final GetEarliestMsgStoretimeResponseHeader responseHeader = (GetEarliestMsgStoretimeResponseHeader) response.readCustomHeader();
+ responseHeader.setTimestamp(offsetResponseHeader.getTimestamp());
+ response.setCode(ResponseCode.SUCCESS);
+ response.setRemark(null);
+ return response;
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
}
private RemotingCommand getEarliestMsgStoretime(ChannelHandlerContext ctx,
@@ -684,6 +891,12 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
final GetEarliestMsgStoretimeRequestHeader requestHeader =
(GetEarliestMsgStoretimeRequestHeader) request.decodeCommandCustomHeader(GetEarliestMsgStoretimeRequestHeader.class);
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader, false);
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+
long timestamp =
this.brokerController.getMessageStore().getEarliestMessageTime(requestHeader.getTopic(), requestHeader.getQueueId());
@@ -806,6 +1019,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+
+
private RemotingCommand getTopicStatsInfo(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
@@ -819,8 +1034,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
response.setRemark("topic[" + topic + "] not exist");
return response;
}
-
TopicStatsTable topicStatsTable = new TopicStatsTable();
+
for (int i = 0; i < topicConfig.getWriteQueueNums(); i++) {
MessageQueue mq = new MessageQueue();
mq.setTopic(topic);
@@ -951,6 +1166,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
continue;
}
+ TopicQueueMappingDetail mappingDetail = this.brokerController.getTopicQueueMappingManager().getTopicQueueMapping(topic);
+
{
SubscriptionData findSubscriptionData =
this.brokerController.getConsumerManager().findSubscriptionData(requestHeader.getConsumerGroup(), topic);
@@ -978,17 +1195,26 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
requestHeader.getConsumerGroup(),
topic,
i);
- if (consumerOffset < 0)
- consumerOffset = 0;
+ // the consumerOffset cannot be zero for static topic because of the "double read check" strategy
+ // just remain the logic for dynamic topic
+ // maybe we should remove it in the future
+ if (mappingDetail == null) {
+ if (consumerOffset < 0)
+ consumerOffset = 0;
+ }
offsetWrapper.setBrokerOffset(brokerOffset);
offsetWrapper.setConsumerOffset(consumerOffset);
- long timeOffset = consumerOffset - 1;
- if (timeOffset >= 0) {
- long lastTimestamp = this.brokerController.getMessageStore().getMessageStoreTimeStamp(topic, i, timeOffset);
- if (lastTimestamp > 0) {
- offsetWrapper.setLastTimestamp(lastTimestamp);
+ // the consumeOffset is not in this broker for static topic
+ // and may get the wrong result
+ if (mappingDetail == null) {
+ long timeOffset = consumerOffset - 1;
+ if (timeOffset >= 0) {
+ long lastTimestamp = this.brokerController.getMessageStore().getMessageStoreTimeStamp(topic, i, timeOffset);
+ if (lastTimestamp > 0) {
+ offsetWrapper.setLastTimestamp(lastTimestamp);
+ }
}
}
@@ -1711,11 +1937,16 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
TopicConfig topicConfig = this.brokerController.getTopicConfigManager().getTopicConfigTable().get(requestHeader.getTopic());
if (topicConfig == null) {
log.error("No topic in this broker, client: {} topic: {}", ctx.channel().remoteAddress(), requestHeader.getTopic());
- response.setCode(ResponseCode.SYSTEM_ERROR);
+ //be care of the response code, should set "not-exist" explictly
+ response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark("No topic in this broker. topic: " + requestHeader.getTopic());
return response;
}
- String content = JSONObject.toJSONString(topicConfig);
+ TopicQueueMappingDetail topicQueueMappingDetail = null;
+ if (Boolean.TRUE.equals(requestHeader.getLo())) {
+ topicQueueMappingDetail = this.brokerController.getTopicQueueMappingManager().getTopicQueueMapping(requestHeader.getTopic());
+ }
+ String content = JSONObject.toJSONString(new TopicConfigAndQueueMapping(topicConfig, topicQueueMappingDetail));
try {
response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
} catch (UnsupportedEncodingException e) {
@@ -1730,588 +1961,4 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return response;
}
-
- private RemotingCommand updateTopicLogicalQueueMapping(ChannelHandlerContext ctx,
- RemotingCommand request) throws RemotingCommandException {
- UpdateTopicLogicalQueueMappingRequestBody requestBody = RemotingSerializable.decode(request.getBody(), UpdateTopicLogicalQueueMappingRequestBody.class);
- RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SYSTEM_ERROR);
- response.setRemark("unknown error");
- if (requestBody == null) {
- response.setRemark("decode null");
- return response;
- }
- String topic = requestBody.getTopic();
- int queueId = requestBody.getQueueId();
- String brokerName = this.brokerController.getBrokerConfig().getBrokerName();
- int logicalQueueIdx = requestBody.getLogicalQueueIdx();
- log.info("updateTopicLogicalQueueMapping topic={} queueId={} logicalQueueIndex={}", topic, queueId, logicalQueueIdx);
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- TopicConfig topicConfig = topicConfigManager.selectTopicConfig(topic);
- if (topicConfig == null) {
- log.warn("updateTopicLogicalQueueMapping topic={} queueId={} logicalQueueIndex={} topic not exist", topic, queueId, logicalQueueIdx);
- response.setRemark("topic not exist");
- return response;
- }
-
- LogicalQueuesInfoInBroker logicalQueuesInfo;
- LogicalQueueRouteData newQueueRouteData = new LogicalQueueRouteData();
- newQueueRouteData.setBrokerAddr(this.brokerController.getBrokerAddr());
- newQueueRouteData.setMessageQueue(new MessageQueue(topic, brokerName, queueId));
- if (logicalQueueIdx >= 0) {
- // add logical queue
- newQueueRouteData.setLogicalQueueIndex(logicalQueueIdx);
- newQueueRouteData.setLogicalQueueDelta(0L);
- newQueueRouteData.setState(MessageQueueRouteState.Normal);
- logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- logicalQueuesInfo.writeLock().lock();
- try {
- // verify whether this message queue is already set up
- for (List<LogicalQueueRouteData> queueRouteDataList : logicalQueuesInfo.values()) {
- for (Iterator<LogicalQueueRouteData> iterator = queueRouteDataList.iterator(); iterator.hasNext(); ) {
- LogicalQueueRouteData queueRouteData = iterator.next();
- if (Objects.equals(queueRouteData.getMessageQueue(), newQueueRouteData.getMessageQueue())) {
- if (queueRouteData.getLogicalQueueIndex() == logicalQueueIdx) {
- log.info("updateTopicLogicalQueueMapping topic={} queueId={} logicalQueueIndex={} already set up", topic, queueId, logicalQueueIdx);
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark("already set up");
- return response;
- } else {
- log.warn("updateTopicLogicalQueueMapping topic={} queueId={} already assigned logicalQueueIdx={}, will reassign as logicalQueueIdx={}", topic, queueRouteData.getMessageQueue(), queueRouteData.getLogicalQueueIndex(), newQueueRouteData.getLogicalQueueIndex());
- iterator.remove();
- break;
- }
- }
- }
- }
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.computeIfAbsent(logicalQueueIdx, ignore -> Lists.newArrayListWithExpectedSize(1));
- int idx = Collections.binarySearch(queueRouteDataList, newQueueRouteData, Comparator.comparingLong(LogicalQueueRouteData::getLogicalQueueDelta).thenComparingInt(LogicalQueueRouteData::getStateOrdinal));
- if (idx >= 0) {
- log.warn("updateTopicLogicalQueueMapping topic={} queueId={} logicalQueueIdx={} found same logicalQueueOffset and will replace, exist {}, new {}", topic, queueId, logicalQueueIdx, queueRouteDataList.get(idx), newQueueRouteData);
- queueRouteDataList.set(idx, newQueueRouteData);
- response.setCode(ResponseCode.SYSTEM_ERROR);
- response.setRemark("duplicate logicalQueueOffset found");
- } else {
- idx = -idx - 1;
- queueRouteDataList.add(idx, newQueueRouteData);
- logicalQueuesInfo.updateQueueRouteDataByQueueId(newQueueRouteData.getQueueId(), newQueueRouteData);
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark("set up");
- log.info("updateTopicLogicalQueueMapping topic={} queueId={} logicalQueueIdx={} added as #{}", topic, queueId, logicalQueueIdx, idx);
- }
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
- } else {
- // delete logical queue
- logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- logicalQueuesInfo.writeLock().lock();
- try {
- for (List<LogicalQueueRouteData> queueRouteDataList : logicalQueuesInfo.values()) {
- queueRouteDataList.removeIf(queueRouteData -> Objects.equals(queueRouteData.getMessageQueue(), newQueueRouteData.getMessageQueue()));
- }
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
- logicalQueuesInfo.updateQueueRouteDataByQueueId(newQueueRouteData.getQueueId(), null);
- this.brokerController.getSendMessageProcessor().getInFlyWritingCounterMap().remove(new TopicQueueId(topic, queueId));
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark("deleted");
- log.info("updateTopicLogicalQueueMapping topic={} queueId={} deleted as logical queue", topic, queueId, logicalQueueIdx);
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfig, topicConfigManager.getDataVersion());
-
- return response;
- }
-
- private RemotingCommand deleteTopicLogicalQueueMapping(ChannelHandlerContext ctx,
- RemotingCommand request) throws RemotingCommandException {
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- DeleteTopicLogicalQueueRequestHeader requestHeader =
- (DeleteTopicLogicalQueueRequestHeader) request.decodeCommandCustomHeader(DeleteTopicLogicalQueueRequestHeader.class);
- String topic = requestHeader.getTopic();
- log.info("deleteTopicLogicalQueueMapping topic={}", topic);
-
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark("already deleted");
- log.info("deleteTopicLogicalQueueMapping topic={} already deleted", topic);
- return response;
- }
- String brokerName = this.brokerController.getBrokerConfig().getBrokerName();
- long size = logicalQueuesInfo.values().stream().flatMap(Collection::stream).filter(v -> Objects.equals(v.getBrokerName(), brokerName)).count();
- if (size > 0) {
- response.setCode(ResponseCode.SYSTEM_ERROR);
- response.setRemark(String.format(Locale.ENGLISH, "still %d message queues", size));
- log.info("deleteTopicLogicalQueueMapping topic={} still {} message queues", topic, size);
- return response;
- }
- topicConfigManager.deleteQueueRouteData(topic);
- response.setCode(ResponseCode.SUCCESS);
- response.setRemark("deleted");
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- log.info("deleteTopicLogicalQueueMapping topic={} deleted", topic);
- return response;
- }
-
- private RemotingCommand queryTopicLogicalQueueMapping(ChannelHandlerContext ctx,
- RemotingCommand request) throws RemotingCommandException {
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- QueryTopicLogicalQueueMappingRequestHeader requestHeader =
- (QueryTopicLogicalQueueMappingRequestHeader) request.decodeCommandCustomHeader(QueryTopicLogicalQueueMappingRequestHeader.class);
- String topic = requestHeader.getTopic();
- log.info("queryTopicLogicalQueueMapping topic={}", topic);
- LogicalQueuesInfoInBroker logicalQueuesInfo = brokerController.getTopicConfigManager().selectLogicalQueuesInfo(topic);
- TreeMap<Integer, List<LogicalQueueRouteData>> result = null;
- if (logicalQueuesInfo != null) {
- result = Maps.newTreeMap();
- logicalQueuesInfo.readLock().lock();
- try {
- for (Map.Entry<Integer, List<LogicalQueueRouteData>> entry : logicalQueuesInfo.entrySet()) {
- Integer k = entry.getKey();
- List<LogicalQueueRouteData> v = entry.getValue();
- result.put(k, ImmutableList.copyOf(v));
- }
- } finally {
- logicalQueuesInfo.readLock().unlock();
- }
- }
- response.setCode(ResponseCode.SUCCESS);
- response.setBody(RemotingSerializable.encode(result));
- return response;
- }
-
- private void sealLogicalQueueRouteData(LogicalQueueRouteData queueRouteData,
- MessageStore messageStore) throws TimeoutException, InterruptedException {
- queueRouteData.setState(MessageQueueRouteState.ReadOnly);
-
- String topic = queueRouteData.getTopic();
- int queueId = queueRouteData.getQueueId();
-
- // busy wait for all in-fly messages to be finished
- TopicQueueId key = new TopicQueueId(topic, queueId);
- long startTime = System.currentTimeMillis();
- while (true) {
- LongAdder counter = this.brokerController.getSendMessageProcessor().getInFlyWritingCounterMap().get(key);
- if (counter == null || counter.sum() == 0) {
- break;
- }
- if (System.currentTimeMillis() - startTime > 10_000) {
- throw new TimeoutException();
- }
- Thread.sleep(100);
- }
- // busy wait for all CQ to be finished
- while (messageStore.getMaxOffsetInQueue(topic, queueId, true) != messageStore.getMaxOffsetInQueue(topic, queueId, false)) {
- if (System.currentTimeMillis() - startTime > 10_000) {
- throw new TimeoutException();
- }
- Thread.sleep(100);
- }
-
- long firstMsgQueueOffset = messageStore.getMinOffsetInQueue(topic, queueId);
- long lastMsgQueueOffset = messageStore.getMaxOffsetInQueue(topic, queueId, false) - 1;
- long firstMsgTimeMillis = 0L;
- long lastMsgTimeMillis = 0L;
- boolean expired = false;
- if (firstMsgQueueOffset == lastMsgQueueOffset) {
- // no message at all
- expired = true;
- } else {
- long minPhyOffset = messageStore.getMinPhyOffset();
- long lastMsgCommitLogOffset = lastMsgQueueOffset >= 0 ? messageStore.getCommitLogOffsetInQueue(topic, queueId, lastMsgQueueOffset) : -1;
- if (lastMsgCommitLogOffset < minPhyOffset) {
- // commitLog already cleaned
- expired = true;
- } else {
- long firstMsgCommitLogOffset = firstMsgQueueOffset >= 0 ? messageStore.getCommitLogOffsetInQueue(topic, queueId, firstMsgQueueOffset) : -1;
- MessageExt firstMsg = firstMsgCommitLogOffset >= 0 ? messageStore.lookMessageByOffset(firstMsgCommitLogOffset) : null;
- firstMsgTimeMillis = firstMsg != null ? firstMsg.getStoreTimestamp() : 0L;
-
- MessageExt lastMsg = lastMsgCommitLogOffset >= 0 ? messageStore.lookMessageByOffset(lastMsgCommitLogOffset) : null;
- lastMsgTimeMillis = lastMsg != null ? lastMsg.getStoreTimestamp() : 0L;
- }
- }
-
- queueRouteData.setOffsetMax(lastMsgQueueOffset + 1);
- queueRouteData.setFirstMsgTimeMillis(firstMsgTimeMillis);
- queueRouteData.setLastMsgTimeMillis(lastMsgTimeMillis);
-
- if (expired) {
- queueRouteData.setState(MessageQueueRouteState.Expired);
- }
- }
-
- private RemotingCommand sealTopicLogicalQueue(ChannelHandlerContext ctx, RemotingCommand request) {
- SealTopicLogicalQueueRequestBody requestBody = RemotingSerializable.decode(request.getBody(), SealTopicLogicalQueueRequestBody.class);
- if (requestBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- String topic = requestBody.getTopic();
- int queueId = requestBody.getQueueId();
- int logicalQueueIdx = requestBody.getLogicalQueueIndex();
- log.info("sealTopicLogicalQueue topic={} queueId={}", topic, queueId);
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- TopicConfig topicConfig = topicConfigManager.selectTopicConfig(topic);
- if (topicConfig == null) {
- log.warn("sealTopicLogicalQueue topic={} queueId={} topic not exist", topic, queueId);
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not exist");
- }
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
-
- LogicalQueueRouteData resultQueueRouteData;
-
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- logicalQueuesInfo.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(logicalQueueIdx);
- if (queueRouteDataList == null) {
- log.info("sealTopicLogicalQueue topic={} queueId={} logicalQueueIdx={} not exist", topic, queueId, logicalQueueIdx);
- response.setRemark("logical queue not exist");
- return response;
- }
- List<LogicalQueueRouteData> foundQueues = queueRouteDataList.stream()
- .filter(queueRouteData -> queueId == queueRouteData.getQueueId()).collect(Collectors.toList());
- if (foundQueues.isEmpty()) {
- log.info("sealTopicLogicalQueue topic={} queueId={} logicalQueueIdx={} queueId={} not exist", topic, queueId, logicalQueueIdx, queueId);
- response.setRemark("message queue not exist");
- return response;
- }
- Optional<LogicalQueueRouteData> firstMainQueueRouteDataOptional = foundQueues.stream().filter(LogicalQueueRouteData::isWritable).findFirst();
- if (!firstMainQueueRouteDataOptional.isPresent()) {
- log.info("sealTopicLogicalQueue topic={} queueId={} logicalQueueIdx={} queueId={} already sealed", topic, queueId, logicalQueueIdx, queueId);
- response.setRemark("message queue already sealed");
- return response;
- }
- resultQueueRouteData = firstMainQueueRouteDataOptional.get();
- } finally {
- logicalQueuesInfo.readLock().unlock();
- }
- try {
- sealLogicalQueueRouteData(resultQueueRouteData, brokerController.getMessageStore());
- } catch (InterruptedException e) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "thread interrupted");
- } catch (TimeoutException e) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "seal timeout");
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- response.setBody(RemotingSerializable.encode(resultQueueRouteData));
- return response;
- }
-
- private RemotingCommand reuseTopicLogicalQueue(ChannelHandlerContext ctx, RemotingCommand request) {
- ReuseTopicLogicalQueueRequestBody requestBody = RemotingSerializable.decode(request.getBody(), ReuseTopicLogicalQueueRequestBody.class);
- if (requestBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- String topic = requestBody.getTopic();
- int queueId = requestBody.getQueueId();
- String brokerName = this.brokerController.getBrokerConfig().getBrokerName();
- MessageQueue mq = new MessageQueue(topic, brokerName, queueId);
- int logicalQueueIdx = requestBody.getLogicalQueueIndex();
- MessageQueueRouteState messageQueueRouteState = requestBody.getMessageQueueRouteState();
- log.info("reuseTopicLogicalQueue topic={} queueId={} logicalQueueIdx={} messageQueueRouteState={}", topic, queueId, logicalQueueIdx, messageQueueRouteState);
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "queue route data not found");
- }
- if (queueId >= topicConfigManager.selectTopicConfig(topic).getWriteQueueNums()) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "found no queue");
- }
- logicalQueuesInfo.writeLock().lock();
- LogicalQueueRouteData queueRouteData = new LogicalQueueRouteData(
- logicalQueueIdx,
- MessageQueueRouteState.WriteOnly.equals(messageQueueRouteState) ? -1 : 0,
- new MessageQueue(topic, brokerName, queueId),
- messageQueueRouteState,
- this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId, false),
- -1,
- -1,
- -1,
- this.brokerController.getBrokerAddr()
- );
- try {
- if (logicalQueuesInfo.values().stream().flatMap(Collection::stream).filter(v -> Objects.equals(v.getMessageQueue(), mq)).anyMatch(LogicalQueueRouteData::isWritable)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "queue writable");
- }
- logicalQueuesInfo.computeIfAbsent(logicalQueueIdx, ignore -> Lists.newArrayListWithExpectedSize(1)).add(queueRouteData);
- logicalQueuesInfo.updateQueueRouteDataByQueueId(queueId, queueRouteData);
- this.brokerController.getSendMessageProcessor().getInFlyWritingCounterMap().remove(new TopicQueueId(topic, queueId));
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
- response.setBody(RemotingSerializable.encode(queueRouteData));
- return response;
- }
-
- private RemotingCommand createMessageQueueForLogicalQueue(ChannelHandlerContext ctx, RemotingCommand request) {
- CreateMessageQueueForLogicalQueueRequestBody requestBody = RemotingSerializable.decode(request.getBody(), CreateMessageQueueForLogicalQueueRequestBody.class);
- if (requestBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- String topic = requestBody.getTopic();
- int logicalQueueIdx = requestBody.getLogicalQueueIndex();
- MessageQueueRouteState messageQueueStatus = requestBody.getMessageQueueStatus();
- log.info("createMessageQueueForLogicalQueue topic={} logicalQueueIdx={}", topic, logicalQueueIdx);
-
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo;
- LogicalQueueRouteData queueRouteData;
- TopicConfig topicConfig;
- while (true) {
- topicConfig = topicConfigManager.selectTopicConfig(topic);
- if (topicConfig == null || topicConfig.getWriteQueueNums() == 0) {
- // create topic if not exist
- topicConfig = ConcurrentHashMapUtil.computeIfAbsent(topicConfigManager.getTopicConfigTable(), topic, s -> new TopicConfig(topic, 0, 0, this.brokerController.getBrokerConfig().getBrokerPermission()));
- logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- } else {
- logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- }
- if (topicConfig.getWriteQueueNums() > 0 && logicalQueuesInfo == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not enable logical queue");
- }
- TopicConfig newTopicConfig = new TopicConfig(topicConfig);
- newTopicConfig.setWriteQueueNums(newTopicConfig.getWriteQueueNums() + 1);
- newTopicConfig.setReadQueueNums(newTopicConfig.getReadQueueNums() + 1);
-
- int queueId = newTopicConfig.getWriteQueueNums() - 1;
- queueRouteData = new LogicalQueueRouteData();
- queueRouteData.setLogicalQueueIndex(logicalQueueIdx);
- queueRouteData.setBrokerAddr(this.brokerController.getBrokerAddr());
- queueRouteData.setMessageQueue(new MessageQueue(topic, this.brokerController.getBrokerConfig().getBrokerName(), queueId));
- if (messageQueueStatus != null) {
- queueRouteData.setState(messageQueueStatus);
- switch (messageQueueStatus) {
- case WriteOnly:
- case Expired:
- queueRouteData.setLogicalQueueDelta(-1L);
- break;
- default:
- queueRouteData.setLogicalQueueDelta(0L);
- }
- }
-
- logicalQueuesInfo.writeLock().lock();
- try {
- List<LogicalQueueRouteData> l = logicalQueuesInfo.computeIfAbsent(logicalQueueIdx, i -> Lists.newArrayListWithExpectedSize(1));
- if (MessageQueueRouteState.WriteOnly.equals(messageQueueStatus) && l.stream().anyMatch(LogicalQueueRouteData::isWriteOnly)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "multiple WriteOnly queue");
- } else if (MessageQueueRouteState.Normal.equals(messageQueueStatus) && l.stream().anyMatch(LogicalQueueRouteData::isWritable)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "multiple writable queue");
- }
- if (topicConfigManager.replaceTopicConfig(topic, topicConfig, newTopicConfig)) {
- l.add(queueRouteData);
- logicalQueuesInfo.updateQueueRouteDataByQueueId(queueId, queueRouteData);
- break;
- }
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfig, topicConfigManager.getDataVersion());
-
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
- response.setBody(RemotingSerializable.encode(queueRouteData));
- return response;
- }
-
- private RemotingCommand migrateTopicLogicalQueuePrepare(ChannelHandlerContext ctx, RemotingCommand request) {
- MigrateLogicalQueueBody reqRespBody = RemotingSerializable.decode(request.getBody(), MigrateLogicalQueueBody.class);
- if (reqRespBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- LogicalQueueRouteData fromQueueRouteData = reqRespBody.getFromQueueRouteData();
- LogicalQueueRouteData toQueueRouteData = reqRespBody.getToQueueRouteData();
- log.info("migrateTopicLogicalQueuePrepare fromQueueRouteData={} toQueueRouteData={}", fromQueueRouteData, toQueueRouteData);
- final MessageQueue fromMessageQueue = fromQueueRouteData.getMessageQueue();
- final int fromQueueId = fromQueueRouteData.getQueueId();
- final long fromOffsetDelta = fromQueueRouteData.getOffsetDelta();
- int logicalQueueIndex = fromQueueRouteData.getLogicalQueueIndex();
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- String topic = fromQueueRouteData.getTopic();
- if (!topicConfigManager.getTopicConfigTable().containsKey(topic)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not exist");
- }
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not enable logical queue");
- }
- logicalQueuesInfo.writeLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(logicalQueueIndex);
- if (queueRouteDataList == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, String.format(Locale.ENGLISH, "logical queue %d not exist", logicalQueueIndex));
- }
- fromQueueRouteData = null;
- for (LogicalQueueRouteData v : queueRouteDataList) {
- if (v.isSameTo(fromMessageQueue, fromOffsetDelta)) {
- fromQueueRouteData = v;
- break;
- }
- }
- if (fromQueueRouteData == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, String.format(Locale.ENGLISH, "message queue %d not exist", fromQueueId));
- }
- if (!MessageQueueRouteState.Normal.equals(fromQueueRouteData.getState())) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, String.format(Locale.ENGLISH, "message queue %d not normal state", fromQueueId));
- }
- reqRespBody.setFromQueueRouteData(fromQueueRouteData);
- if (fromQueueRouteData.isWritable()) {
- sealLogicalQueueRouteData(fromQueueRouteData, brokerController.getMessageStore());
- }
- toQueueRouteData.setLogicalQueueDelta(fromQueueRouteData.getLogicalQueueDelta() + fromQueueRouteData.getMessagesCount());
- queueRouteDataList.add(toQueueRouteData);
- } catch (InterruptedException e) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "thread interrupted");
- } catch (TimeoutException e) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "seal timeout");
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
- response.setBody(RemotingSerializable.encode(reqRespBody));
- return response;
- }
-
- private RemotingCommand migrateTopicLogicalQueueCommit(ChannelHandlerContext ctx, RemotingCommand request) {
- MigrateLogicalQueueBody reqRespBody = RemotingSerializable.decode(request.getBody(), MigrateLogicalQueueBody.class);
- if (reqRespBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- LogicalQueueRouteData fromQueueRouteData = reqRespBody.getFromQueueRouteData();
- LogicalQueueRouteData toQueueRouteData = reqRespBody.getToQueueRouteData();
- log.info("migrateTopicLogicalQueueCommit toQueueRouteData={}", toQueueRouteData);
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- String topic = toQueueRouteData.getTopic();
- int toQueueId = toQueueRouteData.getQueueId();
- if (!topicConfigManager.getTopicConfigTable().containsKey(topic)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not exist");
- }
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not enable logical queue");
- }
- LogicalQueueRouteData queueRouteData;
- logicalQueuesInfo.writeLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(toQueueRouteData.getLogicalQueueIndex());
- if (queueRouteDataList == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, String.format(Locale.ENGLISH, "logical queue %d not exist", toQueueRouteData.getLogicalQueueIndex()));
- }
- queueRouteDataList.stream().filter(fromQueueRouteData::isSameTo).forEach(d -> {
- d.copyFrom(fromQueueRouteData);
- });
- queueRouteData = queueRouteDataList.stream().filter(toQueueRouteData::isSameTo).findFirst().orElse(null);
- if (queueRouteData == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, String.format(Locale.ENGLISH, "message queue %d-%d not exist", toQueueId, toQueueRouteData.getOffsetDelta()));
- }
- queueRouteData.setLogicalQueueDelta(toQueueRouteData.getLogicalQueueDelta());
- queueRouteData.setState(MessageQueueRouteState.Normal);
- toQueueRouteData.setState(MessageQueueRouteState.Normal);
- if (toQueueRouteData.getBrokerAddr() != null) {
- queueRouteData.setBrokerAddr(toQueueRouteData.getBrokerAddr());
- }
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
- response.setBody(RemotingSerializable.encode(reqRespBody));
- return response;
- }
-
- private RemotingCommand migrateTopicLogicalQueueNotify(ChannelHandlerContext ctx, RemotingCommand request) {
- MigrateLogicalQueueBody requestBody = RemotingSerializable.decode(request.getBody(), MigrateLogicalQueueBody.class);
- if (requestBody == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "decode error");
- }
- LogicalQueueRouteData fromQueueRouteData = requestBody.getFromQueueRouteData();
- LogicalQueueRouteData toQueueRouteData = requestBody.getToQueueRouteData();
- log.info("migrateTopicLogicalQueueNotify fromQueueRouteData={} toQueueRouteData={}", fromQueueRouteData, toQueueRouteData);
- TopicConfigManager topicConfigManager = this.brokerController.getTopicConfigManager();
- String topic = toQueueRouteData.getTopic();
- if (!topicConfigManager.getTopicConfigTable().containsKey(topic)) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not exist");
- }
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.selectLogicalQueuesInfo(topic);
- if (logicalQueuesInfo == null) {
- return RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "topic not enable logical queue");
- }
- List<LogicalQueueRouteData> requestQueueRouteDataList = Lists.newArrayList(fromQueueRouteData, toQueueRouteData);
- boolean toQueueRouteDataFound = false;
- logicalQueuesInfo.writeLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.computeIfAbsent(toQueueRouteData.getLogicalQueueIndex(), ignore -> Lists.newArrayListWithExpectedSize(1));
- for (LogicalQueueRouteData v : queueRouteDataList) {
- for (Iterator<LogicalQueueRouteData> iterator = requestQueueRouteDataList.iterator(); iterator.hasNext(); ) {
- LogicalQueueRouteData queueRouteData = iterator.next();
- if (queueRouteData.isSameTo(v)) {
- v.copyFrom(queueRouteData);
- if (queueRouteData.getBrokerAddr() != null) {
- v.setBrokerAddr(queueRouteData.getBrokerAddr());
- }
- if (!toQueueRouteDataFound && toQueueRouteData.isSameTo(v)) {
- toQueueRouteDataFound = true;
- }
- iterator.remove();
- break;
- }
- }
- if (requestQueueRouteDataList.isEmpty()) {
- break;
- }
- }
- if (!queueRouteDataList.isEmpty() && !toQueueRouteDataFound) {
- // if this broker has this logical queue before, it should add latest writable route here, so that SendMessage request can be proxied
- queueRouteDataList.add(toQueueRouteData);
- }
- } finally {
- logicalQueuesInfo.writeLock().unlock();
- }
-
- topicConfigManager.getDataVersion().nextVersion();
- topicConfigManager.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(topicConfigManager.selectTopicConfig(topic), topicConfigManager.getDataVersion());
-
- final RemotingCommand response = RemotingCommand.createResponseCommand(null);
- response.setCode(ResponseCode.SUCCESS);
- return response;
- }
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
index 77317a6..04e705b 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
@@ -17,11 +17,9 @@
package org.apache.rocketmq.broker.processor;
import io.netty.channel.ChannelHandlerContext;
-import java.util.List;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ConsumerGroupInfo;
import org.apache.rocketmq.common.constant.LoggerName;
-import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.GetConsumerListByGroupRequestHeader;
@@ -31,6 +29,12 @@ import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetRequestHead
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetResponseHeader;
+import org.apache.rocketmq.common.rpc.RpcRequest;
+import org.apache.rocketmq.common.rpc.RpcResponse;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
@@ -38,6 +42,10 @@ import org.apache.rocketmq.remoting.netty.AsyncNettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+import java.util.List;
+
+import static org.apache.rocketmq.remoting.protocol.RemotingCommand.buildErrorResponse;
+
public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
@@ -109,6 +117,12 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
final UpdateConsumerOffsetRequestHeader requestHeader =
(UpdateConsumerOffsetRequestHeader) request
.decodeCommandCustomHeader(UpdateConsumerOffsetRequestHeader.class);
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader);
+
+ RemotingCommand rewriteResult = this.brokerController.getTopicQueueMappingManager().rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), requestHeader.getConsumerGroup(),
requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
response.setCode(ResponseCode.SUCCESS);
@@ -116,6 +130,77 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
return response;
}
+
+ public RemotingCommand rewriteRequestForStaticTopic(QueryConsumerOffsetRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", requestHeader.getTopic(), requestHeader.getQueueId(), mappingDetail.getBname()));
+ }
+ List<LogicQueueMappingItem> mappingItemList = mappingContext.getMappingItemList();
+ if (mappingItemList.size() == 1
+ && mappingItemList.get(0).getLogicOffset() == 0) {
+ //as physical, just let it go
+ requestHeader.setQueueId(mappingContext.getLeaderItem().getQueueId());
+ return null;
+ }
+ //double read check
+ List<LogicQueueMappingItem> itemList = mappingContext.getMappingItemList();
+ //by default, it is -1
+ long offset = -1;
+ //double read, first from leader, then from second leader
+ for (int i = itemList.size() - 1; i >= 0; i--) {
+ LogicQueueMappingItem mappingItem = itemList.get(i);
+ if (mappingItem.getBname().equals(mappingDetail.getBname())) {
+ offset = this.brokerController.getConsumerOffsetManager().queryOffset(requestHeader.getConsumerGroup(), requestHeader.getTopic(), mappingItem.getQueueId());
+ if (offset >= 0) {
+ break;
+ } else {
+ //not found
+ continue;
+ }
+ } else {
+ //maybe we need to reconstruct an object
+ requestHeader.setBname(mappingItem.getBname());
+ requestHeader.setQueueId(mappingItem.getQueueId());
+ requestHeader.setLo(false);
+ requestHeader.setSetZeroIfNotFound(false);
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.QUERY_CONSUMER_OFFSET, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ if (rpcResponse.getCode() == ResponseCode.SUCCESS) {
+ offset = ((QueryConsumerOffsetResponseHeader) rpcResponse.getHeader()).getOffset();
+ break;
+ } else if (rpcResponse.getCode() == ResponseCode.QUERY_NOT_FOUND) {
+ continue;
+ } else {
+ //this should not happen
+ throw new RuntimeException("Unknown response code " + rpcResponse.getCode());
+ }
+ }
+ }
+ final RemotingCommand response = RemotingCommand.createResponseCommand(QueryConsumerOffsetResponseHeader.class);
+ final QueryConsumerOffsetResponseHeader responseHeader = (QueryConsumerOffsetResponseHeader) response.readCustomHeader();
+ if (offset >= 0) {
+ responseHeader.setOffset(offset);
+ response.setCode(ResponseCode.SUCCESS);
+ response.setRemark(null);
+ } else {
+ response.setCode(ResponseCode.QUERY_NOT_FOUND);
+ response.setRemark("Not found, maybe this group consumer boot first");
+ }
+ return response;
+ } catch (Throwable t) {
+ t.printStackTrace();
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
private RemotingCommand queryConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request)
throws RemotingCommandException {
final RemotingCommand response =
@@ -126,6 +211,13 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
(QueryConsumerOffsetRequestHeader) request
.decodeCommandCustomHeader(QueryConsumerOffsetRequestHeader.class);
+
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader);
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+
long offset =
this.brokerController.getConsumerOffsetManager().queryOffset(
requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
@@ -138,7 +230,10 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
long minOffset =
this.brokerController.getMessageStore().getMinOffsetInQueue(requestHeader.getTopic(),
requestHeader.getQueueId());
- if (minOffset <= 0
+ if (requestHeader.getSetZeroIfNotFound() != null && Boolean.FALSE.equals(requestHeader.getSetZeroIfNotFound())) {
+ response.setCode(ResponseCode.QUERY_NOT_FOUND);
+ response.setRemark("Not found, do not set to zero, maybe this group boot first");
+ } else if (minOffset <= 0
&& !this.brokerController.getMessageStore().checkInDiskByConsumeOffset(
requestHeader.getTopic(), requestHeader.getQueueId(), 0)) {
responseHeader.setOffset(0L);
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/PullMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/PullMessageProcessor.java
index e61ef11..0f1ca23 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/PullMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/PullMessageProcessor.java
@@ -16,17 +16,13 @@
*/
package org.apache.rocketmq.broker.processor;
-import com.google.common.collect.ImmutableList;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.FileRegion;
-import java.nio.ByteBuffer;
-import java.util.List;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ConsumerGroupInfo;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.filter.ConsumerFilterData;
import org.apache.rocketmq.broker.filter.ConsumerFilterManager;
import org.apache.rocketmq.broker.filter.ExpressionForRetryMessageFilter;
@@ -43,17 +39,22 @@ import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.filter.ExpressionType;
import org.apache.rocketmq.common.filter.FilterAPI;
import org.apache.rocketmq.common.help.FAQUrl;
-import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.PullMessageResponseHeader;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.common.protocol.topic.OffsetMovedEvent;
+import org.apache.rocketmq.common.rpc.RpcClientUtils;
+import org.apache.rocketmq.common.rpc.RpcRequest;
+import org.apache.rocketmq.common.rpc.RpcResponse;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.sysflag.PullSysFlag;
@@ -67,7 +68,6 @@ import org.apache.rocketmq.remoting.netty.AsyncNettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.RequestTask;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
import org.apache.rocketmq.store.GetMessageResult;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageFilter;
@@ -75,6 +75,11 @@ import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+import static org.apache.rocketmq.remoting.protocol.RemotingCommand.buildErrorResponse;
+
public class PullMessageProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private final BrokerController brokerController;
@@ -95,6 +100,200 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
return false;
}
+
+
+ private RemotingCommand rewriteRequestForStaticTopic(PullMessageRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ String topic = mappingContext.getTopic();
+ Integer globalId = mappingContext.getGlobalId();
+ // if the leader? consider the order consumer, which will lock the mq
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d cannot find mapping item in request process of current broker %s", topic, globalId, mappingDetail.getBname()));
+ }
+ Long globalOffset = requestHeader.getQueueOffset();
+
+ LogicQueueMappingItem mappingItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), globalOffset, true);
+ mappingContext.setCurrentItem(mappingItem);
+
+ if (globalOffset < mappingItem.getLogicOffset()) {
+ //handleOffsetMoved
+ //If the physical queue is reused, we should handle the PULL_OFFSET_MOVED independently
+ //Otherwise, we could just transfer it to the physical process
+ }
+ //below are physical info
+ String bname = mappingItem.getBname();
+ Integer phyQueueId = mappingItem.getQueueId();
+ Long phyQueueOffset = mappingItem.computePhysicalQueueOffset(globalOffset);
+ requestHeader.setQueueId(phyQueueId);
+ requestHeader.setQueueOffset(phyQueueOffset);
+ if (mappingItem.checkIfEndOffsetDecided()
+ && requestHeader.getMaxMsgNums() != null) {
+ requestHeader.setMaxMsgNums((int) Math.min(mappingItem.getEndOffset() - mappingItem.getStartOffset(), requestHeader.getMaxMsgNums()));
+ }
+
+ if (mappingDetail.getBname().equals(bname)) {
+ //just let it go, do the local pull process
+ return null;
+ }
+
+ int sysFlag = requestHeader.getSysFlag();
+ requestHeader.setLo(false);
+ requestHeader.setBname(bname);
+ sysFlag = PullSysFlag.clearSuspendFlag(sysFlag);
+ sysFlag = PullSysFlag.clearCommitOffsetFlag(sysFlag);
+ requestHeader.setSysFlag(sysFlag);
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.PULL_MESSAGE, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+
+ PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) rpcResponse.getHeader();
+ {
+ RemotingCommand rewriteResult = rewriteResponseForStaticTopic(requestHeader, responseHeader, mappingContext, rpcResponse.getCode());
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+ }
+ return RpcClientUtils.createCommandForRpcResponse(rpcResponse);
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
+ private RemotingCommand rewriteResponseForStaticTopic(PullMessageRequestHeader requestHeader, PullMessageResponseHeader responseHeader,
+ TopicQueueMappingContext mappingContext, final int code) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ LogicQueueMappingItem leaderItem = mappingContext.getLeaderItem();
+
+ LogicQueueMappingItem currentItem = mappingContext.getCurrentItem();
+
+ LogicQueueMappingItem earlistItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), 0L, true);
+
+ assert currentItem.getLogicOffset() >= 0;
+
+ long requestOffset = requestHeader.getQueueOffset();
+ long nextBeginOffset = responseHeader.getNextBeginOffset();
+ long minOffset = responseHeader.getMinOffset();
+ long maxOffset = responseHeader.getMaxOffset();
+ int responseCode = code;
+
+ //consider the following situations
+ // 1. read from slave, currently not supported
+ // 2. the middle queue is truncated because of deleting commitlog
+ if (code != ResponseCode.SUCCESS) {
+ //note the currentItem maybe both the leader and the earliest
+ boolean isRevised = false;
+ if (leaderItem.getGen() == currentItem.getGen()) {
+ //read the leader
+ if (requestOffset > maxOffset) {
+ //actually, we need do nothing, but keep the code structure here
+ if (code == ResponseCode.PULL_OFFSET_MOVED) {
+ responseCode = ResponseCode.PULL_OFFSET_MOVED;
+ nextBeginOffset = maxOffset;
+ } else {
+ //maybe current broker is the slave
+ responseCode = code;
+ }
+ } else if (requestOffset < minOffset) {
+ nextBeginOffset = minOffset;
+ responseCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
+ } else {
+ responseCode = code;
+ }
+ }
+ //note the currentItem maybe both the leader and the earliest
+ if (earlistItem.getGen() == currentItem.getGen()) {
+ //read the earliest one
+ if (requestOffset < minOffset) {
+ if (code == ResponseCode.PULL_OFFSET_MOVED) {
+ responseCode = ResponseCode.PULL_OFFSET_MOVED;
+ nextBeginOffset = minOffset;
+ } else {
+ //maybe read from slave, but we still set it to moved
+ responseCode = ResponseCode.PULL_OFFSET_MOVED;
+ nextBeginOffset = minOffset;
+ }
+ } else if (requestOffset >= maxOffset) {
+ //just move to another item
+ LogicQueueMappingItem nextItem = TopicQueueMappingUtils.findNext(mappingContext.getMappingItemList(), currentItem, true);
+ if (nextItem != null) {
+ isRevised = true;
+ currentItem = nextItem;
+ nextBeginOffset = currentItem.getStartOffset();
+ minOffset = currentItem.getStartOffset();
+ maxOffset = minOffset;
+ responseCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
+ } else {
+ //maybe the next one's logic offset is -1
+ responseCode = ResponseCode.PULL_NOT_FOUND;
+ }
+ } else {
+ //let it go
+ responseCode = code;
+ }
+ }
+
+ //read from the middle item, ignore the PULL_OFFSET_MOVED
+ if (!isRevised
+ && leaderItem.getGen() != currentItem.getGen()
+ && earlistItem.getGen() != currentItem.getGen()) {
+ if (requestOffset < minOffset) {
+ nextBeginOffset = minOffset;
+ responseCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
+ } else if (requestOffset >= maxOffset) {
+ //just move to another item
+ LogicQueueMappingItem nextItem = TopicQueueMappingUtils.findNext(mappingContext.getMappingItemList(), currentItem, true);
+ if (nextItem != null) {
+ currentItem = nextItem;
+ nextBeginOffset = currentItem.getStartOffset();
+ minOffset = currentItem.getStartOffset();
+ maxOffset = minOffset;
+ responseCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
+ } else {
+ //maybe the next one's logic offset is -1
+ responseCode = ResponseCode.PULL_NOT_FOUND;
+ }
+ } else {
+ responseCode = code;
+ }
+ }
+ }
+
+ //handle nextBeginOffset
+ //the next begin offset should no more than the end offset
+ if (currentItem.checkIfEndOffsetDecided()
+ && nextBeginOffset >= currentItem.getEndOffset()) {
+ nextBeginOffset = currentItem.getEndOffset();
+ }
+ responseHeader.setNextBeginOffset(currentItem.computeStaticQueueOffsetStrictly(nextBeginOffset));
+ //handle min offset
+ responseHeader.setMinOffset(currentItem.computeStaticQueueOffsetStrictly(Math.max(currentItem.getStartOffset(), minOffset)));
+ //handle max offset
+ responseHeader.setMaxOffset(Math.max(currentItem.computeStaticQueueOffsetStrictly(maxOffset),
+ TopicQueueMappingDetail.computeMaxOffsetFromMapping(mappingDetail, mappingContext.getGlobalId())));
+ //set the offsetDelta
+ responseHeader.setOffsetDelta(currentItem.computeOffsetDelta());
+
+ if (code != ResponseCode.SUCCESS) {
+ return RemotingCommand.createResponseCommandWithHeader(responseCode, responseHeader);
+ } else {
+ return null;
+ }
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
+
private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
throws RemotingCommandException {
RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
@@ -132,25 +331,33 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
final long suspendTimeoutMillisLong = hasSuspendFlag ? requestHeader.getSuspendTimeoutMillis() : 0;
- String topic = requestHeader.getTopic();
- TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(topic);
+ TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
if (null == topicConfig) {
- log.error("the topic {} not exist, consumer: {}", topic, RemotingHelper.parseChannelRemoteAddr(channel));
+ log.error("the topic {} not exist, consumer: {}", requestHeader.getTopic(), RemotingHelper.parseChannelRemoteAddr(channel));
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
- response.setRemark(String.format("topic[%s] not exist, apply first please! %s", topic, FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
+ response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(), FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
return response;
}
if (!PermName.isReadable(topicConfig.getPerm())) {
response.setCode(ResponseCode.NO_PERMISSION);
- response.setRemark("the topic[" + topic + "] pulling message is forbidden");
+ response.setRemark("the topic[" + requestHeader.getTopic() + "] pulling message is forbidden");
return response;
}
- int queueId = requestHeader.getQueueId();
- if (queueId < 0 || queueId >= topicConfig.getReadQueueNums()) {
+
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader, false);
+
+ {
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+ }
+
+ if (requestHeader.getQueueId() < 0 || requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) {
String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] consumer:[%s]",
- queueId, topic, topicConfig.getReadQueueNums(), channel.remoteAddress());
+ requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(), channel.remoteAddress());
log.warn(errorInfo);
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(errorInfo);
@@ -162,11 +369,11 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
if (hasSubscriptionFlag) {
try {
subscriptionData = FilterAPI.build(
- topic, requestHeader.getSubscription(), requestHeader.getExpressionType()
+ requestHeader.getTopic(), requestHeader.getSubscription(), requestHeader.getExpressionType()
);
if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
consumerFilterData = ConsumerFilterManager.build(
- topic, requestHeader.getConsumerGroup(), requestHeader.getSubscription(),
+ requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getSubscription(),
requestHeader.getExpressionType(), requestHeader.getSubVersion()
);
assert consumerFilterData != null;
@@ -195,9 +402,9 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
return response;
}
- subscriptionData = consumerGroupInfo.findSubscriptionData(topic);
+ subscriptionData = consumerGroupInfo.findSubscriptionData(requestHeader.getTopic());
if (null == subscriptionData) {
- log.warn("the consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), topic);
+ log.warn("the consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), requestHeader.getTopic());
response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
response.setRemark("the consumer's subscription not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
return response;
@@ -211,7 +418,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
return response;
}
if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
- consumerFilterData = this.brokerController.getConsumerFilterManager().get(topic,
+ consumerFilterData = this.brokerController.getConsumerFilterManager().get(requestHeader.getTopic(),
requestHeader.getConsumerGroup());
if (consumerFilterData == null) {
response.setCode(ResponseCode.FILTER_DATA_NOT_EXIST);
@@ -220,7 +427,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
}
if (consumerFilterData.getClientVersion() < requestHeader.getSubVersion()) {
log.warn("The broker's consumer filter data is not latest, group: {}, topic: {}, serverV: {}, clientV: {}",
- requestHeader.getConsumerGroup(), topic, consumerFilterData.getClientVersion(), requestHeader.getSubVersion());
+ requestHeader.getConsumerGroup(), requestHeader.getTopic(), consumerFilterData.getClientVersion(), requestHeader.getSubVersion());
response.setCode(ResponseCode.FILTER_DATA_NOT_LATEST);
response.setRemark("the consumer's consumer filter data not latest");
return response;
@@ -235,6 +442,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
return response;
}
+
MessageFilter messageFilter;
if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData,
@@ -244,72 +452,12 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
this.brokerController.getConsumerFilterManager());
}
- long offset = requestHeader.getQueueOffset();
- int maxMsgNums = requestHeader.getMaxMsgNums();
-
- LogicalQueuesInfoInBroker logicalQueuesInfo = this.brokerController.getTopicConfigManager().selectLogicalQueuesInfo(topic);
- LogicalQueueRouteData queueRouteData = null;
- if (logicalQueuesInfo != null) {
- int responseErrorCode = ResponseCode.SUCCESS;
- queueRouteData = logicalQueuesInfo.queryQueueRouteDataByQueueId(queueId, offset);
- if (queueRouteData != null) {
- if (queueRouteData.isWriteOnly()) {
- responseErrorCode = ResponseCode.PULL_NOT_FOUND;
- response.setRemark("logical queue write only");
- } else if (queueRouteData.isExpired()) {
- responseErrorCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
- response.setRemark("logical queue expired");
- prepareRedirectResponse(response, logicalQueuesInfo, queueRouteData);
- } else if (MessageQueueRouteState.ReadOnly.equals(queueRouteData.getState()) && queueRouteData.getOffsetMax() >= 0) {
- if (offset >= queueRouteData.getOffsetMax()) {
- responseErrorCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
- response.setRemark("queue offset exceed offsetMax");
- prepareRedirectResponse(response, logicalQueuesInfo, queueRouteData);
- } else if (offset + maxMsgNums > queueRouteData.getOffsetMax()) {
- if ((queueRouteData.getOffsetMax() - 1 <= this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId)) &&
- (this.brokerController.getMessageStore().getCommitLogOffsetInQueue(topic, queueId, queueRouteData.getOffsetMax() - 1) < this.brokerController.getMessageStore().getMinPhyOffset())) {
- responseErrorCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
- response.setRemark("queue offset removed");
- prepareRedirectResponse(response, logicalQueuesInfo, queueRouteData);
- } else {
- maxMsgNums = (int) (queueRouteData.getOffsetMax() - offset);
- if (maxMsgNums <= 0) {
- responseErrorCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
- response.setRemark("queue offset out of range");
- prepareRedirectResponse(response, logicalQueuesInfo, queueRouteData);
- }
- }
- }
- }
- } else {
- responseErrorCode = ResponseCode.PULL_RETRY_IMMEDIATELY;
- response.setRemark("no suitable queue");
- response.addExtField(MessageConst.PROPERTY_REDIRECT, "1");
- // instruct client to refresh all
- response.setBody(null);
- queueRouteData = logicalQueuesInfo.queryQueueRouteDataByQueueId(queueId, 0L);
- }
- if (responseErrorCode != ResponseCode.SUCCESS) {
- response.setCode(responseErrorCode);
- responseHeader.setMinOffset(offset);
- responseHeader.setMaxOffset(queueRouteData != null ? queueRouteData.getOffsetMax() : offset);
- responseHeader.setNextBeginOffset(queueRouteData != null ? queueRouteData.getOffsetMax() : offset);
- responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
- return response;
- }
- }
-
final GetMessageResult getMessageResult =
- this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), topic,
- queueId, offset, maxMsgNums, messageFilter);
+ this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
+ requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
if (getMessageResult != null) {
response.setRemark(getMessageResult.getStatus().name());
- long nextBeginOffset = getMessageResult.getNextBeginOffset();
- if (queueRouteData != null && queueRouteData.getOffsetMax() >= 0 && nextBeginOffset > queueRouteData.getOffsetMax()) {
- // prevent from pulling messages from next logical queue route data
- nextBeginOffset = queueRouteData.getOffsetMax();
- }
- responseHeader.setNextBeginOffset(nextBeginOffset);
+ responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset());
responseHeader.setMinOffset(getMessageResult.getMinOffset());
// this does not need to be modified since it's not an accurate value under logical queue.
responseHeader.setMaxOffset(getMessageResult.getMaxOffset());
@@ -360,9 +508,9 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
// XXX: warn and notify me
log.info("the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}",
requestHeader.getQueueOffset(),
- nextBeginOffset,
- topic,
- queueId,
+ getMessageResult.getNextBeginOffset(),
+ requestHeader.getTopic(),
+ requestHeader.getQueueId(),
requestHeader.getConsumerGroup()
);
} else {
@@ -387,7 +535,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
case OFFSET_TOO_SMALL:
response.setCode(ResponseCode.PULL_OFFSET_MOVED);
log.info("the request offset too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}",
- requestHeader.getConsumerGroup(), topic, requestHeader.getQueueOffset(),
+ requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(),
getMessageResult.getMinOffset(), channel.remoteAddress());
break;
default:
@@ -395,11 +543,14 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
break;
}
+
+
+
if (this.hasConsumeMessageHook()) {
ConsumeMessageContext context = new ConsumeMessageContext();
context.setConsumerGroup(requestHeader.getConsumerGroup());
- context.setTopic(topic);
- context.setQueueId(queueId);
+ context.setTopic(requestHeader.getTopic());
+ context.setQueueId(requestHeader.getQueueId());
String owner = request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER);
@@ -437,6 +588,12 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
this.executeConsumeMessageHookBefore(context);
}
+ //rewrite the response for the
+ RemotingCommand rewriteResult = rewriteResponseForStaticTopic(requestHeader, responseHeader, mappingContext, response.getCode());
+ if (rewriteResult != null) {
+ response = rewriteResult;
+ }
+
switch (response.getCode()) {
case ResponseCode.SUCCESS:
@@ -483,6 +640,9 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
}
+ String topic = requestHeader.getTopic();
+ long offset = requestHeader.getQueueOffset();
+ int queueId = requestHeader.getQueueId();
PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
@@ -490,20 +650,6 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
break;
}
- if (queueRouteData != null) {
- logicalQueuesInfo.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(queueRouteData.getLogicalQueueIndex());
- MessageQueue latestMessageQueue = queueRouteDataList.get(queueRouteDataList.size() - 1).getMessageQueue();
- if (!latestMessageQueue.getBrokerName().equals(brokerController.getBrokerConfig().getBrokerName()) || latestMessageQueue.getQueueId() != queueId) {
- // There are other newer message queue, instruct client to refresh meta-data to access these
- prepareRedirectResponse(response, logicalQueuesInfo, queueRouteData);
- }
- } finally {
- logicalQueuesInfo.readLock().unlock();
- }
- }
-
case ResponseCode.PULL_RETRY_IMMEDIATELY:
break;
case ResponseCode.PULL_OFFSET_MOVED:
@@ -518,7 +664,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
event.setConsumerGroup(requestHeader.getConsumerGroup());
event.setMessageQueue(mq);
event.setOffsetRequest(requestHeader.getQueueOffset());
- event.setOffsetNew(nextBeginOffset);
+ event.setOffsetNew(getMessageResult.getNextBeginOffset());
this.generateOffsetMovedEvent(event);
log.warn(
"PULL_OFFSET_MOVED:correction offset. topic={}, groupId={}, requestOffset={}, newOffset={}, suggestBrokerId={}",
@@ -547,20 +693,11 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
&& this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;
if (storeOffsetEnable) {
this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel),
- requestHeader.getConsumerGroup(), topic, queueId, requestHeader.getCommitOffset());
+ requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
}
return response;
}
- private void prepareRedirectResponse(RemotingCommand response, LogicalQueuesInfoInBroker logicalQueuesInfo,
- LogicalQueueRouteData queueRouteData) {
- LogicalQueueRouteData nextReadableLogicalQueueRouteData = logicalQueuesInfo.nextAvailableLogicalRouteData(queueRouteData, LogicalQueueRouteData::isReadable);
- if (nextReadableLogicalQueueRouteData != null) {
- response.addExtField(MessageConst.PROPERTY_REDIRECT, "1");
- response.setBody(RemotingSerializable.encode(ImmutableList.of(queueRouteData, nextReadableLogicalQueueRouteData)));
- }
- }
-
public boolean hasConsumeMessageHook() {
return consumeMessageHookList != null && !this.consumeMessageHookList.isEmpty();
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
index 91463a4..17f19cb 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
@@ -28,10 +28,13 @@ import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.mqtrace.ConsumeMessageContext;
import org.apache.rocketmq.broker.mqtrace.ConsumeMessageHook;
import org.apache.rocketmq.broker.mqtrace.SendMessageContext;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.TopicFilterType;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.help.FAQUrl;
@@ -60,6 +63,8 @@ import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import static org.apache.rocketmq.remoting.protocol.RemotingCommand.buildErrorResponse;
+
public class SendMessageProcessor extends AbstractSendMessageProcessor implements NettyRequestProcessor {
private List<ConsumeMessageHook> consumeMessageHookList;
@@ -96,16 +101,54 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
if (requestHeader == null) {
return CompletableFuture.completedFuture(null);
}
+ TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader, true);
+ RemotingCommand rewriteResult = this.brokerController.getTopicQueueMappingManager().rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ if (rewriteResult != null) {
+ return CompletableFuture.completedFuture(rewriteResult);
+ }
mqtraceContext = buildMsgContext(ctx, requestHeader);
this.executeSendMessageHookBefore(ctx, request, mqtraceContext);
if (requestHeader.isBatch()) {
- return this.asyncSendBatchMessage(ctx, request, mqtraceContext, requestHeader);
+ return this.asyncSendBatchMessage(ctx, request, mqtraceContext, requestHeader, mappingContext);
} else {
- return this.asyncSendMessage(ctx, request, mqtraceContext, requestHeader);
+ return this.asyncSendMessage(ctx, request, mqtraceContext, requestHeader, mappingContext);
}
}
}
+
+ /**
+ * If the response is not null, it meets some errors
+ * @return
+ */
+
+
+ private RemotingCommand rewriteResponseForStaticTopic(SendMessageResponseHeader responseHeader, TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+
+ LogicQueueMappingItem mappingItem = mappingContext.getLeaderItem();
+ if (mappingItem == null) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname()));
+ }
+ //no need to care the broker name
+ long staticLogicOffset = mappingItem.computeStaticQueueOffsetLoosely(responseHeader.getQueueOffset());
+ if (staticLogicOffset < 0) {
+ //if the logic offset is -1, just let it go
+ //maybe we need a dynamic config
+ //return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d convert offset error in current broker %s", mappingContext.getTopic(), mappingContext.getGlobalId(), mappingDetail.getBname()));
+ }
+ responseHeader.setQueueId(mappingContext.getGlobalId());
+ responseHeader.setQueueOffset(staticLogicOffset);
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ return null;
+ }
+
@Override
public boolean rejectRequest() {
return this.brokerController.getMessageStore().isOSPageCacheBusy() ||
@@ -171,6 +214,14 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
return CompletableFuture.completedFuture(response);
}
+ if (requestHeader.getOriginTopic() != null
+ && !msgExt.getTopic().equals(requestHeader.getOriginTopic())) {
+ //here just do some fence in case of some unexpected offset is income
+ response.setCode(ResponseCode.SYSTEM_ERROR);
+ response.setRemark("look message by offset failed to check the topic name" + requestHeader.getOffset());
+ return CompletableFuture.completedFuture(response);
+ }
+
final String retryTopic = msgExt.getProperty(MessageConst.PROPERTY_RETRY_TOPIC);
if (null == retryTopic) {
MessageAccessor.putProperty(msgExt, MessageConst.PROPERTY_RETRY_TOPIC, msgExt.getTopic());
@@ -264,7 +315,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
private CompletableFuture<RemotingCommand> asyncSendMessage(ChannelHandlerContext ctx, RemotingCommand request,
SendMessageContext mqtraceContext,
- SendMessageRequestHeader requestHeader) {
+ SendMessageRequestHeader requestHeader,
+ TopicQueueMappingContext mappingContext) {
final RemotingCommand response = preSend(ctx, request, requestHeader);
final SendMessageResponseHeader responseHeader = (SendMessageResponseHeader)response.readCustomHeader();
@@ -310,12 +362,6 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgInner.getProperties()));
}
- LogicalQueueContext logicalQueueContext = super.buildLogicalQueueContext(msgInner.getTopic(), msgInner.getQueueId(), response);
- CompletableFuture<RemotingCommand> future = logicalQueueContext.hookBeforePut(ctx, requestHeader, request, response);
- if (future != null) {
- return future;
- }
-
CompletableFuture<PutMessageResult> putMessageResult = null;
String transFlag = origProps.get(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (transFlag != null && Boolean.parseBoolean(transFlag)) {
@@ -324,15 +370,13 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
response.setRemark(
"the broker[" + this.brokerController.getBrokerConfig().getBrokerIP1()
+ "] sending transaction message is forbidden");
- logicalQueueContext.hookAfterPut(null);
return CompletableFuture.completedFuture(response);
}
putMessageResult = this.brokerController.getTransactionalMessageService().asyncPrepareMessage(msgInner);
} else {
putMessageResult = this.brokerController.getMessageStore().asyncPutMessage(msgInner);
}
- logicalQueueContext.hookAfterPut(putMessageResult);
- return handlePutMessageResultFuture(putMessageResult, response, request, msgInner, responseHeader, mqtraceContext, ctx, queueIdInt);
+ return handlePutMessageResultFuture(putMessageResult, response, request, msgInner, responseHeader, mqtraceContext, ctx, queueIdInt, requestHeader, mappingContext);
}
private CompletableFuture<RemotingCommand> handlePutMessageResultFuture(CompletableFuture<PutMessageResult> putMessageResult,
@@ -342,9 +386,11 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
SendMessageResponseHeader responseHeader,
SendMessageContext sendMessageContext,
ChannelHandlerContext ctx,
- int queueIdInt) {
+ int queueIdInt,
+ SendMessageRequestHeader requestHeader,
+ TopicQueueMappingContext mappingContext) {
return putMessageResult.thenApply((r) ->
- handlePutMessageResult(r, response, request, msgInner, responseHeader, sendMessageContext, ctx, queueIdInt)
+ handlePutMessageResult(r, response, request, msgInner, responseHeader, sendMessageContext, ctx, queueIdInt, requestHeader, mappingContext)
);
}
@@ -396,7 +442,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
private RemotingCommand sendMessage(final ChannelHandlerContext ctx,
final RemotingCommand request,
final SendMessageContext sendMessageContext,
- final SendMessageRequestHeader requestHeader) throws RemotingCommandException {
+ final SendMessageRequestHeader requestHeader,
+ final TopicQueueMappingContext mappingContext) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(SendMessageResponseHeader.class);
final SendMessageResponseHeader responseHeader = (SendMessageResponseHeader)response.readCustomHeader();
@@ -465,14 +512,15 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
putMessageResult = this.brokerController.getMessageStore().putMessage(msgInner);
}
- return handlePutMessageResult(putMessageResult, response, request, msgInner, responseHeader, sendMessageContext, ctx, queueIdInt);
+ return handlePutMessageResult(putMessageResult, response, request, msgInner, responseHeader, sendMessageContext, ctx, queueIdInt, requestHeader, mappingContext);
}
private RemotingCommand handlePutMessageResult(PutMessageResult putMessageResult, RemotingCommand response,
RemotingCommand request, MessageExt msg,
SendMessageResponseHeader responseHeader, SendMessageContext sendMessageContext, ChannelHandlerContext ctx,
- int queueIdInt) {
+ int queueIdInt, SendMessageRequestHeader requestHeader,
+ TopicQueueMappingContext mappingContext) {
if (putMessageResult == null) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("store putMessage return null");
@@ -549,6 +597,11 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
responseHeader.setQueueId(queueIdInt);
responseHeader.setQueueOffset(putMessageResult.getAppendMessageResult().getLogicsOffset());
+ RemotingCommand rewriteResult = rewriteResponseForStaticTopic(responseHeader, mappingContext);
+ if (rewriteResult != null) {
+ return rewriteResult;
+ }
+
doResponse(ctx, request, response);
if (hasSendMessageHook()) {
@@ -582,7 +635,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
private CompletableFuture<RemotingCommand> asyncSendBatchMessage(ChannelHandlerContext ctx, RemotingCommand request,
SendMessageContext mqtraceContext,
- SendMessageRequestHeader requestHeader) {
+ SendMessageRequestHeader requestHeader,
+ TopicQueueMappingContext mappingContext) {
final RemotingCommand response = preSend(ctx, request, requestHeader);
final SendMessageResponseHeader responseHeader = (SendMessageResponseHeader)response.readCustomHeader();
@@ -623,17 +677,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
String clusterName = this.brokerController.getBrokerConfig().getBrokerClusterName();
MessageAccessor.putProperty(messageExtBatch, MessageConst.PROPERTY_CLUSTER, clusterName);
- LogicalQueueContext logicalQueueContext = super.buildLogicalQueueContext(messageExtBatch.getTopic(), messageExtBatch.getQueueId(), response);
- CompletableFuture<RemotingCommand> future = logicalQueueContext.hookBeforePut(ctx, requestHeader, request, response);
- if (future != null) {
- return future;
- }
-
CompletableFuture<PutMessageResult> putMessageResult = this.brokerController.getMessageStore().asyncPutMessages(messageExtBatch);
-
- logicalQueueContext.hookAfterPut(putMessageResult);
-
- return handlePutMessageResultFuture(putMessageResult, response, request, messageExtBatch, responseHeader, mqtraceContext, ctx, queueIdInt);
+ return handlePutMessageResultFuture(putMessageResult, response, request, messageExtBatch, responseHeader, mqtraceContext, ctx, queueIdInt, requestHeader, mappingContext);
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/slave/SlaveSynchronize.java b/broker/src/main/java/org/apache/rocketmq/broker/slave/SlaveSynchronize.java
index 7b5e564..9bb09bd 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/slave/SlaveSynchronize.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/slave/SlaveSynchronize.java
@@ -16,18 +16,19 @@
*/
package org.apache.rocketmq.broker.slave;
-import java.io.IOException;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.subscription.SubscriptionGroupManager;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.constant.LoggerName;
-import org.apache.rocketmq.logging.InternalLogger;
-import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.common.protocol.body.ConsumerOffsetSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.SubscriptionGroupWrapper;
-import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
+import org.apache.rocketmq.common.protocol.body.TopicConfigAndMappingSerializeWrapper;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
+import java.io.IOException;
+
public class SlaveSynchronize {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private final BrokerController brokerController;
@@ -56,7 +57,7 @@ public class SlaveSynchronize {
String masterAddrBak = this.masterAddr;
if (masterAddrBak != null && !masterAddrBak.equals(brokerController.getBrokerAddr())) {
try {
- TopicConfigSerializeWrapper topicWrapper =
+ TopicConfigAndMappingSerializeWrapper topicWrapper =
this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
if (!this.brokerController.getTopicConfigManager().getDataVersion()
.equals(topicWrapper.getDataVersion())) {
@@ -67,9 +68,17 @@ public class SlaveSynchronize {
this.brokerController.getTopicConfigManager().getTopicConfigTable()
.putAll(topicWrapper.getTopicConfigTable());
this.brokerController.getTopicConfigManager().persist();
-
- log.info("Update slave topic config from master, {}", masterAddrBak);
}
+ if (topicWrapper.getTopicQueueMappingDetailMap() != null
+ && !topicWrapper.getMappingDataVersion().equals(this.brokerController.getTopicQueueMappingManager().getDataVersion())) {
+ this.brokerController.getTopicQueueMappingManager().getDataVersion()
+ .assignNewOne(topicWrapper.getMappingDataVersion());
+ this.brokerController.getTopicQueueMappingManager().getTopicQueueMappingTable().clear();
+ this.brokerController.getTopicQueueMappingManager().getTopicQueueMappingTable()
+ .putAll(topicWrapper.getTopicQueueMappingDetailMap());
+ this.brokerController.getTopicQueueMappingManager().persist();
+ }
+ log.info("Update slave topic config from master, {}", masterAddrBak);
} catch (Exception e) {
log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
index bf69005..4af230d 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
@@ -17,20 +17,16 @@
package org.apache.rocketmq.broker.topic;
import java.util.Iterator;
-import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
-import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
-import java.util.stream.Collectors;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.BrokerPathConfigHelper;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.common.ConfigManager;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MixAll;
@@ -39,15 +35,10 @@ import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.protocol.body.KVTable;
import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
-import org.apache.rocketmq.srvutil.ConcurrentHashMapUtil;
-import org.apache.rocketmq.store.CleanFilesHook;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
public class TopicConfigManager extends ConfigManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
@@ -61,19 +52,8 @@ public class TopicConfigManager extends ConfigManager {
private final DataVersion dataVersion = new DataVersion();
private transient BrokerController brokerController;
- private final ConcurrentMap<String, LogicalQueuesInfoInBroker> logicalQueuesInfoTable = new ConcurrentHashMap<>();
- private final CleanFilesHook logicalQueueCleanHook = new CleanFilesHook() {
- @Override public void execute(DefaultMessageStore defaultMessageStore, long deleteCount) {
- if (deleteCount == 0) {
- return;
- }
- TopicConfigManager.this.logicalQueueClean();
- }
-
- @Override public String getName() {
- return TopicConfigManager.class.getSimpleName() + ".logicalQueueCleanHook";
- }
- };
+ public TopicConfigManager() {
+ }
public TopicConfigManager(BrokerController brokerController) {
this.brokerController = brokerController;
@@ -441,8 +421,6 @@ public class TopicConfigManager extends ConfigManager {
public TopicConfigSerializeWrapper buildTopicConfigSerializeWrapper() {
TopicConfigSerializeWrapper topicConfigSerializeWrapper = new TopicConfigSerializeWrapper();
topicConfigSerializeWrapper.setTopicConfigTable(this.topicConfigTable);
- String brokerName = this.brokerController.getBrokerConfig().getBrokerName();
- topicConfigSerializeWrapper.setLogicalQueuesInfoMap(this.logicalQueuesInfoTable.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new LogicalQueuesInfoInBroker(e.getValue(), data -> Objects.equals(data.getBrokerName(), brokerName)))));
topicConfigSerializeWrapper.setDataVersion(this.dataVersion);
return topicConfigSerializeWrapper;
}
@@ -474,7 +452,6 @@ public class TopicConfigManager extends ConfigManager {
public String encode(final boolean prettyFormat) {
TopicConfigSerializeWrapper topicConfigSerializeWrapper = new TopicConfigSerializeWrapper();
topicConfigSerializeWrapper.setTopicConfigTable(this.topicConfigTable);
- topicConfigSerializeWrapper.setLogicalQueuesInfoMap(this.logicalQueuesInfoTable.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new LogicalQueuesInfoInBroker(e.getValue()))));
topicConfigSerializeWrapper.setDataVersion(this.dataVersion);
return topicConfigSerializeWrapper.toJson(prettyFormat);
}
@@ -494,92 +471,4 @@ public class TopicConfigManager extends ConfigManager {
public ConcurrentMap<String, TopicConfig> getTopicConfigTable() {
return topicConfigTable;
}
-
- public LogicalQueuesInfoInBroker selectLogicalQueuesInfo(String topicName) {
- return this.logicalQueuesInfoTable.get(topicName);
- }
-
- public LogicalQueuesInfoInBroker getOrCreateLogicalQueuesInfo(String topicName) {
- return ConcurrentHashMapUtil.computeIfAbsent(this.logicalQueuesInfoTable, topicName, ignored -> new LogicalQueuesInfoInBroker());
- }
-
- public boolean replaceTopicConfig(String topic, TopicConfig oldTopicConfig, TopicConfig newTopicConfig) {
- boolean ok = this.topicConfigTable.replace(topic, oldTopicConfig, newTopicConfig);
- if (ok) {
- this.dataVersion.nextVersion();
- persist(topic, newTopicConfig);
- }
- return ok;
- }
-
- public CleanFilesHook getLogicalQueueCleanHook() {
- return logicalQueueCleanHook;
- }
-
- void logicalQueueClean() {
- String brokerName = this.brokerController.getBrokerConfig().getBrokerName();
- MessageStore messageStore = this.brokerController.getMessageStore();
- for (Entry<String, LogicalQueuesInfoInBroker> entry : this.logicalQueuesInfoTable.entrySet()) {
- String topic = entry.getKey();
- LogicalQueuesInfoInBroker logicalQueuesInfo = entry.getValue();
- Lock readLock = logicalQueuesInfo.readLock();
- Lock writeLock = logicalQueuesInfo.writeLock();
- boolean changed = false;
- readLock.lock();
- try {
- for (List<LogicalQueueRouteData> list : logicalQueuesInfo.values()) {
- while (!list.isEmpty()) {
- LogicalQueueRouteData logicalQueueRouteData = list.get(0);
- String brokerBelongs;
- if (brokerName.equals(logicalQueueRouteData.getBrokerName())) {
- if (logicalQueueRouteData.isWritable()) {
- break;
- }
- boolean canRemove = logicalQueueRouteData.isExpired() || logicalQueueRouteData.getMessagesCount() == 0;
- if (!canRemove) {
- // do not use getMinOffsetInQueue method, since it is using ConsumeQueue data, but not CommitLog, CQ data is not accurate after CommitLog cleaning.
- long commitLogOffset = messageStore.getCommitLogOffsetInQueue(topic, logicalQueueRouteData.getQueueId(), logicalQueueRouteData.getOffsetMax() - 1);
- canRemove = commitLogOffset == 0 || messageStore.getMinPhyOffset() > commitLogOffset;
- }
- if (!canRemove) {
- break;
- }
- brokerBelongs = "self";
- } else {
- brokerBelongs = "other";
- }
- readLock.unlock();
- writeLock.lock();
- try {
- list.remove(0);
- } finally {
- readLock.lock();
- writeLock.unlock();
- }
- log.info("logicalQueueClean remove {} broker {}", brokerBelongs, logicalQueueRouteData);
- changed = true;
- }
- }
- if (changed) {
- logicalQueuesInfo = new LogicalQueuesInfoInBroker(logicalQueuesInfo);
- }
- } finally {
- readLock.unlock();
- }
- if (changed) {
- this.dataVersion.nextVersion();
- this.persist(topic, logicalQueuesInfo);
- this.brokerController.registerIncrementBrokerData(this.selectTopicConfig(topic), this.dataVersion);
- log.info("registerIncrementBrokerData because logicalQueueClean: {}", topic);
- }
- }
- }
-
- public void deleteQueueRouteData(String topic) {
- if (this.logicalQueuesInfoTable.remove(topic) != null) {
- log.info("delete queueRouteData config OK, topic: {}", topic);
- this.dataVersion.nextVersion();
- persist(topic, (LogicalQueuesInfoInBroker) null);
- }
- }
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingCleanService.java b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingCleanService.java
new file mode 100644
index 0000000..0533668
--- /dev/null
+++ b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingCleanService.java
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.broker.topic;
+
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.broker.out.BrokerOuterAPI;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.MixAll;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.admin.TopicOffset;
+import org.apache.rocketmq.common.admin.TopicStatsTable;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.common.protocol.RequestCode;
+import org.apache.rocketmq.common.protocol.header.GetTopicConfigRequestHeader;
+import org.apache.rocketmq.common.protocol.header.GetTopicStatsInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.route.TopicRouteData;
+import org.apache.rocketmq.common.rpc.ClientMetadata;
+import org.apache.rocketmq.common.rpc.RpcClient;
+import org.apache.rocketmq.common.rpc.RpcRequest;
+import org.apache.rocketmq.common.rpc.RpcResponse;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
+import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class TopicQueueMappingCleanService extends ServiceThread {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+
+ private TopicQueueMappingManager topicQueueMappingManager;
+ private BrokerOuterAPI brokerOuterAPI;
+ private RpcClient rpcClient;
+ private MessageStoreConfig messageStoreConfig;
+ private BrokerConfig brokerConfig;
+
+ public TopicQueueMappingCleanService(BrokerController brokerController) {
+ this.topicQueueMappingManager = brokerController.getTopicQueueMappingManager();
+ this.rpcClient = brokerController.getBrokerOuterAPI().getRpcClient();
+ this.messageStoreConfig = brokerController.getMessageStoreConfig();
+ this.brokerConfig = brokerController.getBrokerConfig();
+ this.brokerOuterAPI = brokerController.getBrokerOuterAPI();
+ }
+
+ @Override
+ public String getServiceName() {
+ return TopicQueueMappingCleanService.class.getSimpleName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start topic queue mapping clean service thread!");
+ while (!this.isStopped()) {
+ try {
+ this.waitForRunning(5L * 60 * 1000);
+ } catch (Throwable ignored) {
+ }
+ try {
+ cleanItemExpired();
+ } catch (Throwable t) {
+ log.error("topic queue mapping cleanItemExpired failed", t);
+ }
+ try {
+ cleanItemListMoreThanSecondGen();
+ } catch (Throwable t) {
+ log.error("topic queue mapping cleanItemListMoreThanSecondGen failed", t);
+ }
+
+ }
+ log.info("End topic queue mapping clean service thread!");
+ }
+
+
+
+ public void cleanItemExpired() {
+ String when = messageStoreConfig.getDeleteWhen();
+ if (!UtilAll.isItTimeToDo(when)) {
+ return;
+ }
+ boolean changed = false;
+ long start = System.currentTimeMillis();
+ try {
+ for (String topic : this.topicQueueMappingManager.getTopicQueueMappingTable().keySet()) {
+ try {
+ if (isStopped()) {
+ break;
+ }
+ TopicQueueMappingDetail mappingDetail = this.topicQueueMappingManager.getTopicQueueMappingTable().get(topic);
+ if (mappingDetail == null
+ || mappingDetail.getHostedQueues().isEmpty()) {
+ continue;
+ }
+ if (!mappingDetail.getBname().equals(brokerConfig.getBrokerName())) {
+ log.warn("The TopicQueueMappingDetail [{}] should not exist in this broker", mappingDetail);
+ continue;
+ }
+ Set<String> brokers = new HashSet<>();
+ for (List<LogicQueueMappingItem> items: mappingDetail.getHostedQueues().values()) {
+ if (items.size() <= 1) {
+ continue;
+ }
+ if (!TopicQueueMappingUtils.checkIfLeader(items, mappingDetail)) {
+ continue;
+ }
+ LogicQueueMappingItem earlistItem = items.get(0);
+ brokers.add(earlistItem.getBname());
+ }
+ Map<String, TopicStatsTable> statsTable = new HashMap<>();
+ for (String broker: brokers) {
+ GetTopicStatsInfoRequestHeader header = new GetTopicStatsInfoRequestHeader();
+ header.setTopic(topic);
+ header.setBname(broker);
+ header.setLo(false);
+ try {
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.GET_TOPIC_STATS_INFO, header, null);
+ RpcResponse rpcResponse = rpcClient.invoke(rpcRequest, brokerConfig.getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ statsTable.put(broker, (TopicStatsTable) rpcResponse.getBody());
+ } catch (Throwable rt) {
+ log.error("Get remote topic {} state info failed from broker {}", topic, broker, rt);
+ }
+ }
+ Map<Integer, List<LogicQueueMappingItem>> newHostedQueues = new HashMap<>();
+ boolean changedForTopic = false;
+ for (Map.Entry<Integer, List<LogicQueueMappingItem>> entry : mappingDetail.getHostedQueues().entrySet()) {
+ Integer qid = entry.getKey();
+ List<LogicQueueMappingItem> items = entry.getValue();
+ if (items.size() <= 1) {
+ continue;
+ }
+ if (!TopicQueueMappingUtils.checkIfLeader(items, mappingDetail)) {
+ continue;
+ }
+ LogicQueueMappingItem earlistItem = items.get(0);
+ TopicStatsTable topicStats = statsTable.get(earlistItem.getBname());
+ if (topicStats == null) {
+ continue;
+ }
+ TopicOffset topicOffset = topicStats.getOffsetTable().get(new MessageQueue(topic, earlistItem.getBname(), earlistItem.getQueueId()));
+ if (topicOffset == null) {
+ //this may should not happen
+ log.error("Get null topicOffset for {} {}",topic, earlistItem);
+ continue;
+ }
+ //ignore the maxOffset < 0, which may in case of some error
+ if (topicOffset.getMaxOffset() == topicOffset.getMinOffset()
+ || topicOffset.getMaxOffset() == 0) {
+ List<LogicQueueMappingItem> newItems = new ArrayList<>(items);
+ boolean result = newItems.remove(earlistItem);
+ if (result) {
+ changedForTopic = true;
+ newHostedQueues.put(qid, newItems);
+ }
+ log.info("The logic queue item {} {} is removed {} because of {}", topic, earlistItem, result, topicOffset);
+ }
+ }
+ if (changedForTopic) {
+ TopicQueueMappingDetail newMappingDetail = new TopicQueueMappingDetail(mappingDetail.getTopic(), mappingDetail.getTotalQueues(), mappingDetail.getBname(), mappingDetail.getEpoch());
+ newMappingDetail.getHostedQueues().putAll(mappingDetail.getHostedQueues());
+ newMappingDetail.getHostedQueues().putAll(newHostedQueues);
+ this.topicQueueMappingManager.updateTopicQueueMapping(newMappingDetail, false, true, false);
+ changed = true;
+ }
+ } catch (Throwable tt) {
+ log.error("Try CleanItemExpired failed for {}", topic, tt);
+ } finally {
+ UtilAll.sleep(10);
+ }
+ }
+ } catch (Throwable t) {
+ log.error("Try cleanItemExpired failed", t);
+ } finally {
+ if (changed) {
+ this.topicQueueMappingManager.getDataVersion().nextVersion();
+ this.topicQueueMappingManager.persist();
+ log.info("CleanItemExpired changed");
+ }
+ log.info("cleanItemExpired cost {} ms", System.currentTimeMillis() - start);
+ }
+ }
+
+ public void cleanItemListMoreThanSecondGen() {
+ String when = messageStoreConfig.getDeleteWhen();
+ if (!UtilAll.isItTimeToDo(when)) {
+ return;
+ }
+ boolean changed = false;
+ long start = System.currentTimeMillis();
+ try {
+ ClientMetadata clientMetadata = new ClientMetadata();
+ for (String topic : this.topicQueueMappingManager.getTopicQueueMappingTable().keySet()) {
+ try {
+ if (isStopped()) {
+ break;
+ }
+ TopicQueueMappingDetail mappingDetail = this.topicQueueMappingManager.getTopicQueueMappingTable().get(topic);
+ if (mappingDetail == null
+ || mappingDetail.getHostedQueues().isEmpty()) {
+ continue;
+ }
+ if (!mappingDetail.getBname().equals(brokerConfig.getBrokerName())) {
+ log.warn("The TopicQueueMappingDetail [{}] should not exist in this broker", mappingDetail);
+ continue;
+ }
+ Map<Integer, String> qid2CurrLeaderBroker = new HashMap<>();
+ for (Map.Entry<Integer, List<LogicQueueMappingItem>> entry : mappingDetail.getHostedQueues().entrySet()) {
+ Integer qId = entry.getKey();
+ List<LogicQueueMappingItem> items = entry.getValue();
+ if (items.isEmpty()) {
+ continue;
+ }
+ LogicQueueMappingItem leaderItem = items.get(items.size() - 1);
+ if (!leaderItem.getBname().equals(mappingDetail.getBname())) {
+ qid2CurrLeaderBroker.put(qId, leaderItem.getBname());
+ }
+ }
+ if (qid2CurrLeaderBroker.isEmpty()) {
+ continue;
+ }
+ //find the topic route
+ TopicRouteData topicRouteData = brokerOuterAPI.getTopicRouteInfoFromNameServer(topic, brokerConfig.getForwardTimeout());
+ clientMetadata.freshTopicRoute(topic, topicRouteData);
+ Map<Integer, String> qid2RealLeaderBroker = new HashMap<>();
+ //fine the real leader
+ for (Map.Entry<Integer, String> entry : qid2CurrLeaderBroker.entrySet()) {
+ qid2RealLeaderBroker.put(entry.getKey(), clientMetadata.getBrokerNameFromMessageQueue(new MessageQueue(topic, TopicQueueMappingUtils.getMockBrokerName(mappingDetail.getScope()), entry.getKey())));
+ }
+
+ //find the mapping detail of real leader
+ Map<String, TopicQueueMappingDetail> mappingDetailMap = new HashMap<>();
+ for (Map.Entry<Integer, String> entry : qid2RealLeaderBroker.entrySet()) {
+ if (entry.getValue().startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX)) {
+ continue;
+ }
+ String broker = entry.getValue();
+ GetTopicConfigRequestHeader header = new GetTopicConfigRequestHeader();
+ header.setTopic(topic);
+ header.setBname(broker);
+ header.setLo(true);
+ try {
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.GET_TOPIC_CONFIG, header, null);
+ RpcResponse rpcResponse = rpcClient.invoke(rpcRequest, brokerConfig.getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ TopicQueueMappingDetail mappingDetailRemote = ((TopicConfigAndQueueMapping) rpcResponse.getBody()).getMappingDetail();
+ if (broker.equals(mappingDetailRemote.getBname())) {
+ mappingDetailMap.put(broker, mappingDetailRemote);
+ }
+ } catch (Throwable rt) {
+ log.error("Get remote topic {} state info failed from broker {}", topic, broker, rt);
+ }
+ }
+ //check all the info
+ Set<Integer> ids2delete = new HashSet<>();
+ for (Map.Entry<Integer, String> entry : qid2CurrLeaderBroker.entrySet()) {
+ Integer qId = entry.getKey();
+ String currLeaderBroker = entry.getValue();
+ String realLeaderBroker = qid2RealLeaderBroker.get(qId);
+ TopicQueueMappingDetail remoteMappingDetail = mappingDetailMap.get(realLeaderBroker);
+ if (remoteMappingDetail == null
+ || remoteMappingDetail.getTotalQueues() != mappingDetail.getTotalQueues()
+ || remoteMappingDetail.getEpoch() != mappingDetail.getEpoch()) {
+ continue;
+ }
+ List<LogicQueueMappingItem> items = remoteMappingDetail.getHostedQueues().get(qId);
+ if (items.isEmpty()) {
+ continue;
+ }
+ LogicQueueMappingItem leaderItem = items.get(items.size() - 1);
+ if (!realLeaderBroker.equals(leaderItem.getBname())) {
+ continue;
+ }
+ //all the check is ok
+ if (!realLeaderBroker.equals(currLeaderBroker)) {
+ ids2delete.add(qId);
+ }
+ }
+ for (Integer qid : ids2delete) {
+ List<LogicQueueMappingItem> items = mappingDetail.getHostedQueues().remove(qid);
+ changed = true;
+ if (items != null) {
+ log.info("Remove the ItemListMoreThanSecondGen topic {} qid {} items {}", topic, qid, items);
+ }
+ }
+ } catch (Throwable tt) {
+ log.error("Try cleanItemListMoreThanSecondGen failed for topic {}", topic, tt);
+ } finally {
+ UtilAll.sleep(10);
+ }
+ }
+ } catch (Throwable t) {
+ log.error("Try cleanItemListMoreThanSecondGen failed", t);
+ } finally {
+ if (changed) {
+ this.topicQueueMappingManager.getDataVersion().nextVersion();
+ this.topicQueueMappingManager.persist();
+ }
+ log.info("Try cleanItemListMoreThanSecondGen cost {} ms", System.currentTimeMillis() - start);
+ }
+ }
+
+
+
+
+}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java
new file mode 100644
index 0000000..56fc792
--- /dev/null
+++ b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.broker.topic;
+
+import com.alibaba.fastjson.JSON;
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.broker.BrokerPathConfigHelper;
+import org.apache.rocketmq.common.ConfigManager;
+import org.apache.rocketmq.common.DataVersion;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.body.TopicQueueMappingSerializeWrapper;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
+import org.apache.rocketmq.common.rpc.TopicRequestHeader;
+import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import static org.apache.rocketmq.remoting.protocol.RemotingCommand.buildErrorResponse;
+
+public class TopicQueueMappingManager extends ConfigManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
+ private static final long LOCK_TIMEOUT_MILLIS = 3000;
+ private transient final Lock lock = new ReentrantLock();
+
+ //this data version should be equal to the TopicConfigManager
+ private final DataVersion dataVersion = new DataVersion();
+ private transient BrokerController brokerController;
+
+ private final ConcurrentMap<String, TopicQueueMappingDetail> topicQueueMappingTable = new ConcurrentHashMap<>();
+
+
+ public TopicQueueMappingManager(BrokerController brokerController) {
+ this.brokerController = brokerController;
+ }
+
+ public void updateTopicQueueMapping(TopicQueueMappingDetail newDetail, boolean force, boolean isClean, boolean flush) throws Exception {
+ boolean locked = false;
+ boolean updated = false;
+ TopicQueueMappingDetail oldDetail = null;
+ try {
+
+ if (lock.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
+ locked = true;
+ } else {
+ return;
+ }
+ if (newDetail == null) {
+ return;
+ }
+ assert newDetail.getBname().equals(this.brokerController.getBrokerConfig().getBrokerName());
+
+ newDetail.getHostedQueues().forEach((queueId, items) -> {
+ TopicQueueMappingUtils.checkLogicQueueMappingItemOffset(items);
+ });
+
+ oldDetail = topicQueueMappingTable.get(newDetail.getTopic());
+ if (oldDetail == null) {
+ topicQueueMappingTable.put(newDetail.getTopic(), newDetail);
+ updated = true;
+ return;
+ }
+ if (force) {
+ //bakeup the old items
+ oldDetail.getHostedQueues().forEach((queueId, items) -> {
+ newDetail.getHostedQueues().putIfAbsent(queueId, items);
+ });
+ topicQueueMappingTable.put(newDetail.getTopic(), newDetail);
+ updated = true;
+ return;
+ }
+ //do more check
+ if (newDetail.getEpoch() < oldDetail.getEpoch()) {
+ throw new RuntimeException(String.format("Can't accept data with small epoch %d < %d", newDetail.getEpoch(), oldDetail.getEpoch()));
+ }
+ if (!newDetail.getScope().equals(oldDetail.getScope())) {
+ throw new RuntimeException(String.format("Can't accept data with unmatched scope %s != %s", newDetail.getScope(), oldDetail.getScope()));
+ }
+ boolean epochEqual = newDetail.getEpoch() == oldDetail.getEpoch();
+ for (Integer globalId : oldDetail.getHostedQueues().keySet()) {
+ List<LogicQueueMappingItem> oldItems = oldDetail.getHostedQueues().get(globalId);
+ List<LogicQueueMappingItem> newItems = newDetail.getHostedQueues().get(globalId);
+ if (newItems == null) {
+ if (epochEqual) {
+ throw new RuntimeException("Cannot accept equal epoch with null data");
+ } else {
+ newDetail.getHostedQueues().put(globalId, oldItems);
+ }
+ } else {
+ TopicQueueMappingUtils.makeSureLogicQueueMappingItemImmutable(oldItems, newItems, epochEqual, isClean);
+ }
+ }
+ topicQueueMappingTable.put(newDetail.getTopic(), newDetail);
+ updated = true;
+ } finally {
+ if (locked) {
+ this.lock.unlock();
+ }
+ if (updated && flush) {
+ this.dataVersion.nextVersion();
+ this.persist();
+ log.info("Update topic queue mapping from [{}] to [{}], force {}", oldDetail, newDetail, force);
+ }
+ }
+
+ }
+
+ public void delete(final String topic) {
+ TopicQueueMappingDetail old = this.topicQueueMappingTable.remove(topic);
+ if (old != null) {
+ log.info("delete topic queue mapping OK, topic queue mapping: {}", old);
+ this.dataVersion.nextVersion();
+ this.persist();
+ } else {
+ log.warn("delete topic queue mapping failed, topic: {} not exists", topic);
+ }
+ }
+
+ public TopicQueueMappingDetail getTopicQueueMapping(String topic) {
+ return topicQueueMappingTable.get(topic);
+ }
+
+ @Override
+ public String encode(boolean pretty) {
+ TopicQueueMappingSerializeWrapper wrapper = new TopicQueueMappingSerializeWrapper();
+ wrapper.setTopicQueueMappingInfoMap(topicQueueMappingTable);
+ wrapper.setDataVersion(this.dataVersion);
+ return JSON.toJSONString(wrapper, pretty);
+ }
+
+ @Override
+ public String encode() {
+ return encode(false);
+ }
+
+ @Override
+ public String configFilePath() {
+ return BrokerPathConfigHelper.getTopicQueueMappingPath(this.brokerController.getMessageStoreConfig()
+ .getStorePathRootDir());
+ }
+
+ @Override
+ public void decode(String jsonString) {
+ if (jsonString != null) {
+ TopicQueueMappingSerializeWrapper wrapper = TopicQueueMappingSerializeWrapper.fromJson(jsonString, TopicQueueMappingSerializeWrapper.class);
+ if (wrapper != null) {
+ this.topicQueueMappingTable.putAll(wrapper.getTopicQueueMappingInfoMap());
+ this.dataVersion.assignNewOne(wrapper.getDataVersion());
+ }
+ }
+ }
+
+ public ConcurrentMap<String, TopicQueueMappingDetail> getTopicQueueMappingTable() {
+ return topicQueueMappingTable;
+ }
+
+ public DataVersion getDataVersion() {
+ return dataVersion;
+ }
+
+ public TopicQueueMappingContext buildTopicQueueMappingContext(TopicRequestHeader requestHeader) {
+ return buildTopicQueueMappingContext(requestHeader, false);
+ }
+
+ //Do not return a null context
+ public TopicQueueMappingContext buildTopicQueueMappingContext(TopicRequestHeader requestHeader, boolean selectOneWhenMiss) {
+ //should disable logic queue explicitly, otherwise the old client may cause dirty data to newly created static topic
+ if (requestHeader.getLo() != null
+ && Boolean.FALSE.equals(requestHeader.getLo())) {
+ return new TopicQueueMappingContext(requestHeader.getTopic(), null, null, null, null);
+ }
+ String topic = requestHeader.getTopic();
+ Integer globalId = null;
+ if (requestHeader instanceof TopicQueueRequestHeader) {
+ globalId = ((TopicQueueRequestHeader) requestHeader).getQueueId();
+ }
+
+ TopicQueueMappingDetail mappingDetail = getTopicQueueMapping(topic);
+ if (mappingDetail == null) {
+ //it is not static topic
+ return new TopicQueueMappingContext(topic, null, null, null, null);
+ }
+ assert mappingDetail.getBname().equals(this.brokerController.getBrokerConfig().getBrokerName());
+
+ if (globalId == null) {
+ return new TopicQueueMappingContext(topic, null, mappingDetail, null, null);
+ }
+
+ //If not find mappingItem, it encounters some errors
+ if (globalId < 0 && !selectOneWhenMiss) {
+ return new TopicQueueMappingContext(topic, globalId, mappingDetail, null, null);
+ }
+
+ if (globalId < 0) {
+ try {
+ if (!mappingDetail.getHostedQueues().isEmpty()) {
+ //do not check
+ globalId = mappingDetail.getHostedQueues().keySet().iterator().next();
+ }
+ } catch (Throwable ignored) {
+ }
+ }
+ if (globalId < 0) {
+ return new TopicQueueMappingContext(topic, globalId, mappingDetail, null, null);
+ }
+
+ List<LogicQueueMappingItem> mappingItemList = TopicQueueMappingDetail.getMappingInfo(mappingDetail, globalId);
+ LogicQueueMappingItem leaderItem = null;
+ if (mappingItemList != null
+ && mappingItemList.size() > 0) {
+ leaderItem = mappingItemList.get(mappingItemList.size() - 1);
+ }
+ return new TopicQueueMappingContext(topic, globalId, mappingDetail, mappingItemList, leaderItem);
+ }
+
+
+ public RemotingCommand rewriteRequestForStaticTopic(TopicQueueRequestHeader requestHeader, TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", requestHeader.getTopic(), requestHeader.getQueueId(), mappingDetail.getBname()));
+ }
+ LogicQueueMappingItem mappingItem = mappingContext.getLeaderItem();
+ requestHeader.setQueueId(mappingItem.getQueueId());
+ return null;
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
+}
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/BrokerOuterAPITest.java b/broker/src/test/java/org/apache/rocketmq/broker/BrokerOuterAPITest.java
index 339ed11..daf771a 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/BrokerOuterAPITest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/BrokerOuterAPITest.java
@@ -84,7 +84,7 @@ public class BrokerOuterAPITest {
private BrokerOuterAPI brokerOuterAPI;
public void init() throws Exception {
- brokerOuterAPI = new BrokerOuterAPI(new NettyClientConfig(), null);
+ brokerOuterAPI = new BrokerOuterAPI(new NettyClientConfig(), brokerController);
Field field = BrokerOuterAPI.class.getDeclaredField("remotingClient");
field.setAccessible(true);
field.set(brokerOuterAPI, nettyRemotingClient);
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/BrokerStartupTest.java b/broker/src/test/java/org/apache/rocketmq/broker/BrokerStartupTest.java
index c8da08d..ce370a3 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/BrokerStartupTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/BrokerStartupTest.java
@@ -20,6 +20,7 @@ package org.apache.rocketmq.broker;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Properties;
+import org.apache.rocketmq.common.MixAll;
import org.junit.Assert;
import org.junit.Test;
@@ -34,8 +35,21 @@ public class BrokerStartupTest {
Class<BrokerStartup> clazz = BrokerStartup.class;
Method method = clazz.getDeclaredMethod("properties2SystemEnv", Properties.class);
method.setAccessible(true);
- System.setProperty("rocketmq.namesrv.domain", "value");
- method.invoke(null, properties);
- Assert.assertEquals("value", System.getProperty("rocketmq.namesrv.domain"));
+ {
+ properties.put("rmqAddressServerDomain", "value1");
+ properties.put("rmqAddressServerSubGroup", "value2");
+ method.invoke(null, properties);
+ Assert.assertEquals("value1", System.getProperty("rocketmq.namesrv.domain"));
+ Assert.assertEquals("value2", System.getProperty("rocketmq.namesrv.domain.subgroup"));
+ }
+ {
+ properties.put("rmqAddressServerDomain", MixAll.WS_DOMAIN_NAME);
+ properties.put("rmqAddressServerSubGroup", MixAll.WS_DOMAIN_SUBGROUP);
+ method.invoke(null, properties);
+ Assert.assertEquals(MixAll.WS_DOMAIN_NAME, System.getProperty("rocketmq.namesrv.domain"));
+ Assert.assertEquals(MixAll.WS_DOMAIN_SUBGROUP, System.getProperty("rocketmq.namesrv.domain.subgroup"));
+ }
+
+
}
}
\ No newline at end of file
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
index d6cc8f9..2141f2c 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
@@ -16,22 +16,10 @@
*/
package org.apache.rocketmq.broker.processor;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
import com.google.common.collect.Sets;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
-import java.net.InetSocketAddress;
-import java.net.UnknownHostException;
-import java.nio.ByteBuffer;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.atomic.LongAdder;
import org.apache.rocketmq.broker.BrokerController;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.topic.TopicConfigManager;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
@@ -42,28 +30,17 @@ import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
-import org.apache.rocketmq.common.protocol.body.CreateMessageQueueForLogicalQueueRequestBody;
-import org.apache.rocketmq.common.protocol.body.MigrateLogicalQueueBody;
-import org.apache.rocketmq.common.protocol.body.ReuseTopicLogicalQueueRequestBody;
-import org.apache.rocketmq.common.protocol.body.SealTopicLogicalQueueRequestBody;
-import org.apache.rocketmq.common.protocol.body.UpdateTopicLogicalQueueMappingRequestBody;
import org.apache.rocketmq.common.protocol.header.CreateTopicRequestHeader;
-import org.apache.rocketmq.common.protocol.header.DeleteTopicLogicalQueueRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.GetTopicConfigRequestHeader;
-import org.apache.rocketmq.common.protocol.header.QueryTopicLogicalQueueMappingRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResumeCheckHalfMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
import org.apache.rocketmq.store.AppendMessageResult;
import org.apache.rocketmq.store.AppendMessageStatus;
import org.apache.rocketmq.store.MappedFile;
@@ -73,7 +50,6 @@ import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.assertj.core.util.Lists;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -81,13 +57,16 @@ import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
+import java.net.InetSocketAddress;
+import java.net.UnknownHostException;
+import java.nio.ByteBuffer;
+import java.util.Set;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.LongAdder;
+
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
-import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
@@ -122,8 +101,7 @@ public class AdminBrokerProcessorTest {
public void init() throws Exception {
brokerController.setMessageStore(messageStore);
- doReturn(sendMessageProcessor).when(brokerController).getSendMessageProcessor();
- when(sendMessageProcessor.getInFlyWritingCounterMap()).thenReturn(inFlyWritingCouterMap);
+ //doReturn(sendMessageProcessor).when(brokerController).getSendMessageProcessor();
adminBrokerProcessor = new AdminBrokerProcessor(brokerController);
@@ -225,271 +203,13 @@ public class AdminBrokerProcessorTest {
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_TOPIC_CONFIG, requestHeader);
request.makeCustomHeaderToNet();
RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
+ assertThat(response.getCode()).isEqualTo(ResponseCode.TOPIC_NOT_EXIST);
assertThat(response.getRemark()).contains("No topic in this broker.");
}
}
- @Test
- public void testUpdateTopicLogicalQueueMapping() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
-
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_TOPIC_LOGICAL_QUEUE_MAPPING, null);
- UpdateTopicLogicalQueueMappingRequestBody requestBody = new UpdateTopicLogicalQueueMappingRequestBody();
- requestBody.setTopic(topic);
- requestBody.setQueueId(queueId);
- requestBody.setLogicalQueueIdx(logicalQueueIndex);
- request.setBody(requestBody.encode());
- RemotingCommand response;
- response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(topicConfigManager.getOrCreateLogicalQueuesInfo(topic).get(logicalQueueIndex).get(0)).isEqualTo(new LogicalQueueRouteData(logicalQueueIndex, 0L, new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId), MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr()));
-
- // delete
- requestBody.setLogicalQueueIdx(-1);
- request.setBody(requestBody.encode());
- response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(topicConfigManager.getOrCreateLogicalQueuesInfo(topic).get(logicalQueueIndex)).isEmpty();
- verify(inFlyWritingCouterMap).remove(new TopicQueueId(topic, queueId));
- }
-
- @Test
- public void testDeleteTopicLogicalQueueMapping() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(logicalQueueIndex, 0L, new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId), MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr())));
-
- DeleteTopicLogicalQueueRequestHeader requestHeader = new DeleteTopicLogicalQueueRequestHeader();
- requestHeader.setTopic(topic);
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.DELETE_TOPIC_LOGICAL_QUEUE_MAPPING, requestHeader);
- request.makeCustomHeaderToNet();
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
- assertThat(response.getRemark()).isEqualTo("still 1 message queues");
-
- logicalQueuesInfo.remove(logicalQueueIndex);
- response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(topicConfigManager.selectLogicalQueuesInfo(topic)).isNull();
- }
-
- @Test
- public void testQueryTopicLogicalQueueMapping() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(logicalQueueIndex, 0L, new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId), MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr())));
-
- QueryTopicLogicalQueueMappingRequestHeader requestHeader = new QueryTopicLogicalQueueMappingRequestHeader();
- requestHeader.setTopic(topic);
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.QUERY_TOPIC_LOGICAL_QUEUE_MAPPING, requestHeader);
- request.makeCustomHeaderToNet();
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- Map<Integer, List<LogicalQueueRouteData>> m = JSON.parseObject(response.getBody(), new TypeReference<Map<Integer, List<LogicalQueueRouteData>>>() {
- }.getType());
- assertThat(m.get(logicalQueueIndex)).isEqualTo(logicalQueuesInfo.get(logicalQueueIndex));
- }
-
- @Test
- public void testSealTopicLogicalQueue() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- MessageQueue mq = new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId);
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr())));
-
- when(messageStore.getMaxOffsetInQueue(eq(topic), eq(queueId), anyBoolean())).thenReturn(100L);
- when(messageStore.getMinOffsetInQueue(eq(topic), eq(queueId))).thenReturn(0L);
- when(messageStore.getMinPhyOffset()).thenReturn(1000L);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId), eq(0L))).thenReturn(2000L);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId), eq(99L))).thenReturn(3000L);
- MessageExt firstMsg = mock(MessageExt.class);
- when(firstMsg.getStoreTimestamp()).thenReturn(200L);
- when(messageStore.lookMessageByOffset(eq(2000L))).thenReturn(firstMsg);
- MessageExt lastMsg = mock(MessageExt.class);
- when(lastMsg.getStoreTimestamp()).thenReturn(300L);
- when(messageStore.lookMessageByOffset(eq(3000L))).thenReturn(lastMsg);
-
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.SEAL_TOPIC_LOGICAL_QUEUE, null);
- SealTopicLogicalQueueRequestBody requestBody = new SealTopicLogicalQueueRequestBody();
- requestBody.setTopic(topic);
- requestBody.setQueueId(queueId);
- requestBody.setLogicalQueueIndex(logicalQueueIndex);
- request.setBody(requestBody.encode());
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- LogicalQueueRouteData wantLogicalQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.ReadOnly, 0, 100, 200, 300, brokerController.getBrokerAddr());
- assertThat(logicalQueuesInfo.get(logicalQueueIndex).get(0)).isEqualTo(wantLogicalQueueRouteData);
- assertThat((LogicalQueueRouteData) JSON.parseObject(response.getBody(), LogicalQueueRouteData.class)).isEqualTo(wantLogicalQueueRouteData);
-
- // expired
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr())));
- when(messageStore.getMinPhyOffset()).thenReturn(10000L);
- response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- wantLogicalQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.Expired, 0, 100, 0, 0, brokerController.getBrokerAddr());
- assertThat(logicalQueuesInfo.get(logicalQueueIndex).get(0)).isEqualTo(wantLogicalQueueRouteData);
- assertThat((LogicalQueueRouteData) JSON.parseObject(response.getBody(), LogicalQueueRouteData.class)).isEqualTo(wantLogicalQueueRouteData);
-
- // expired and empty
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.Normal, 0L, -1, -1, -1, brokerController.getBrokerAddr())));
- when(messageStore.getMinOffsetInQueue(eq(topic), eq(queueId))).thenReturn(100L);
- response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- wantLogicalQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 0L, mq, MessageQueueRouteState.Expired, 0, 100, 0, 0, brokerController.getBrokerAddr());
- assertThat(logicalQueuesInfo.get(logicalQueueIndex).get(0)).isEqualTo(wantLogicalQueueRouteData);
- assertThat((LogicalQueueRouteData) JSON.parseObject(response.getBody(), LogicalQueueRouteData.class)).isEqualTo(wantLogicalQueueRouteData);
- }
-
- @Test
- public void testReuseTopicLogicalQueue() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- MessageQueue mq = new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId);
- LogicalQueueRouteData logicalQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 500L, mq, MessageQueueRouteState.Expired, 100L, 200L, 300L, 400L, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(logicalQueueRouteData));
- LogicalQueueRouteData wantData0 = new LogicalQueueRouteData(logicalQueueRouteData);
-
- when(messageStore.getMaxOffsetInQueue(eq(topic), eq(queueId), anyBoolean())).thenReturn(600L);
-
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.REUSE_TOPIC_LOGICAL_QUEUE, null);
- ReuseTopicLogicalQueueRequestBody requestBody = new ReuseTopicLogicalQueueRequestBody();
- requestBody.setTopic(topic);
- requestBody.setQueueId(queueId);
- requestBody.setLogicalQueueIndex(logicalQueueIndex);
- requestBody.setMessageQueueRouteState(MessageQueueRouteState.WriteOnly);
- request.setBody(requestBody.encode());
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- LogicalQueueRouteData wantData1 = new LogicalQueueRouteData(logicalQueueIndex, -1L, mq, MessageQueueRouteState.WriteOnly, 600L, -1, -1, -1, brokerController.getBrokerAddr());
- assertThat((LogicalQueueRouteData) JSON.parseObject(response.getBody(), LogicalQueueRouteData.class)).isEqualTo(wantData1);
- assertThat(logicalQueuesInfo.get(logicalQueueIndex)).isEqualTo(Arrays.asList(wantData0, wantData1));
- verify(inFlyWritingCouterMap).remove(new TopicQueueId(topic, queueId));
- }
-
- @Test
- public void testCreateMessageQueueForLogicalQueue() throws Exception {
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- TopicConfig topicConfig = topicConfigManager.selectTopicConfig(topic);
- topicConfig.setWriteQueueNums(0);
- topicConfig.setReadQueueNums(0);
- int queueId = 0;
- assertThat(topicConfigManager.selectLogicalQueuesInfo(topic)).isNull();
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.CREATE_MESSAGE_QUEUE_FOR_LOGICAL_QUEUE, null);
- CreateMessageQueueForLogicalQueueRequestBody requestBody = new CreateMessageQueueForLogicalQueueRequestBody();
- requestBody.setTopic(topic);
- requestBody.setLogicalQueueIndex(logicalQueueIndex);
- requestBody.setMessageQueueStatus(MessageQueueRouteState.WriteOnly);
- request.setBody(requestBody.encode());
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).withFailMessage("remark: %s", response.getRemark()).isEqualTo(ResponseCode.SUCCESS);
- LogicalQueueRouteData wantLogicalQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, -1L, new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId), MessageQueueRouteState.WriteOnly, 0L, -1, -1, -1, brokerController.getBrokerAddr());
- assertThat((LogicalQueueRouteData) JSON.parseObject(response.getBody(), LogicalQueueRouteData.class)).isEqualTo(wantLogicalQueueRouteData);
- assertThat(topicConfigManager.selectLogicalQueuesInfo(topic).get(logicalQueueIndex).get(0)).isEqualTo(wantLogicalQueueRouteData);
- }
-
- @Test
- public void testMigrateTopicLogicalQueuePrepare() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- MessageQueue mq = new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId);
- LogicalQueueRouteData fromQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 500L, mq, MessageQueueRouteState.Normal, 10L, -1L, -1L, -1L, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(fromQueueRouteData)));
-
- when(messageStore.getMaxOffsetInQueue(eq(topic), eq(queueId), anyBoolean())).thenReturn(100L);
- when(messageStore.getMinOffsetInQueue(eq(topic), eq(queueId))).thenReturn(10L);
- when(messageStore.getMinPhyOffset()).thenReturn(1000L);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId), eq(10L))).thenReturn(2000L);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId), eq(99L))).thenReturn(3000L);
- MessageExt firstMsg = mock(MessageExt.class);
- when(firstMsg.getStoreTimestamp()).thenReturn(200L);
- when(messageStore.lookMessageByOffset(eq(2000L))).thenReturn(firstMsg);
- MessageExt lastMsg = mock(MessageExt.class);
- when(lastMsg.getStoreTimestamp()).thenReturn(300L);
- when(messageStore.lookMessageByOffset(eq(3000L))).thenReturn(lastMsg);
-
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_PREPARE, null);
- MigrateLogicalQueueBody requestBody = new MigrateLogicalQueueBody();
- requestBody.setFromQueueRouteData(fromQueueRouteData);
- LogicalQueueRouteData toQueueRouteData = new LogicalQueueRouteData();
- toQueueRouteData.setMessageQueue(new MessageQueue(topic, "toBroker", 1));
- requestBody.setToQueueRouteData(toQueueRouteData);
- request.setBody(requestBody.encode());
-
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).withFailMessage("remark: %s", response.getRemark()).isEqualTo(ResponseCode.SUCCESS);
- fromQueueRouteData.setState(MessageQueueRouteState.ReadOnly);
- fromQueueRouteData.setOffsetMax(100L);
- fromQueueRouteData.setFirstMsgTimeMillis(200L);
- fromQueueRouteData.setLastMsgTimeMillis(300L);
- toQueueRouteData.setLogicalQueueDelta(590L);
- MigrateLogicalQueueBody responseBody = RemotingSerializable.decode(response.getBody(), MigrateLogicalQueueBody.class);
- assertThat(responseBody.getFromQueueRouteData()).isEqualTo(fromQueueRouteData);
- assertThat(responseBody.getToQueueRouteData()).isEqualTo(toQueueRouteData);
- assertThat(logicalQueuesInfo.get(logicalQueueIndex)).isEqualTo(Lists.newArrayList(fromQueueRouteData, toQueueRouteData));
- }
-
- @Test
- public void testMigrateTopicLogicalQueueCommit() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- MessageQueue mq = new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), queueId);
- LogicalQueueRouteData fromQueueRouteData = new LogicalQueueRouteData();
- fromQueueRouteData.setMessageQueue(new MessageQueue(topic, "fromBroker", 0));
- LogicalQueueRouteData toQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 500L, mq, MessageQueueRouteState.Normal, 500L, -1L, -1L, -1L, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(toQueueRouteData)));
-
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_COMMIT, null);
- MigrateLogicalQueueBody requestBody = new MigrateLogicalQueueBody();
- requestBody.setFromQueueRouteData(fromQueueRouteData);
- requestBody.setToQueueRouteData(toQueueRouteData);
- request.setBody(requestBody.encode());
-
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).withFailMessage("remark: %s", response.getRemark()).isEqualTo(ResponseCode.SUCCESS);
- MigrateLogicalQueueBody responseBody = RemotingSerializable.decode(response.getBody(), MigrateLogicalQueueBody.class);
- assertThat(responseBody.getFromQueueRouteData()).isEqualTo(fromQueueRouteData);
- assertThat(responseBody.getToQueueRouteData()).isEqualTo(toQueueRouteData);
- assertThat(logicalQueuesInfo.get(logicalQueueIndex)).isEqualTo(Lists.newArrayList(toQueueRouteData));
- }
-
- @Test
- public void testMigrateTopicLogicalQueueNotify() throws Exception {
- int queueId = 0;
- int logicalQueueIndex = 0;
- TopicConfigManager topicConfigManager = brokerController.getTopicConfigManager();
- LogicalQueuesInfoInBroker logicalQueuesInfo = topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- LogicalQueueRouteData fromQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 100L, new MessageQueue(topic, "fromBroker", queueId), MessageQueueRouteState.ReadOnly, 10L, 410L, 200L, 300L, brokerController.getBrokerAddr());
- LogicalQueueRouteData toQueueRouteData = new LogicalQueueRouteData(logicalQueueIndex, 500L, new MessageQueue(topic, "toBroker", queueId), MessageQueueRouteState.Normal, 500L, -1L, -1L, -1L, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(logicalQueueIndex, Lists.newArrayList(new LogicalQueueRouteData(fromQueueRouteData)));
-
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_NOTIFY, null);
- MigrateLogicalQueueBody requestBody = new MigrateLogicalQueueBody();
- requestBody.setFromQueueRouteData(fromQueueRouteData);
- requestBody.setToQueueRouteData(toQueueRouteData);
- request.setBody(requestBody.encode());
-
- RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
- assertThat(response.getCode()).withFailMessage("remark: %s", response.getRemark()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(logicalQueuesInfo.get(logicalQueueIndex)).isEqualTo(Lists.newArrayList(fromQueueRouteData, toQueueRouteData));
- }
private RemotingCommand buildCreateTopicRequest(String topic) {
CreateTopicRequestHeader requestHeader = new CreateTopicRequestHeader();
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/PullMessageProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/PullMessageProcessorTest.java
index b637755..8d3f624 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/PullMessageProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/PullMessageProcessorTest.java
@@ -18,33 +18,21 @@ package org.apache.rocketmq.broker.processor;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ClientChannelInfo;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.filter.ExpressionMessageFilter;
import org.apache.rocketmq.broker.mqtrace.ConsumeMessageContext;
import org.apache.rocketmq.broker.mqtrace.ConsumeMessageHook;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
-import org.apache.rocketmq.common.message.MessageConst;
-import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.header.PullMessageResponseHeader;
import org.apache.rocketmq.common.protocol.heartbeat.ConsumeType;
import org.apache.rocketmq.common.protocol.heartbeat.ConsumerData;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
@@ -53,7 +41,6 @@ import org.apache.rocketmq.store.GetMessageResult;
import org.apache.rocketmq.store.GetMessageStatus;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.assertj.core.util.Lists;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -61,15 +48,17 @@ import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
-import static java.util.Optional.ofNullable;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.ArgumentMatchers.intThat;
-import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@@ -205,94 +194,6 @@ public class PullMessageProcessorTest {
assertThat(response.getCode()).isEqualTo(ResponseCode.PULL_OFFSET_MOVED);
}
- @Test
- public void testProcessRequest_LogicalQueue() throws Exception {
- String brokerName = brokerController.getBrokerConfig().getBrokerName();
- int queueId = 1;
-
- GetMessageResult getMessageResult = createGetMessageResult();
- when(messageStore.getMessage(anyString(), eq(topic), eq(queueId), eq(456L), anyInt(), any(ExpressionMessageFilter.class))).thenReturn(getMessageResult);
- when(messageStore.getMaxOffsetInQueue(eq(topic), eq(queueId))).thenReturn(2000L);
- when(messageStore.getMinPhyOffset()).thenReturn(0L);
-
- LogicalQueuesInfoInBroker logicalQueuesInfo = brokerController.getTopicConfigManager().getOrCreateLogicalQueuesInfo(topic);
- LogicalQueueRouteData queueRouteData1 = new LogicalQueueRouteData(0, 0, new MessageQueue(topic, brokerName, queueId), MessageQueueRouteState.Normal, 0, -1, -1, -1, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(0, Lists.newArrayList(queueRouteData1));
- logicalQueuesInfo.updateQueueRouteDataByQueueId(queueRouteData1.getQueueId(), queueRouteData1);
-
- // normal
- {
- final RemotingCommand request = createPullMsgCommand(RequestCode.PULL_MESSAGE);
- RemotingCommand response = pullMessageProcessor.processRequest(handlerContext, request);
- assertThat(response).isNotNull();
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- }
- // write only
- queueRouteData1.setState(MessageQueueRouteState.WriteOnly);
- {
- final RemotingCommand request = createPullMsgCommand(RequestCode.PULL_MESSAGE);
- RemotingCommand response = pullMessageProcessor.processRequest(handlerContext, request);
- assertThat(response).isNotNull();
- assertThat(response.getCode()).isEqualTo(ResponseCode.PULL_NOT_FOUND);
- }
- // no message and redirect
- queueRouteData1.setState(MessageQueueRouteState.ReadOnly);
- queueRouteData1.setOffsetMax(460);
- queueRouteData1.setFirstMsgTimeMillis(100);
- queueRouteData1.setLastMsgTimeMillis(200);
- LogicalQueueRouteData queueRouteData2 = new LogicalQueueRouteData(0, 460, new MessageQueue(topic, "broker2", 1), MessageQueueRouteState.Normal, 0, -1, -1, -1, brokerController.getBrokerAddr());
- logicalQueuesInfo.get(0).add(queueRouteData2);
- getMessageResult.setStatus(GetMessageStatus.OFFSET_FOUND_NULL);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId), eq(460L - 1L))).thenReturn(1000L);
- {
- final RemotingCommand request = createPullMsgCommand(RequestCode.PULL_MESSAGE);
- RemotingCommand response = pullMessageProcessor.processRequest(handlerContext, request);
- assertThat(response).isNotNull();
- assertThat(response.getCode()).isEqualTo(ResponseCode.PULL_NOT_FOUND);
- assertThat(response.getExtFields()).containsKey(MessageConst.PROPERTY_REDIRECT);
- }
- // same message queue has two routes
- queueRouteData2.setState(MessageQueueRouteState.ReadOnly);
- queueRouteData2.setOffsetMax(50);
- queueRouteData2.setFirstMsgTimeMillis(300);
- queueRouteData2.setLastMsgTimeMillis(400);
- LogicalQueueRouteData queueRouteData3 = new LogicalQueueRouteData(0, 510, new MessageQueue(topic, queueRouteData2.getBrokerName(), queueId), MessageQueueRouteState.Normal, 460, -1, -1, -1, queueRouteData1.getBrokerAddr());
- logicalQueuesInfo.get(0).add(queueRouteData3);
- logicalQueuesInfo.updateQueueRouteDataByQueueId(queueRouteData3.getQueueId(), queueRouteData3);
- {
- GetMessageResult getMessageResult2 = createGetMessageResult();
- getMessageResult2.setStatus(GetMessageStatus.FOUND);
- getMessageResult2.setNextBeginOffset(460);
- when(messageStore.getMessage(anyString(), eq(queueRouteData1.getTopic()), eq(queueRouteData1.getQueueId()), eq(456L), eq(4), any(ExpressionMessageFilter.class))).thenReturn(getMessageResult2);
- }
- {
- GetMessageResult getMessageResult2 = createGetMessageResult();
- getMessageResult2.setStatus(GetMessageStatus.FOUND);
- getMessageResult2.setNextBeginOffset(470);
- lenient().when(messageStore.getMessage(anyString(), eq(queueRouteData1.getTopic()), eq(queueRouteData1.getQueueId()), eq(456L), intThat(i -> i > 4), any(ExpressionMessageFilter.class))).thenReturn(getMessageResult2);
- }
- {
- final RemotingCommand request = createPullMsgCommand(RequestCode.PULL_MESSAGE);
- RemotingCommand response = pullMessageProcessor.processRequest(handlerContext, request);
- assertThat(response).isNotNull();
- assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(ofNullable(response.getExtFields()).orElse(new HashMap<>())).doesNotContainKey(MessageConst.PROPERTY_REDIRECT);
- PullMessageResponseHeader header = (PullMessageResponseHeader) response.readCustomHeader();
- assertThat(header.getNextBeginOffset()).isEqualTo(460);
- }
- {
- when(messageStore.getMinPhyOffset()).thenReturn(100000L);
-
- final RemotingCommand request = createPullMsgCommand(RequestCode.PULL_MESSAGE);
- RemotingCommand response = pullMessageProcessor.processRequest(handlerContext, request);
- assertThat(response).isNotNull();
- assertThat(response.getCode()).isEqualTo(ResponseCode.PULL_RETRY_IMMEDIATELY);
- assertThat(ofNullable(response.getExtFields()).orElse(new HashMap<>())).containsKey(MessageConst.PROPERTY_REDIRECT);
- PullMessageResponseHeader header = (PullMessageResponseHeader) response.readCustomHeader();
- assertThat(header.getNextBeginOffset()).isEqualTo(460);
- }
- }
-
private RemotingCommand createPullMsgCommand(int requestCode) {
PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
requestHeader.setCommitOffset(123L);
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/SendMessageProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/SendMessageProcessorTest.java
index e768469..b4ae34d 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/SendMessageProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/SendMessageProcessorTest.java
@@ -18,14 +18,7 @@ package org.apache.rocketmq.broker.processor;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.TimeUnit;
import org.apache.rocketmq.broker.BrokerController;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
import org.apache.rocketmq.broker.mqtrace.SendMessageContext;
import org.apache.rocketmq.broker.mqtrace.SendMessageHook;
import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
@@ -35,13 +28,10 @@ import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.ConsumerSendMsgBackRequestHeader;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
@@ -55,7 +45,6 @@ import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.assertj.core.util.Lists;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -65,6 +54,12 @@ import org.mockito.invocation.InvocationOnMock;
import org.mockito.junit.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
@@ -229,62 +224,6 @@ public class SendMessageProcessorTest {
assertThat(response[0].getCode()).isEqualTo(ResponseCode.SUCCESS);
}
-
- @Test
- public void testProcessRequest_LogicalQueue() throws Exception {
- when(messageStore.asyncPutMessage(any(MessageExtBrokerInner.class)))
- .thenReturn(CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.PUT_OK, new AppendMessageResult(AppendMessageStatus.PUT_OK))));
-
- LogicalQueuesInfoInBroker logicalQueuesInfo = brokerController.getTopicConfigManager().getOrCreateLogicalQueuesInfo(topic);
- LogicalQueueRouteData queueRouteData1 = new LogicalQueueRouteData(0, 0, new MessageQueue(topic, brokerController.getBrokerConfig().getBrokerName(), 1), MessageQueueRouteState.Normal, 0, -1, -1, -1, brokerController.getBrokerAddr());
- logicalQueuesInfo.put(0, Lists.newArrayList(queueRouteData1));
- logicalQueuesInfo.updateQueueRouteDataByQueueId(queueRouteData1.getQueueId(), queueRouteData1);
-
- SendMessageRequestHeader requestHeader = createSendMsgRequestHeader();
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
- request.setBody(new byte[] {'a'});
- request.makeCustomHeaderToNet();
-
- // normal
- RemotingCommand responseToReturn;
- {
- CompletableFuture<RemotingCommand> responseFuture = new CompletableFuture<>();
- doAnswer(invocation -> {
- responseFuture.complete(invocation.getArgument(0));
- return null;
- }).when(handlerContext).writeAndFlush(any(Object.class));
-
- responseToReturn = sendMessageProcessor.processRequest(handlerContext, request);
- if (responseToReturn == null) {
- responseToReturn = responseFuture.get(3, TimeUnit.SECONDS);
- }
- }
-
- assertThat(responseToReturn.getCode()).isEqualTo(ResponseCode.SUCCESS);
- assertThat(responseToReturn.getOpaque()).isEqualTo(request.getOpaque());
-
- // read only
- queueRouteData1.setState(MessageQueueRouteState.ReadOnly);
- responseToReturn = sendMessageProcessor.processRequest(handlerContext, request);
- assertThat(responseToReturn.getCode()).isEqualTo(ResponseCode.NO_PERMISSION);
- assertThat(responseToReturn.getRemark()).contains("not writable");
-
- // read only and forward
- logicalQueuesInfo.get(0).add(new LogicalQueueRouteData(0, 100, new MessageQueue(topic, "broker2", 1), MessageQueueRouteState.Normal, 0, -1, -1, -1, brokerController.getBrokerAddr()));
-
- responseToReturn = sendMessageProcessor.processRequest(handlerContext, request);
- assertThat(responseToReturn.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
- assertThat(responseToReturn.getRemark()).contains("forward error");
-
- // read only and redirect
- requestHeader = (SendMessageRequestHeader) request.readCustomHeader();
- requestHeader.setSysFlag(MessageSysFlag.LOGICAL_QUEUE_FLAG);
- request.makeCustomHeaderToNet();
- responseToReturn = sendMessageProcessor.processRequest(handlerContext, request);
- assertThat(responseToReturn.getCode()).isEqualTo(ResponseCode.NO_PERMISSION);
- assertThat(responseToReturn.getExtFields()).containsKey(MessageConst.PROPERTY_REDIRECT);
- }
-
private RemotingCommand createSendTransactionMsgCommand(int requestCode) {
SendMessageRequestHeader header = createSendMsgRequestHeader();
int sysFlag = header.getSysFlag();
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
deleted file mode 100644
index f504965..0000000
--- a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
+++ /dev/null
@@ -1,138 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.rocketmq.broker.topic;
-
-import com.google.common.collect.Lists;
-import java.nio.file.Files;
-import java.nio.file.Paths;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import org.apache.rocketmq.broker.BrokerController;
-import org.apache.rocketmq.broker.domain.LogicalQueuesInfoInBroker;
-import org.apache.rocketmq.common.BrokerConfig;
-import org.apache.rocketmq.common.DataVersion;
-import org.apache.rocketmq.common.TopicConfig;
-import org.apache.rocketmq.common.message.MessageQueue;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.junit.MockitoJUnitRunner;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyLong;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.Mockito.never;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-
-@RunWith(MockitoJUnitRunner.class)
-public class TopicConfigManagerTest {
- @Mock
- private DefaultMessageStore messageStore;
- @Mock
- private BrokerController brokerController;
-
- private TopicConfigManager topicConfigManager;
-
- private static final String topic = "FooBar";
- private static final String broker1Name = "broker1";
- private static final String broker1Addr = "127.0.0.1:12345";
- private static final int queueId1 = 1;
- private static final String broker2Name = "broker2";
- private static final String broker2Addr = "127.0.0.2:12345";
- private static final int queueId2 = 2;
-
- @Before
- public void before() {
- BrokerConfig brokerConfig = new BrokerConfig();
- brokerConfig.setBrokerName(broker1Name);
- when(brokerController.getBrokerConfig()).thenReturn(brokerConfig);
-
- when(brokerController.getMessageStore()).thenReturn(messageStore);
-
- MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
- messageStoreConfig.setStorePathRootDir(System.getProperty("java.io.tmpdir"));
- when(brokerController.getMessageStoreConfig()).thenReturn(messageStoreConfig);
-
- this.topicConfigManager = new TopicConfigManager(brokerController);
- this.topicConfigManager.getTopicConfigTable().put(topic, new TopicConfig(topic));
- }
-
- @After
- public void after() throws Exception {
- if (topicConfigManager != null) {
- Files.deleteIfExists(Paths.get(topicConfigManager.configFilePath()));
- }
- }
-
- @Test
- public void logicalQueueCleanTest() {
- LogicalQueuesInfoInBroker info = this.topicConfigManager.getOrCreateLogicalQueuesInfo(topic);
- topicConfigManager.logicalQueueClean();
- assertThat(info).isEmpty();
-
- final int logicalQueueIndex = 0;
- LogicalQueueRouteData queueRouteData1 = new LogicalQueueRouteData(logicalQueueIndex, 0, new MessageQueue(topic, broker1Name, queueId1), MessageQueueRouteState.Normal, 0, -1, -1, -1, broker1Addr);
- List<LogicalQueueRouteData> l = Lists.newArrayList(new LogicalQueueRouteData(queueRouteData1));
- info.put(logicalQueueIndex, l);
-
- topicConfigManager.logicalQueueClean();
- assertThat(info.get(logicalQueueIndex)).isEqualTo(Collections.singletonList(queueRouteData1));
- verify(messageStore, never()).getCommitLogOffsetInQueue(eq(topic), eq(queueId1), anyLong());
- verify(messageStore, never()).getMinPhyOffset();
- verify(brokerController, never()).registerIncrementBrokerData(ArgumentMatchers.<TopicConfig>argThat(arg -> topic.equals(arg.getTopicName())), any(DataVersion.class));
-
- LogicalQueueRouteData queueRouteData2 = new LogicalQueueRouteData(logicalQueueIndex, 100, new MessageQueue(topic, broker2Name, queueId2), MessageQueueRouteState.Normal, 0, -1, -1, -1, broker2Addr);
- l.add(new LogicalQueueRouteData(queueRouteData2));
- queueRouteData1 = l.get(0);
- queueRouteData1.setState(MessageQueueRouteState.ReadOnly);
- queueRouteData1.setOffsetMax(100);
- queueRouteData1.setFirstMsgTimeMillis(200);
- queueRouteData1.setLastMsgTimeMillis(300);
- queueRouteData1 = new LogicalQueueRouteData(queueRouteData1);
- LogicalQueueRouteData queueRouteData3 = new LogicalQueueRouteData(logicalQueueIndex, 200, new MessageQueue(topic, broker1Name, queueId1), MessageQueueRouteState.Normal, 100, -1, -1, -1, broker1Addr);
- l.add(new LogicalQueueRouteData(queueRouteData3));
- queueRouteData2 = l.get(1);
- queueRouteData2.setState(MessageQueueRouteState.ReadOnly);
- queueRouteData2.setOffsetMax(100);
- queueRouteData2.setFirstMsgTimeMillis(400);
- queueRouteData2.setLastMsgTimeMillis(500);
- queueRouteData2 = new LogicalQueueRouteData(queueRouteData2);
- when(messageStore.getCommitLogOffsetInQueue(eq(topic), eq(queueId1), eq(queueRouteData1.getOffsetMax() - 1))).thenReturn(1000L);
- when(messageStore.getMinPhyOffset()).thenReturn(0L);
- topicConfigManager.logicalQueueClean();
- assertThat(info.get(logicalQueueIndex)).isEqualTo(Arrays.asList(queueRouteData1, queueRouteData2, queueRouteData3));
- verify(messageStore).getCommitLogOffsetInQueue(eq(topic), eq(queueId1), eq(queueRouteData1.getOffsetMax() - 1));
- verify(messageStore).getMinPhyOffset();
- verify(brokerController, never()).registerIncrementBrokerData(ArgumentMatchers.<TopicConfig>argThat(arg -> topic.equals(arg.getTopicName())), any(DataVersion.class));
-
- when(messageStore.getMinPhyOffset()).thenReturn(2000L);
- topicConfigManager.logicalQueueClean();
- assertThat(info.get(logicalQueueIndex)).isEqualTo(Collections.singletonList(queueRouteData3));
- verify(brokerController).registerIncrementBrokerData(ArgumentMatchers.<TopicConfig>argThat(arg -> topic.equals(arg.getTopicName())), any(DataVersion.class));
- }
-}
\ No newline at end of file
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManagerTest.java b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManagerTest.java
new file mode 100644
index 0000000..6b4faab
--- /dev/null
+++ b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManagerTest.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.broker.topic;
+
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
+import org.apache.rocketmq.common.statictopic.TopicRemappingDetailWrapper;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnitRunner;
+
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import static org.mockito.Mockito.when;
+
+@RunWith(MockitoJUnitRunner.class)
+public class TopicQueueMappingManagerTest {
+ @Mock
+ private BrokerController brokerController;
+ private static final String broker1Name = "broker1";
+
+ @Before
+ public void before() {
+ BrokerConfig brokerConfig = new BrokerConfig();
+ brokerConfig.setBrokerName(broker1Name);
+ when(brokerController.getBrokerConfig()).thenReturn(brokerConfig);
+
+ MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+ messageStoreConfig.setStorePathRootDir(System.getProperty("java.io.tmpdir"));
+ messageStoreConfig.setDeleteWhen("01;02;03;04;05;06;07;08;09;10;11;12;13;14;15;16;17;18;19;20;21;22;23;00");
+ when(brokerController.getMessageStoreConfig()).thenReturn(messageStoreConfig);
+ }
+
+
+ private void delete(TopicQueueMappingManager topicQueueMappingManager) throws Exception {
+ if (topicQueueMappingManager == null) {
+ return;
+ }
+ Files.deleteIfExists(Paths.get(topicQueueMappingManager.configFilePath()));
+ Files.deleteIfExists(Paths.get(topicQueueMappingManager.configFilePath() + ".bak"));
+
+
+ }
+
+ @Test
+ public void testEncodeDecode() throws Exception {
+ Map<String, TopicQueueMappingDetail> mappingDetailMap = new HashMap<>();
+ TopicQueueMappingManager topicQueueMappingManager = null;
+ Set<String> brokers = new HashSet<String>();
+ brokers.add(broker1Name);
+ {
+ for (int i = 0; i < 10; i++) {
+ String topic = UUID.randomUUID().toString();
+ int queueNum = 10;
+ TopicRemappingDetailWrapper topicRemappingDetailWrapper = TopicQueueMappingUtils.createTopicConfigMapping(topic, queueNum, brokers, new HashMap<>());
+ Assert.assertEquals(1, topicRemappingDetailWrapper.getBrokerConfigMap().size());
+ TopicQueueMappingDetail topicQueueMappingDetail = topicRemappingDetailWrapper.getBrokerConfigMap().values().iterator().next().getMappingDetail();
+ Assert.assertEquals(queueNum, topicQueueMappingDetail.getHostedQueues().size());
+ mappingDetailMap.put(topic, topicQueueMappingDetail);
+ }
+ }
+
+ {
+ topicQueueMappingManager = new TopicQueueMappingManager(brokerController);
+ Assert.assertTrue(topicQueueMappingManager.load());
+ Assert.assertEquals(0, topicQueueMappingManager.getTopicQueueMappingTable().size());
+ for (TopicQueueMappingDetail mappingDetail : mappingDetailMap.values()) {
+ for (int i = 0; i < 10; i++) {
+ topicQueueMappingManager.updateTopicQueueMapping(mappingDetail, false, false, true);
+ }
+ }
+ topicQueueMappingManager.persist();
+ }
+
+ {
+ topicQueueMappingManager = new TopicQueueMappingManager(brokerController);
+ Assert.assertTrue(topicQueueMappingManager.load());
+ Assert.assertEquals(mappingDetailMap.size(), topicQueueMappingManager.getTopicQueueMappingTable().size());
+ for (TopicQueueMappingDetail topicQueueMappingDetail: topicQueueMappingManager.getTopicQueueMappingTable().values()) {
+ Assert.assertEquals(topicQueueMappingDetail, mappingDetailMap.get(topicQueueMappingDetail.getTopic()));
+ }
+ }
+ delete(topicQueueMappingManager);
+ }
+}
\ No newline at end of file
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
index cf8cbb0..8a6340b 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
@@ -691,7 +691,7 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
public void sendMessageBack(MessageExt msg, int delayLevel)
throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
msg.setTopic(withNamespace(msg.getTopic()));
- this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, null);
+ this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, (String) null);
}
/**
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/PullResult.java b/client/src/main/java/org/apache/rocketmq/client/consumer/PullResult.java
index 30d9952..d887542 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/PullResult.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/PullResult.java
@@ -26,6 +26,7 @@ public class PullResult {
private final long maxOffset;
private List<MessageExt> msgFoundList;
+
public PullResult(PullStatus pullStatus, long nextBeginOffset, long minOffset, long maxOffset,
List<MessageExt> msgFoundList) {
super();
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStore.java b/client/src/main/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStore.java
index 6b76238..fc0b29c 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStore.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStore.java
@@ -16,26 +16,28 @@
*/
package org.apache.rocketmq.client.consumer.store;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.atomic.AtomicLong;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.client.exception.OffsetNotFoundException;
import org.apache.rocketmq.client.impl.FindBrokerResult;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.client.log.ClientLogger;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.UtilAll;
-import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.exception.RemotingException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+
/**
* Remote storage implementation
*/
@@ -94,7 +96,7 @@ public class RemoteBrokerOffsetStore implements OffsetStore {
return brokerOffset;
}
// No offset in broker
- catch (MQBrokerException e) {
+ catch (OffsetNotFoundException e) {
return -1;
}
//Other exceptions
@@ -108,7 +110,7 @@ public class RemoteBrokerOffsetStore implements OffsetStore {
}
}
- return -1;
+ return -3;
}
@Override
@@ -199,10 +201,10 @@ public class RemoteBrokerOffsetStore implements OffsetStore {
@Override
public void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
MQBrokerException, InterruptedException, MQClientException {
- FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
+ FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
}
if (findBrokerResult != null) {
@@ -226,11 +228,11 @@ public class RemoteBrokerOffsetStore implements OffsetStore {
private long fetchConsumeOffsetFromBroker(MessageQueue mq) throws RemotingException, MQBrokerException,
InterruptedException, MQClientException {
- FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
+ FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
}
if (findBrokerResult != null) {
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/DeleteTopicLogicalQueueRequestHeader.java b/client/src/main/java/org/apache/rocketmq/client/exception/OffsetNotFoundException.java
similarity index 59%
rename from common/src/main/java/org/apache/rocketmq/common/protocol/header/DeleteTopicLogicalQueueRequestHeader.java
rename to client/src/main/java/org/apache/rocketmq/client/exception/OffsetNotFoundException.java
index fa8d50d..e73bbbf 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/DeleteTopicLogicalQueueRequestHeader.java
+++ b/client/src/main/java/org/apache/rocketmq/client/exception/OffsetNotFoundException.java
@@ -14,24 +14,18 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.common.protocol.header;
+package org.apache.rocketmq.client.exception;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
-import org.apache.rocketmq.remoting.annotation.CFNotNull;
-import org.apache.rocketmq.remoting.exception.RemotingCommandException;
+public class OffsetNotFoundException extends MQBrokerException {
-public class DeleteTopicLogicalQueueRequestHeader implements CommandCustomHeader {
- @CFNotNull
- private String topic;
-
- @Override public void checkFields() throws RemotingCommandException {
+ public OffsetNotFoundException() {
}
- public String getTopic() {
- return topic;
+ public OffsetNotFoundException(int responseCode, String errorMessage) {
+ super(responseCode, errorMessage);
}
- public void setTopic(String topic) {
- this.topic = topic;
+ public OffsetNotFoundException(int responseCode, String errorMessage, String brokerAddr) {
+ super(responseCode, errorMessage, brokerAddr);
}
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
index dce830c..0a6e005 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
@@ -16,34 +16,40 @@
*/
package org.apache.rocketmq.client.impl;
-import com.google.common.base.Objects;
-import com.google.common.collect.Lists;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
import org.apache.rocketmq.client.QueryResult;
import org.apache.rocketmq.client.Validators;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
-import org.apache.rocketmq.client.exception.MQRedirectException;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.client.impl.producer.TopicPublishInfo;
import org.apache.rocketmq.client.log.ClientLogger;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.help.FAQUrl;
+import org.apache.rocketmq.common.protocol.NamespaceUtil;
+import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.common.message.MessageClientIDSetter;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageId;
import org.apache.rocketmq.common.message.MessageQueue;
-import org.apache.rocketmq.common.protocol.NamespaceUtil;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.QueryMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.QueryMessageResponseHeader;
import org.apache.rocketmq.common.protocol.route.BrokerData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
-import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.InvokeCallback;
import org.apache.rocketmq.remoting.common.RemotingUtil;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
@@ -51,17 +57,6 @@ import org.apache.rocketmq.remoting.exception.RemotingException;
import org.apache.rocketmq.remoting.netty.ResponseFuture;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.locks.ReadWriteLock;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-
public class MQAdminImpl {
private final InternalLogger log = ClientLogger.getLog();
@@ -187,21 +182,16 @@ public class MQAdminImpl {
}
public long searchOffset(MessageQueue mq, long timestamp) throws MQClientException {
- LogicalQueueRouteData logicalQueueRouteData = searchLogicalQueueRouteByTimestamp(mq, timestamp);
- if (logicalQueueRouteData != null) {
- mq = logicalQueueRouteData.getMessageQueue();
- }
- String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
if (null == brokerAddr) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
}
if (brokerAddr != null) {
try {
- long offset = this.mQClientFactory.getMQClientAPIImpl().searchOffset(brokerAddr, mq.getTopic(), mq.getQueueId(), timestamp,
+ return this.mQClientFactory.getMQClientAPIImpl().searchOffset(brokerAddr, mq.getTopic(), mq.getQueueId(), timestamp,
timeoutMillis);
- return correctLogicalQueueOffset(offset, logicalQueueRouteData);
} catch (Exception e) {
throw new MQClientException("Invoke Broker[" + brokerAddr + "] exception", e);
}
@@ -211,60 +201,33 @@ public class MQAdminImpl {
}
public long maxOffset(MessageQueue mq) throws MQClientException {
- return this.maxOffset(mq, true);
- }
-
- public long maxOffset(MessageQueue mq, boolean committed) throws MQClientException {
- final MessageQueue origMq = mq;
- String topic = mq.getTopic();
- LogicalQueueRouteData previousQueueRouteData = null;
- for (int i = 0; i < 5; i++) {
- LogicalQueueRouteData maxQueueRouteData = this.searchLogicalQueueRouteByOffset(origMq, Long.MAX_VALUE);
- if (maxQueueRouteData != null) {
- if (previousQueueRouteData != null && Objects.equal(previousQueueRouteData.getMessageQueue(), maxQueueRouteData.getMessageQueue())) {
- throw new MQClientException("Topic route info not latest", null);
- }
- previousQueueRouteData = maxQueueRouteData;
- mq = maxQueueRouteData.getMessageQueue();
- }
- String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
- if (null == brokerAddr) {
- this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
- brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
- }
+ String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
+ if (null == brokerAddr) {
+ this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
+ brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
+ }
- if (brokerAddr != null) {
- try {
- long offset = this.mQClientFactory.getMQClientAPIImpl().getMaxOffset(brokerAddr, topic, mq.getQueueId(), committed, maxQueueRouteData != null, timeoutMillis);
- return correctLogicalQueueOffset(offset, maxQueueRouteData);
- } catch (MQRedirectException e) {
- this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, false, null, Collections.singleton(mq.getQueueId()));
- continue;
- } catch (Exception e) {
- throw new MQClientException("Invoke Broker[" + brokerAddr + "] exception", e);
- }
+ if (brokerAddr != null) {
+ try {
+ return this.mQClientFactory.getMQClientAPIImpl().getMaxOffset(brokerAddr, mq.getTopic(), mq.getQueueId(), timeoutMillis);
+ } catch (Exception e) {
+ throw new MQClientException("Invoke Broker[" + brokerAddr + "] exception", e);
}
- throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
- throw new MQClientException("Redirect exceed max times", null);
+
+ throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
public long minOffset(MessageQueue mq) throws MQClientException {
- LogicalQueueRouteData minQueueRouteData = searchLogicalQueueRouteByOffset(mq, 0L);
- if (minQueueRouteData != null) {
- mq = minQueueRouteData.getMessageQueue();
- }
-
- String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
if (null == brokerAddr) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
}
if (brokerAddr != null) {
try {
- long offset = this.mQClientFactory.getMQClientAPIImpl().getMinOffset(brokerAddr, mq.getTopic(), mq.getQueueId(), timeoutMillis);
- return correctLogicalQueueOffset(offset, minQueueRouteData);
+ return this.mQClientFactory.getMQClientAPIImpl().getMinOffset(brokerAddr, mq.getTopic(), mq.getQueueId(), timeoutMillis);
} catch (Exception e) {
throw new MQClientException("Invoke Broker[" + brokerAddr + "] exception", e);
}
@@ -273,33 +236,11 @@ public class MQAdminImpl {
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
- private List<LogicalQueueRouteData> queryLogicalQueueRouteData(MessageQueue mq) {
- if (MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME.equals(mq.getBrokerName())) {
- TopicRouteData topicRouteData = this.mQClientFactory.queryTopicRouteData(mq.getTopic());
- if (topicRouteData == null) {
- this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- topicRouteData = this.mQClientFactory.queryTopicRouteData(mq.getTopic());
- }
- if (topicRouteData != null) {
- LogicalQueuesInfo logicalQueuesInfo = topicRouteData.getLogicalQueuesInfo();
- if (logicalQueuesInfo != null) {
- return logicalQueuesInfo.get(mq.getQueueId());
- }
- }
- }
- return null;
- }
-
public long earliestMsgStoreTime(MessageQueue mq) throws MQClientException {
- LogicalQueueRouteData minQueueRouteData = searchLogicalQueueRouteByOffset(mq, 0L);
- if (minQueueRouteData != null) {
- mq = minQueueRouteData.getMessageQueue();
- }
-
- String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
if (null == brokerAddr) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(this.mQClientFactory.getBrokerNameFromMessageQueue(mq));
}
if (brokerAddr != null) {
@@ -504,71 +445,4 @@ public class MQAdminImpl {
throw new MQClientException(ResponseCode.TOPIC_NOT_EXIST, "The topic[" + topic + "] not matched route info");
}
-
- private static long correctLogicalQueueOffset(long offset, LogicalQueueRouteData logicalQueueRouteData) {
- if (logicalQueueRouteData == null) {
- return offset;
- }
- return logicalQueueRouteData.toLogicalQueueOffset(offset);
- }
-
- private LogicalQueueRouteData searchLogicalQueueRouteByTimestamp(MessageQueue mq, long timestamp) {
- List<LogicalQueueRouteData> queueRouteDataList = this.queryLogicalQueueRouteData(mq);
- if (queueRouteDataList == null) {
- return null;
- }
- LogicalQueueRouteData logicalQueueRouteData = null;
- for (LogicalQueueRouteData el : queueRouteDataList) {
- if (!el.isReadable()) {
- continue;
- }
- if (logicalQueueRouteData == null && el.getFirstMsgTimeMillis() < 0) {
- logicalQueueRouteData = el;
- } else if (el.getFirstMsgTimeMillis() >= 0) {
- if (el.getFirstMsgTimeMillis() <= timestamp && el.getLastMsgTimeMillis() >= timestamp) {
- logicalQueueRouteData = el;
- break;
- }
- }
- }
- if (logicalQueueRouteData == null) {
- logicalQueueRouteData = queueRouteDataList.get(queueRouteDataList.size() - 1);
- }
- return logicalQueueRouteData;
- }
-
- private LogicalQueueRouteData searchLogicalQueueRouteByOffset(MessageQueue mq, long offset) {
- List<LogicalQueueRouteData> queueRouteDataList = this.queryLogicalQueueRouteData(mq);
- if (queueRouteDataList == null) {
- return null;
- }
- {
- List<LogicalQueueRouteData> list = Lists.newArrayListWithCapacity(queueRouteDataList.size());
- for (LogicalQueueRouteData queueRouteData : queueRouteDataList) {
- if (LogicalQueueRouteData.READABLE_PREDICT.apply(queueRouteData)) {
- list.add(queueRouteData);
- }
- }
- queueRouteDataList = list;
- }
- if (queueRouteDataList.isEmpty()) {
- return null;
- }
- if (offset <= 0) {
- // min
- return Collections.min(queueRouteDataList);
- } else if (offset == Long.MAX_VALUE) {
- // max
- return Collections.max(queueRouteDataList);
- }
- Collections.sort(queueRouteDataList);
- LogicalQueueRouteData searchKey = new LogicalQueueRouteData();
- searchKey.setLogicalQueueDelta(offset);
- int idx = Collections.binarySearch(queueRouteDataList, searchKey);
- if (idx < 0) {
- idx = -idx - 1;
- idx -= 1;
- }
- return queueRouteDataList.get(idx);
- }
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
index 114fd0c..9823e81 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
@@ -16,7 +16,6 @@
*/
package org.apache.rocketmq.client.impl;
-import com.google.common.base.Function;
import java.io.UnsupportedEncodingException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
@@ -42,7 +41,7 @@ import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
-import org.apache.rocketmq.client.exception.MQRedirectException;
+import org.apache.rocketmq.client.exception.OffsetNotFoundException;
import org.apache.rocketmq.client.hook.SendMessageContext;
import org.apache.rocketmq.client.impl.consumer.PullResultExt;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
@@ -82,13 +81,11 @@ import org.apache.rocketmq.common.protocol.body.ConsumeMessageDirectlyResult;
import org.apache.rocketmq.common.protocol.body.ConsumeStatsList;
import org.apache.rocketmq.common.protocol.body.ConsumerConnection;
import org.apache.rocketmq.common.protocol.body.ConsumerRunningInfo;
-import org.apache.rocketmq.common.protocol.body.CreateMessageQueueForLogicalQueueRequestBody;
import org.apache.rocketmq.common.protocol.body.GetConsumerStatusBody;
import org.apache.rocketmq.common.protocol.body.GroupList;
import org.apache.rocketmq.common.protocol.body.KVTable;
import org.apache.rocketmq.common.protocol.body.LockBatchRequestBody;
import org.apache.rocketmq.common.protocol.body.LockBatchResponseBody;
-import org.apache.rocketmq.common.protocol.body.MigrateLogicalQueueBody;
import org.apache.rocketmq.common.protocol.body.ProducerConnection;
import org.apache.rocketmq.common.protocol.body.QueryAssignmentRequestBody;
import org.apache.rocketmq.common.protocol.body.QueryAssignmentResponseBody;
@@ -98,13 +95,10 @@ import org.apache.rocketmq.common.protocol.body.QueryCorrectionOffsetBody;
import org.apache.rocketmq.common.protocol.body.QueueTimeSpan;
import org.apache.rocketmq.common.protocol.body.ResetOffsetBody;
import org.apache.rocketmq.common.protocol.body.SetMessageRequestModeRequestBody;
-import org.apache.rocketmq.common.protocol.body.ReuseTopicLogicalQueueRequestBody;
-import org.apache.rocketmq.common.protocol.body.SealTopicLogicalQueueRequestBody;
import org.apache.rocketmq.common.protocol.body.SubscriptionGroupWrapper;
import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
import org.apache.rocketmq.common.protocol.body.TopicList;
import org.apache.rocketmq.common.protocol.body.UnlockBatchRequestBody;
-import org.apache.rocketmq.common.protocol.body.UpdateTopicLogicalQueueMappingRequestBody;
import org.apache.rocketmq.common.protocol.header.AckMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.ChangeInvisibleTimeRequestHeader;
import org.apache.rocketmq.common.protocol.header.ChangeInvisibleTimeResponseHeader;
@@ -115,7 +109,6 @@ import org.apache.rocketmq.common.protocol.header.CreateAccessConfigRequestHeade
import org.apache.rocketmq.common.protocol.header.CreateTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteAccessConfigRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteSubscriptionGroupRequestHeader;
-import org.apache.rocketmq.common.protocol.header.DeleteTopicLogicalQueueRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.EndTransactionRequestHeader;
import org.apache.rocketmq.common.protocol.header.ExtraInfoUtil;
@@ -149,7 +142,6 @@ import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetResponseHea
import org.apache.rocketmq.common.protocol.header.QueryCorrectionOffsetHeader;
import org.apache.rocketmq.common.protocol.header.QueryMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.QueryTopicConsumeByWhoRequestHeader;
-import org.apache.rocketmq.common.protocol.header.QueryTopicLogicalQueueMappingRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResetOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResumeCheckHalfMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.SearchOffsetRequestHeader;
@@ -176,13 +168,10 @@ import org.apache.rocketmq.common.protocol.header.namesrv.WipeWritePermOfBrokerR
import org.apache.rocketmq.common.protocol.heartbeat.HeartbeatData;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
-import org.apache.rocketmq.common.protocol.route.TopicRouteDataNameSrv;
+import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
-import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.CommandCustomHeader;
import org.apache.rocketmq.remoting.InvokeCallback;
@@ -201,8 +190,6 @@ import org.apache.rocketmq.remoting.protocol.LanguageCode;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-import static com.google.common.base.Optional.fromNullable;
-
public class MQClientAPIImpl {
private final static InternalLogger log = ClientLogger.getLog();
@@ -335,7 +322,7 @@ public class MQClientAPIImpl {
public void createTopic(final String addr, final String defaultTopic, final TopicConfig topicConfig,
final long timeoutMillis)
- throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
+ throws RemotingException, InterruptedException, MQClientException {
CreateTopicRequestHeader requestHeader = new CreateTopicRequestHeader();
requestHeader.setTopic(topicConfig.getTopicName());
requestHeader.setDefaultTopic(defaultTopic);
@@ -624,13 +611,9 @@ public class MQClientAPIImpl {
producer.updateFaultItem(brokerName, System.currentTimeMillis() - responseFuture.getBeginTimestamp(), false);
} catch (Exception e) {
- if (e instanceof MQRedirectException) {
- sendCallback.onException(e);
- } else {
- producer.updateFaultItem(brokerName, System.currentTimeMillis() - responseFuture.getBeginTimestamp(), true);
- onExceptionImpl(brokerName, msg, timeoutMillis - cost, request, sendCallback, topicPublishInfo, instance,
- retryTimesWhenSendFailed, times, e, context, false, producer);
- }
+ producer.updateFaultItem(brokerName, System.currentTimeMillis() - responseFuture.getBeginTimestamp(), true);
+ onExceptionImpl(brokerName, msg, timeoutMillis - cost, request, sendCallback, topicPublishInfo, instance,
+ retryTimesWhenSendFailed, times, e, context, false, producer);
}
} else {
producer.updateFaultItem(brokerName, System.currentTimeMillis() - responseFuture.getBeginTimestamp(), true);
@@ -672,7 +655,7 @@ public class MQClientAPIImpl {
String retryBrokerName = brokerName;//by default, it will send to the same broker
if (topicPublishInfo != null) { //select one message queue accordingly, in order to determine which broker to send
MessageQueue mqChosen = producer.selectOneMessageQueue(topicPublishInfo, brokerName);
- retryBrokerName = mqChosen.getBrokerName();
+ retryBrokerName = instance.getBrokerNameFromMessageQueue(mqChosen);
}
String addr = instance.findBrokerAddressInPublish(retryBrokerName);
log.warn(String.format("async send msg by retry {} times. topic={}, brokerAddr={}, brokerName={}", tmp, msg.getTopic(), addr,
@@ -709,11 +692,6 @@ public class MQClientAPIImpl {
final RemotingCommand response,
final String addr
) throws MQBrokerException, RemotingCommandException {
- HashMap<String, String> extFields = response.getExtFields();
- if (extFields != null && extFields.containsKey(MessageConst.PROPERTY_REDIRECT)) {
- throw new MQRedirectException(response.getBody());
- }
-
SendStatus sendStatus;
switch (response.getCode()) {
case ResponseCode.FLUSH_DISK_TIMEOUT: {
@@ -962,11 +940,6 @@ public class MQClientAPIImpl {
private PullResult processPullResponse(
final RemotingCommand response,
final String addr) throws MQBrokerException, RemotingCommandException {
- HashMap<String, String> extFields = response.getExtFields();
- if (extFields != null && extFields.containsKey(MessageConst.PROPERTY_REDIRECT)) {
- throw new MQRedirectException(response.getBody());
- }
-
PullStatus pullStatus = PullStatus.NO_NEW_MSG;
switch (response.getCode()) {
case ResponseCode.SUCCESS:
@@ -981,6 +954,7 @@ public class MQClientAPIImpl {
case ResponseCode.PULL_OFFSET_MOVED:
pullStatus = PullStatus.OFFSET_ILLEGAL;
break;
+
default:
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
@@ -989,7 +963,7 @@ public class MQClientAPIImpl {
(PullMessageResponseHeader) response.decodeCommandCustomHeader(PullMessageResponseHeader.class);
return new PullResultExt(pullStatus, responseHeader.getNextBeginOffset(), responseHeader.getMinOffset(),
- responseHeader.getMaxOffset(), null, responseHeader.getSuggestWhichBrokerId(), response.getBody());
+ responseHeader.getMaxOffset(), null, responseHeader.getSuggestWhichBrokerId(), response.getBody(), responseHeader.getOffsetDelta());
}
private PopResult processPopResponse(final String brokerName, final RemotingCommand response, String topic,
@@ -1133,27 +1107,14 @@ public class MQClientAPIImpl {
public long getMaxOffset(final String addr, final String topic, final int queueId, final long timeoutMillis)
throws RemotingException, MQBrokerException, InterruptedException {
- return getMaxOffset(addr, topic, queueId, true, false, timeoutMillis);
- }
-
- public long getMaxOffset(final String addr, final String topic, final int queueId, boolean committed,
- boolean fromLogicalQueue,
- final long timeoutMillis)
- throws RemotingException, MQBrokerException, InterruptedException {
GetMaxOffsetRequestHeader requestHeader = new GetMaxOffsetRequestHeader();
requestHeader.setTopic(topic);
requestHeader.setQueueId(queueId);
- requestHeader.setCommitted(committed);
- requestHeader.setLogicalQueue(fromLogicalQueue);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_MAX_OFFSET, requestHeader);
RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(this.clientConfig.isVipChannelEnabled(), addr),
request, timeoutMillis);
assert response != null;
- HashMap<String, String> extFields = response.getExtFields();
- if (extFields != null && extFields.containsKey(MessageConst.PROPERTY_REDIRECT)) {
- throw new MQRedirectException(response.getBody());
- }
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
GetMaxOffsetResponseHeader responseHeader =
@@ -1258,9 +1219,11 @@ public class MQClientAPIImpl {
case ResponseCode.SUCCESS: {
QueryConsumerOffsetResponseHeader responseHeader =
(QueryConsumerOffsetResponseHeader) response.decodeCommandCustomHeader(QueryConsumerOffsetResponseHeader.class);
-
return responseHeader.getOffset();
}
+ case ResponseCode.PULL_NOT_FOUND: {
+ throw new OffsetNotFoundException(response.getCode(), response.getRemark(), addr);
+ }
default:
break;
}
@@ -1649,15 +1612,8 @@ public class MQClientAPIImpl {
public TopicRouteData getTopicRouteInfoFromNameServer(final String topic, final long timeoutMillis,
boolean allowTopicNotExist) throws MQClientException, InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- return getTopicRouteInfoFromNameServer(topic, timeoutMillis, allowTopicNotExist, null);
- }
-
- public TopicRouteData getTopicRouteInfoFromNameServer(final String topic, final long timeoutMillis,
- boolean allowTopicNotExist, Set<Integer> logicalQueueIdsFilter) throws MQClientException, InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
GetRouteInfoRequestHeader requestHeader = new GetRouteInfoRequestHeader();
requestHeader.setTopic(topic);
- requestHeader.setSysFlag(MessageSysFlag.LOGICAL_QUEUE_FLAG);
- requestHeader.setLogicalQueueIdsFilter(logicalQueueIdsFilter);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ROUTEINFO_BY_TOPIC, requestHeader);
@@ -1674,11 +1630,7 @@ public class MQClientAPIImpl {
case ResponseCode.SUCCESS: {
byte[] body = response.getBody();
if (body != null) {
- return fromNullable(RemotingSerializable.decode(body, TopicRouteDataNameSrv.class)).transform(new Function<TopicRouteDataNameSrv, TopicRouteData>() {
- @Override public TopicRouteData apply(TopicRouteDataNameSrv srv) {
- return srv.toTopicRouteData();
- }
- }).orNull();
+ return TopicRouteData.decode(body, TopicRouteData.class);
}
}
default:
@@ -2609,147 +2561,58 @@ public class MQClientAPIImpl {
}
}
- public LogicalQueuesInfo queryTopicLogicalQueue(String brokerAddr, String topic,
- long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- QueryTopicLogicalQueueMappingRequestHeader requestHeader = new QueryTopicLogicalQueueMappingRequestHeader();
- requestHeader.setTopic(topic);
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.QUERY_TOPIC_LOGICAL_QUEUE_MAPPING, requestHeader);
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- return RemotingSerializable.decode(response.getBody(), LogicalQueuesInfo.class);
- }
-
- public void updateTopicLogicalQueue(String brokerAddr, String topic, int queueId, int logicalQueueIndex,
- long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_TOPIC_LOGICAL_QUEUE_MAPPING, null);
- UpdateTopicLogicalQueueMappingRequestBody requestBody = new UpdateTopicLogicalQueueMappingRequestBody();
- requestBody.setTopic(topic);
- requestBody.setQueueId(queueId);
- requestBody.setLogicalQueueIdx(logicalQueueIndex);
- request.setBody(requestBody.encode());
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- }
-
- public void deleteTopicLogicalQueueMapping(String brokerAddr, String topic, long timeoutMillis) throws MQBrokerException, InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- DeleteTopicLogicalQueueRequestHeader requestHeader = new DeleteTopicLogicalQueueRequestHeader();
- requestHeader.setTopic(topic);
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.DELETE_TOPIC_LOGICAL_QUEUE_MAPPING, requestHeader);
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- }
-
- public LogicalQueueRouteData sealTopicLogicalQueue(String brokerAddr, LogicalQueueRouteData queueRouteData, long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.SEAL_TOPIC_LOGICAL_QUEUE, null);
- SealTopicLogicalQueueRequestBody requestBody = new SealTopicLogicalQueueRequestBody();
- MessageQueue messageQueue = queueRouteData.getMessageQueue();
- requestBody.setTopic(messageQueue.getTopic());
- requestBody.setQueueId(messageQueue.getQueueId());
- requestBody.setLogicalQueueIndex(queueRouteData.getLogicalQueueIndex());
- request.setBody(requestBody.encode());
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- return RemotingSerializable.decode(response.getBody(), LogicalQueueRouteData.class);
- }
-
- public LogicalQueueRouteData reuseTopicLogicalQueue(String brokerAddr, String topic, int queueId,
- int logicalQueueIdx,
- MessageQueueRouteState messageQueueRouteState, long timeoutMillis) throws InterruptedException, MQBrokerException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.REUSE_TOPIC_LOGICAL_QUEUE, null);
- ReuseTopicLogicalQueueRequestBody requestBody = new ReuseTopicLogicalQueueRequestBody();
- requestBody.setTopic(topic);
- requestBody.setQueueId(queueId);
- requestBody.setLogicalQueueIndex(logicalQueueIdx);
- requestBody.setMessageQueueRouteState(messageQueueRouteState);
- request.setBody(requestBody.encode());
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- return RemotingSerializable.decode(response.getBody(), LogicalQueueRouteData.class);
- }
-
- public LogicalQueueRouteData createMessageQueueForLogicalQueue(String brokerAddr, String topic, int logicalQueueIdx,
- MessageQueueRouteState messageQueueStatus,
- long timeoutMillis) throws InterruptedException, MQBrokerException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.CREATE_MESSAGE_QUEUE_FOR_LOGICAL_QUEUE, null);
- CreateMessageQueueForLogicalQueueRequestBody requestBody = new CreateMessageQueueForLogicalQueueRequestBody();
- requestBody.setTopic(topic);
- requestBody.setLogicalQueueIndex(logicalQueueIdx);
- requestBody.setMessageQueueStatus(messageQueueStatus);
- request.setBody(requestBody.encode());
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- return RemotingSerializable.decode(response.getBody(), LogicalQueueRouteData.class);
- }
- private MigrateLogicalQueueBody migrateTopicLogicalQueue(int requestCode, String brokerAddr,
- LogicalQueueRouteData fromQueueRouteData, LogicalQueueRouteData toQueueRouteData,
- long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- RemotingCommand request = RemotingCommand.createRequestCommand(requestCode, null);
- MigrateLogicalQueueBody requestBody = new MigrateLogicalQueueBody();
- requestBody.setFromQueueRouteData(fromQueueRouteData);
- requestBody.setToQueueRouteData(toQueueRouteData);
- request.setBody(requestBody.encode());
- RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, timeoutMillis);
- assert response != null;
- if (response.getCode() != ResponseCode.SUCCESS) {
- throw new MQBrokerException(response.getCode(), response.getRemark());
- }
- return response.getBody() != null ? RemotingSerializable.decode(response.getBody(), MigrateLogicalQueueBody.class) : null;
- }
-
- public MigrateLogicalQueueBody migrateTopicLogicalQueuePrepare(String brokerAddr,
- LogicalQueueRouteData fromQueueRouteData, LogicalQueueRouteData toQueueRouteData,
- long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- return migrateTopicLogicalQueue(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_PREPARE, brokerAddr, fromQueueRouteData, toQueueRouteData, timeoutMillis);
- }
-
- public MigrateLogicalQueueBody migrateTopicLogicalQueueCommit(String brokerAddr,
- LogicalQueueRouteData fromQueueRouteData, LogicalQueueRouteData toQueueRouteData,
- long timeoutMillis) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
- return migrateTopicLogicalQueue(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_COMMIT, brokerAddr, fromQueueRouteData, toQueueRouteData, timeoutMillis);
- }
-
- public void migrateTopicLogicalQueueNotify(String brokerAddr,
- LogicalQueueRouteData fromQueueRouteData,
- LogicalQueueRouteData toQueueRouteData,
- long timeoutMillis) throws InterruptedException, MQBrokerException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException {
- migrateTopicLogicalQueue(RequestCode.MIGRATE_TOPIC_LOGICAL_QUEUE_NOTIFY, brokerAddr, fromQueueRouteData, toQueueRouteData, timeoutMillis);
- }
-
- public TopicConfig getTopicConfig(final String brokerAddr, String topic,
+ public TopicConfigAndQueueMapping getTopicConfig(final String brokerAddr, String topic,
long timeoutMillis) throws InterruptedException,
RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException {
GetTopicConfigRequestHeader header = new GetTopicConfigRequestHeader();
header.setTopic(topic);
+ header.setLo(true);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_TOPIC_CONFIG, header);
RemotingCommand response = this.remotingClient
.invokeSync(MixAll.brokerVIPChannel(this.clientConfig.isVipChannelEnabled(), brokerAddr), request, timeoutMillis);
assert response != null;
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
- return RemotingSerializable.decode(response.getBody(), TopicConfig.class);
+ return RemotingSerializable.decode(response.getBody(), TopicConfigAndQueueMapping.class);
+ }
+ //should check the exist
+ case ResponseCode.TOPIC_NOT_EXIST: {
+ //should return null?
+ break;
}
default:
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
}
+
+ public void createStaticTopic(final String addr, final String defaultTopic, final TopicConfig topicConfig, final TopicQueueMappingDetail topicQueueMappingDetail, boolean force,
+ final long timeoutMillis) throws RemotingException, InterruptedException, MQBrokerException {
+ CreateTopicRequestHeader requestHeader = new CreateTopicRequestHeader();
+ requestHeader.setTopic(topicConfig.getTopicName());
+ requestHeader.setDefaultTopic(defaultTopic);
+ requestHeader.setReadQueueNums(topicConfig.getReadQueueNums());
+ requestHeader.setWriteQueueNums(topicConfig.getWriteQueueNums());
+ requestHeader.setPerm(topicConfig.getPerm());
+ requestHeader.setTopicFilterType(topicConfig.getTopicFilterType().name());
+ requestHeader.setTopicSysFlag(topicConfig.getTopicSysFlag());
+ requestHeader.setOrder(topicConfig.isOrder());
+ requestHeader.setForce(force);
+ RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_AND_CREATE_STATIC_TOPIC, requestHeader);
+ request.setBody(topicQueueMappingDetail.encode());
+
+ RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(this.clientConfig.isVipChannelEnabled(), addr),
+ request, timeoutMillis);
+ assert response != null;
+ switch (response.getCode()) {
+ case ResponseCode.SUCCESS: {
+ return;
+ }
+ default:
+ break;
+ }
+
+ throw new MQBrokerException(response.getCode(), response.getRemark());
+ }
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageConcurrentlyService.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageConcurrentlyService.java
index 702aef6..ce9355f 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageConcurrentlyService.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageConcurrentlyService.java
@@ -306,7 +306,7 @@ public class ConsumeMessageConcurrentlyService implements ConsumeMessageService
// Wrap topic with namespace before sending back message.
msg.setTopic(this.defaultMQPushConsumer.withNamespace(msg.getTopic()));
try {
- this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, context.getMessageQueue().getBrokerName());
+ this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, this.defaultMQPushConsumer.queueWithNamespace(context.getMessageQueue()));
return true;
} catch (Exception e) {
log.error("sendMessageBack exception, group: " + this.consumerGroup + " msg: " + msg.toString(), e);
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultLitePullConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultLitePullConsumerImpl.java
index 4822d7d..845b158 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultLitePullConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultLitePullConsumerImpl.java
@@ -16,25 +16,6 @@
*/
package org.apache.rocketmq.client.impl.consumer;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.Executors;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledThreadPoolExecutor;
-import java.util.concurrent.ThreadFactory;
-import java.util.concurrent.TimeUnit;
import org.apache.rocketmq.client.Validators;
import org.apache.rocketmq.client.consumer.DefaultLitePullConsumer;
import org.apache.rocketmq.client.consumer.MessageQueueListener;
@@ -74,6 +55,26 @@ import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.RPCHook;
import org.apache.rocketmq.remoting.exception.RemotingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+
public class DefaultLitePullConsumerImpl implements MQConsumerInner {
private final InternalLogger log = ClientLogger.getLog();
@@ -928,9 +929,6 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
null
);
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
- if (pullResult instanceof PullResultWithLogicalQueues) {
- pullResult = ((PullResultWithLogicalQueues) pullResult).getOrigPullResultExt();
- }
if (!this.consumeMessageHookList.isEmpty()) {
ConsumeMessageContext consumeMessageContext = new ConsumeMessageContext();
consumeMessageContext.setNamespace(defaultLitePullConsumer.getNamespace());
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
index e54d9b6..6d47573 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
@@ -265,9 +265,6 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
null
);
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
- if (pullResult instanceof PullResultWithLogicalQueues) {
- pullResult = ((PullResultWithLogicalQueues) pullResult).getOrigPullResultExt();
- }
//If namespace is not null , reset Topic without namespace.
this.resetTopic(pullResult.getMsgFoundList());
if (!this.consumeMessageHookList.isEmpty()) {
@@ -574,10 +571,15 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
this.offsetStore.updateConsumeOffsetToBroker(mq, offset, isOneway);
}
+ @Deprecated
public void sendMessageBack(MessageExt msg, int delayLevel, final String brokerName, String consumerGroup)
throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
try {
- String brokerAddr = (null != brokerName) ? this.mQClientFactory.findBrokerAddressInPublish(brokerName)
+ String destBrokerName = brokerName;
+ if (destBrokerName != null && destBrokerName.startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX)) {
+ destBrokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(this.defaultMQPullConsumer.queueWithNamespace(new MessageQueue(msg.getTopic(), msg.getBrokerName(), msg.getQueueId())));
+ }
+ String brokerAddr = (null != destBrokerName) ? this.mQClientFactory.findBrokerAddressInPublish(destBrokerName)
: RemotingHelper.parseSocketAddressAddr(msg.getStoreHost());
if (UtilAll.isBlank(consumerGroup)) {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
index 92ee926..2b504cc 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
@@ -72,6 +72,7 @@ import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
+import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.body.ConsumeStatus;
import org.apache.rocketmq.common.protocol.body.ConsumerRunningInfo;
import org.apache.rocketmq.common.protocol.body.PopProcessQueueInfo;
@@ -297,6 +298,9 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
long offset = -1L;
try {
offset = this.rebalanceImpl.computePullFromWhereWithException(pullRequest.getMessageQueue());
+ if (offset < 0) {
+ throw new MQClientException(ResponseCode.SYSTEM_ERROR, "Unexpected offset " + offset);
+ }
} catch (Exception e) {
this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
log.error("Failed to compute pull offset, pullResult: {}", pullRequest, e);
@@ -704,33 +708,53 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
log.info("resume this consumer, {}", this.defaultMQPushConsumer.getConsumerGroup());
}
+ @Deprecated
public void sendMessageBack(MessageExt msg, int delayLevel, final String brokerName)
+ throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
+ sendMessageBack(msg, delayLevel, brokerName, null);
+ }
+
+ public void sendMessageBack(MessageExt msg, int delayLevel, final MessageQueue mq)
+ throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
+ sendMessageBack(msg, delayLevel, null, mq);
+ }
+
+
+ private void sendMessageBack(MessageExt msg, int delayLevel, final String brokerName, final MessageQueue mq)
throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
try {
- String brokerAddr = (null != brokerName) ? this.mQClientFactory.findBrokerAddressInPublish(brokerName)
- : RemotingHelper.parseSocketAddressAddr(msg.getStoreHost());
- this.mQClientFactory.getMQClientAPIImpl().consumerSendMessageBack(brokerAddr, msg,
- this.defaultMQPushConsumer.getConsumerGroup(), delayLevel, 5000, getMaxReconsumeTimes());
+ if ((brokerName != null && brokerName.startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX))
+ || (mq != null && mq.getBrokerName().startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX))) {
+ sendMessageBackAsNormalMessage(msg);
+ } else {
+ String brokerAddr = (null != brokerName) ? this.mQClientFactory.findBrokerAddressInPublish(brokerName)
+ : RemotingHelper.parseSocketAddressAddr(msg.getStoreHost());
+ this.mQClientFactory.getMQClientAPIImpl().consumerSendMessageBack(brokerAddr, msg,
+ this.defaultMQPushConsumer.getConsumerGroup(), delayLevel, 5000, getMaxReconsumeTimes());
+ }
} catch (Exception e) {
log.error("sendMessageBack Exception, " + this.defaultMQPushConsumer.getConsumerGroup(), e);
+ sendMessageBackAsNormalMessage(msg);
+ } finally {
+ msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQPushConsumer.getNamespace()));
+ }
+ }
- Message newMsg = new Message(MixAll.getRetryTopic(this.defaultMQPushConsumer.getConsumerGroup()), msg.getBody());
+ private void sendMessageBackAsNormalMessage(MessageExt msg) throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
+ Message newMsg = new Message(MixAll.getRetryTopic(this.defaultMQPushConsumer.getConsumerGroup()), msg.getBody());
- String originMsgId = MessageAccessor.getOriginMessageId(msg);
- MessageAccessor.setOriginMessageId(newMsg, UtilAll.isBlank(originMsgId) ? msg.getMsgId() : originMsgId);
+ String originMsgId = MessageAccessor.getOriginMessageId(msg);
+ MessageAccessor.setOriginMessageId(newMsg, UtilAll.isBlank(originMsgId) ? msg.getMsgId() : originMsgId);
- newMsg.setFlag(msg.getFlag());
- MessageAccessor.setProperties(newMsg, msg.getProperties());
- MessageAccessor.putProperty(newMsg, MessageConst.PROPERTY_RETRY_TOPIC, msg.getTopic());
- MessageAccessor.setReconsumeTime(newMsg, String.valueOf(msg.getReconsumeTimes() + 1));
- MessageAccessor.setMaxReconsumeTimes(newMsg, String.valueOf(getMaxReconsumeTimes()));
- MessageAccessor.clearProperty(newMsg, MessageConst.PROPERTY_TRANSACTION_PREPARED);
- newMsg.setDelayTimeLevel(3 + msg.getReconsumeTimes());
+ newMsg.setFlag(msg.getFlag());
+ MessageAccessor.setProperties(newMsg, msg.getProperties());
+ MessageAccessor.putProperty(newMsg, MessageConst.PROPERTY_RETRY_TOPIC, msg.getTopic());
+ MessageAccessor.setReconsumeTime(newMsg, String.valueOf(msg.getReconsumeTimes() + 1));
+ MessageAccessor.setMaxReconsumeTimes(newMsg, String.valueOf(getMaxReconsumeTimes()));
+ MessageAccessor.clearProperty(newMsg, MessageConst.PROPERTY_TRANSACTION_PREPARED);
+ newMsg.setDelayTimeLevel(3 + msg.getReconsumeTimes());
- this.mQClientFactory.getDefaultMQProducer().send(newMsg);
- } finally {
- msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQPushConsumer.getNamespace()));
- }
+ this.mQClientFactory.getDefaultMQProducer().send(newMsg);
}
void ackAsync(MessageExt message, String consumerGroup) {
@@ -743,15 +767,21 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
long queueOffset = ExtraInfoUtil.getQueueOffset(extraInfoStrs);
String topic = message.getTopic();
+ String desBrokerName = brokerName;
+ if (brokerName != null && brokerName.startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX)) {
+ desBrokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(this.defaultMQPushConsumer.queueWithNamespace(new MessageQueue(topic, brokerName, queueId)));
+ }
+
+
FindBrokerResult
- findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(desBrokerName, MixAll.MASTER_ID, true);
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
- findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(desBrokerName, MixAll.MASTER_ID, true);
}
if (findBrokerResult == null) {
- log.error("The broker[" + brokerName + "] not exist");
+ log.error("The broker[" + desBrokerName + "] not exist");
return;
}
@@ -784,11 +814,17 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
String[] extraInfoStrs = ExtraInfoUtil.split(extraInfo);
String brokerName = ExtraInfoUtil.getBrokerName(extraInfoStrs);
int queueId = ExtraInfoUtil.getQueueId(extraInfoStrs);
+
+ String desBrokerName = brokerName;
+ if (brokerName != null && brokerName.startsWith(MixAll.LOGICAL_QUEUE_MOCK_BROKER_PREFIX)) {
+ desBrokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(this.defaultMQPushConsumer.queueWithNamespace(new MessageQueue(topic, brokerName, queueId)));
+ }
+
FindBrokerResult
- findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(desBrokerName, MixAll.MASTER_ID, true);
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
- findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(desBrokerName, MixAll.MASTER_ID, true);
}
if (findBrokerResult != null) {
ChangeInvisibleTimeRequestHeader requestHeader = new ChangeInvisibleTimeRequestHeader();
@@ -798,10 +834,11 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
requestHeader.setConsumerGroup(consumerGroup);
requestHeader.setExtraInfo(extraInfo);
requestHeader.setInvisibleTime(invisibleTime);
+ //here the broker should be polished
this.mQClientFactory.getMQClientAPIImpl().changeInvisibleTimeAsync(brokerName, findBrokerResult.getBrokerAddr(), requestHeader, ASYNC_TIMEOUT, callback);
return;
}
- throw new MQClientException("The broker[" + brokerName + "] not exist", null);
+ throw new MQClientException("The broker[" + desBrokerName + "] not exist", null);
}
public int getMaxReconsumeTimes() {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
index 5868e30..30e8439 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
@@ -16,15 +16,12 @@
*/
package org.apache.rocketmq.client.impl.consumer;
-import com.alibaba.fastjson.JSON;
-import com.google.common.base.Objects;
import org.apache.rocketmq.client.consumer.PopCallback;
import org.apache.rocketmq.client.consumer.PullCallback;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
-import org.apache.rocketmq.client.exception.MQRedirectException;
import org.apache.rocketmq.client.hook.FilterMessageContext;
import org.apache.rocketmq.client.hook.FilterMessageHook;
import org.apache.rocketmq.client.impl.CommunicationMode;
@@ -40,12 +37,8 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.header.PopMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
import org.apache.rocketmq.common.sysflag.PullSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
@@ -53,16 +46,12 @@ import org.apache.rocketmq.remoting.exception.RemotingException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.List;
import java.util.Random;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
-import static com.google.common.base.Optional.fromNullable;
-
public class PullAPIWrapper {
private final InternalLogger log = ClientLogger.getLog();
private final MQClientInstance mQClientFactory;
@@ -83,36 +72,13 @@ public class PullAPIWrapper {
public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
final SubscriptionData subscriptionData) {
- final PullResultExt pullResultExt = (PullResultExt) pullResult;
-
- LogicalQueueRouteData queueRouteData = null;
- PullResultWithLogicalQueues pullResultWithLogicalQueues = null;
- if (pullResultExt instanceof PullResultWithLogicalQueues) {
- pullResultWithLogicalQueues = (PullResultWithLogicalQueues) pullResultExt;
- queueRouteData = pullResultWithLogicalQueues.getQueueRouteData();
- }
-
- if (queueRouteData != null) {
- pullResultWithLogicalQueues.setOrigPullResultExt(new PullResultExt(pullResultExt.getPullStatus(),
- queueRouteData.toLogicalQueueOffset(pullResultExt.getNextBeginOffset()),
- queueRouteData.toLogicalQueueOffset(pullResultExt.getMinOffset()),
- // although this maxOffset may not belong to this queue route, but the actual value must be a larger one, and since maxOffset here is not an accurate value, we just do it to make things simple.
- queueRouteData.toLogicalQueueOffset(pullResultExt.getMaxOffset()),
- pullResultExt.getMsgFoundList(),
- pullResultExt.getSuggestWhichBrokerId(),
- pullResultExt.getMessageBinary()));
- }
+ PullResultExt pullResultExt = (PullResultExt) pullResult;
this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
if (PullStatus.FOUND == pullResult.getPullStatus()) {
ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);
- if (queueRouteData != null) {
- // prevent pulled data is out of current queue route, this happens when some commit log data is cleaned in the broker but still pull from it.
- msgList = queueRouteData.filterMessages(msgList);
- }
-
List<MessageExt> msgListFilterAgain = msgList;
if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
@@ -143,8 +109,8 @@ public class PullAPIWrapper {
Long.toString(pullResult.getMaxOffset()));
msg.setBrokerName(mq.getBrokerName());
msg.setQueueId(mq.getQueueId());
- if (queueRouteData != null) {
- msg.setQueueOffset(queueRouteData.toLogicalQueueOffset(msg.getQueueOffset()));
+ if (pullResultExt.getOffsetDelta() != null) {
+ msg.setQueueOffset(pullResultExt.getOffsetDelta() + msg.getQueueOffset());
}
}
@@ -153,7 +119,7 @@ public class PullAPIWrapper {
pullResultExt.setMessageBinary(null);
- return pullResultExt;
+ return pullResult;
}
public void updatePullFromWhichNode(final MessageQueue mq, final long brokerId) {
@@ -182,72 +148,30 @@ public class PullAPIWrapper {
}
public PullResult pullKernelImpl(
- MessageQueue mq,
+ final MessageQueue mq,
final String subExpression,
final String expressionType,
final long subVersion,
- long offset,
+ final long offset,
final int maxNums,
final int sysFlag,
- long commitOffset,
+ final long commitOffset,
final long brokerSuspendMaxTimeMillis,
final long timeoutMillis,
final CommunicationMode communicationMode,
- PullCallback pullCallback
+ final PullCallback pullCallback
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
- if (MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME.equals(mq.getBrokerName())) {
- LogicalQueueContext logicalQueueContext = new LogicalQueueContext(mq, subExpression, expressionType, subVersion, offset, maxNums, sysFlag, commitOffset, brokerSuspendMaxTimeMillis, timeoutMillis, communicationMode, pullCallback);
- while (true) {
- try {
- MessageQueue messageQueue = logicalQueueContext.getModifiedMessageQueue();
- if (messageQueue == null) {
- if (pullCallback != null) {
- pullCallback.onSuccess(logicalQueueContext.getPullResult());
- return null;
- } else {
- return logicalQueueContext.getPullResult();
- }
- }
- PullResult pullResult = this.pullKernelImplWithoutRetry(messageQueue, subExpression, expressionType, subVersion, logicalQueueContext.getModifiedOffset(), maxNums, sysFlag, logicalQueueContext.getModifiedCommitOffset(), brokerSuspendMaxTimeMillis, timeoutMillis, communicationMode, logicalQueueContext.wrapPullCallback());
- return logicalQueueContext.wrapPullResult(pullResult);
- } catch (MQRedirectException e) {
- if (!logicalQueueContext.shouldRetry(e)) {
- throw new MQBrokerException(ResponseCode.SYSTEM_ERROR, "redirect");
- }
- }
- }
- } else {
- return this.pullKernelImplWithoutRetry(mq, subExpression, expressionType, subVersion, offset, maxNums, sysFlag, commitOffset, brokerSuspendMaxTimeMillis, timeoutMillis, communicationMode, pullCallback);
- }
- }
-
- public PullResult pullKernelImplWithoutRetry(
- MessageQueue mq,
- final String subExpression,
- final String expressionType,
- final long subVersion,
- long offset,
- final int maxNums,
- final int sysFlag,
- long commitOffset,
- final long brokerSuspendMaxTimeMillis,
- final long timeoutMillis,
- final CommunicationMode communicationMode,
- PullCallback pullCallback
- ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
- String topic = mq.getTopic();
- int queueId = mq.getQueueId();
-
FindBrokerResult findBrokerResult =
- this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
+ this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq),
this.recalculatePullFromWhichNode(mq), false);
if (null == findBrokerResult) {
- this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
+ this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
findBrokerResult =
- this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
+ this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq),
this.recalculatePullFromWhichNode(mq), false);
}
+
if (findBrokerResult != null) {
{
// check version
@@ -265,8 +189,8 @@ public class PullAPIWrapper {
PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
requestHeader.setConsumerGroup(this.consumerGroup);
- requestHeader.setTopic(topic);
- requestHeader.setQueueId(queueId);
+ requestHeader.setTopic(mq.getTopic());
+ requestHeader.setQueueId(mq.getQueueId());
requestHeader.setQueueOffset(offset);
requestHeader.setMaxMsgNums(maxNums);
requestHeader.setSysFlag(sysFlagInner);
@@ -278,15 +202,18 @@ public class PullAPIWrapper {
String brokerAddr = findBrokerResult.getBrokerAddr();
if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
- brokerAddr = computePullFromWhichFilterServer(topic, brokerAddr);
+ brokerAddr = computePullFromWhichFilterServer(mq.getTopic(), brokerAddr);
}
- return this.mQClientFactory.getMQClientAPIImpl().pullMessage(
+
+ PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
brokerAddr,
requestHeader,
timeoutMillis,
communicationMode,
pullCallback);
+
+ return pullResult;
}
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
@@ -373,10 +300,10 @@ public class PullAPIWrapper {
public void popAsync(MessageQueue mq, long invisibleTime, int maxNums, String consumerGroup,
long timeout, PopCallback popCallback, boolean poll, int initMode, boolean order, String expressionType, String expression)
throws MQClientException, RemotingException, InterruptedException {
- FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(), MixAll.MASTER_ID, true);
+ FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq), MixAll.MASTER_ID, true);
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
- findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(), MixAll.MASTER_ID, true);
+ findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq), MixAll.MASTER_ID, true);
}
if (findBrokerResult != null) {
PopMessageRequestHeader requestHeader = new PopMessageRequestHeader();
@@ -403,228 +330,4 @@ public class PullAPIWrapper {
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
- private class LogicalQueueContext implements PullCallback {
- private final MessageQueue mq;
- private final String subExpression;
- private final String expressionType;
- private final long subVersion;
- private final long offset;
- private final int maxNums;
- private final int sysFlag;
- private final long commitOffset;
- private final long brokerSuspendMaxTimeMillis;
- private final long timeoutMillis;
- private final CommunicationMode communicationMode;
- private final PullCallback pullCallback;
-
- private volatile LogicalQueuesInfo logicalQueuesInfo;
- private volatile LogicalQueueRouteData logicalQueueRouteData;
- private volatile boolean isMaxReadableQueueRoute;
-
- private volatile PullResultExt pullResult = null;
-
- private final AtomicInteger retry = new AtomicInteger();
-
- public LogicalQueueContext(MessageQueue mq, String subExpression, String expressionType, long subVersion,
- long offset, int maxNums, int sysFlag, long commitOffset, long brokerSuspendMaxTimeMillis,
- long timeoutMillis, CommunicationMode communicationMode,
- PullCallback pullCallback) {
- this.mq = mq;
- this.subExpression = subExpression;
- this.expressionType = expressionType;
- this.subVersion = subVersion;
- this.offset = offset;
- this.maxNums = maxNums;
- this.sysFlag = sysFlag;
- this.commitOffset = commitOffset;
- this.brokerSuspendMaxTimeMillis = brokerSuspendMaxTimeMillis;
- this.timeoutMillis = timeoutMillis;
- this.communicationMode = communicationMode;
- this.pullCallback = pullCallback;
-
- this.buildLogicalQueuesInfo();
- }
-
- private boolean notUsingLogicalQueue() {
- return !Objects.equal(mq.getBrokerName(), MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME) || this.logicalQueuesInfo == null;
- }
-
- private void buildLogicalQueuesInfo() {
- TopicRouteData topicRouteData = PullAPIWrapper.this.mQClientFactory.queryTopicRouteData(mq.getTopic());
- if (topicRouteData != null) {
- this.logicalQueuesInfo = topicRouteData.getLogicalQueuesInfo();
- }
- }
-
- @Override public void onSuccess(PullResult pullResult) {
- this.pullCallback.onSuccess(this.wrapPullResult(pullResult));
- }
-
- @Override public void onException(Throwable t) {
- if (!this.shouldRetry(t)) {
- this.pullCallback.onException(t);
- return;
- }
- MessageQueue messageQueue = this.getModifiedMessageQueue();
- if (messageQueue == null) {
- this.pullCallback.onSuccess(this.getPullResult());
- return;
- }
- try {
- PullAPIWrapper.this.pullKernelImplWithoutRetry(messageQueue, subExpression, expressionType, subVersion, this.getModifiedOffset(), maxNums, sysFlag, this.getModifiedCommitOffset(), brokerSuspendMaxTimeMillis, timeoutMillis, communicationMode, this);
- } catch (Exception e) {
- this.pullCallback.onException(e);
- }
- }
-
- public MessageQueue getModifiedMessageQueue() {
- if (this.notUsingLogicalQueue()) {
- return this.mq;
- }
- this.logicalQueuesInfo.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = fromNullable(this.logicalQueuesInfo.get(this.mq.getQueueId())).or(Collections.<LogicalQueueRouteData>emptyList());
- LogicalQueueRouteData searchKey = new LogicalQueueRouteData();
- searchKey.setState(MessageQueueRouteState.Normal);
- searchKey.setLogicalQueueDelta(offset);
- // it's sorted after getTopicRouteInfoFromNameServer
- int startIdx = Collections.binarySearch(queueRouteDataList, searchKey);
- if (startIdx < 0) {
- startIdx = -startIdx - 1;
- // lower entry
- startIdx -= 1;
- }
- this.logicalQueueRouteData = null;
- this.pullResult = null;
- LogicalQueueRouteData lastReadableLogicalQueueRouteData = null; // first item which delta > offset
- LogicalQueueRouteData minReadableLogicalQueueRouteData = null;
- LogicalQueueRouteData maxReadableLogicalQueueRouteData = null;
- for (int i = 0, size = queueRouteDataList.size(); i < size; i++) {
- LogicalQueueRouteData queueRouteData = queueRouteDataList.get(i);
- if (!queueRouteData.isReadable()) {
- continue;
- }
- maxReadableLogicalQueueRouteData = queueRouteData;
- if (minReadableLogicalQueueRouteData == null) {
- minReadableLogicalQueueRouteData = queueRouteData;
- if (i < startIdx) {
- // must consider following `i++` operation when invoke `continue`, so decrease first
- i = startIdx - 1;
- continue;
- }
- }
- if (queueRouteData.getLogicalQueueDelta() > offset) {
- if (this.logicalQueueRouteData != null) {
- if (this.logicalQueueRouteData.toLogicalQueueOffset(this.logicalQueueRouteData.getOffsetMax()) <= offset) {
- this.logicalQueueRouteData = queueRouteData;
- }
- break;
- } else {
- if (lastReadableLogicalQueueRouteData == null) {
- lastReadableLogicalQueueRouteData = queueRouteData;
- }
- }
- } else {
- this.logicalQueueRouteData = queueRouteData;
- }
- }
- if (this.logicalQueueRouteData == null) {
- if (lastReadableLogicalQueueRouteData != null) {
- this.pullResult = new PullResultExt(PullStatus.OFFSET_ILLEGAL, lastReadableLogicalQueueRouteData.getLogicalQueueDelta(), minReadableLogicalQueueRouteData.getLogicalQueueDelta(), maxReadableLogicalQueueRouteData.getLogicalQueueDelta(), null, 0, null);
- return null;
- } else {
- if (maxReadableLogicalQueueRouteData != null) {
- this.logicalQueueRouteData = maxReadableLogicalQueueRouteData;
- } else {
- if (!queueRouteDataList.isEmpty()) {
- this.logicalQueueRouteData = queueRouteDataList.get(queueRouteDataList.size() - 1);
- } else {
- pullResult = new PullResultExt(PullStatus.NO_NEW_MSG, 0, 0, 0, null, 0, null);
- return null;
- }
- }
- }
- }
- this.isMaxReadableQueueRoute = this.logicalQueueRouteData.isSameTo(maxReadableLogicalQueueRouteData);
- return this.logicalQueueRouteData.getMessageQueue();
- } finally {
- this.logicalQueuesInfo.readLock().unlock();
- }
- }
-
- public PullResultExt getPullResult() {
- return pullResult;
- }
-
- public PullCallback wrapPullCallback() {
- if (this.notUsingLogicalQueue()) {
- return this.pullCallback;
- }
- if (!CommunicationMode.ASYNC.equals(this.communicationMode)) {
- return this.pullCallback;
- }
- return this;
- }
-
- public long getModifiedOffset() {
- return this.logicalQueueRouteData.toMessageQueueOffset(this.offset);
- }
-
- public long getModifiedCommitOffset() {
- // TODO should this be modified too? If offset is not in current broker's range, how do we handle it?
- return this.commitOffset;
- }
-
- public void incrRetry() {
- this.retry.incrementAndGet();
- }
-
- public boolean shouldRetry(Throwable t) {
- this.incrRetry();
- if (this.retry.get() >= 3) {
- return false;
- }
- if (t instanceof MQRedirectException) {
- MQRedirectException e = (MQRedirectException) t;
- this.processResponseBody(e.getBody());
- return true;
- }
- return false;
- }
-
- public PullResult wrapPullResult(PullResult pullResult) {
- if (pullResult == null) {
- return null;
- }
- if (this.logicalQueueRouteData == null) {
- return pullResult;
- }
- if (!this.isMaxReadableQueueRoute && PullStatus.NO_MATCHED_MSG.equals(pullResult.getPullStatus())) {
- PullStatus status = PullStatus.OFFSET_ILLEGAL;
- if (pullResult instanceof PullResultExt) {
- PullResultExt pullResultExt = (PullResultExt) pullResult;
- pullResult = new PullResultExt(status, pullResultExt.getNextBeginOffset(), pullResultExt.getMinOffset(), pullResultExt.getMaxOffset(), pullResultExt.getMsgFoundList(), pullResultExt.getSuggestWhichBrokerId(), pullResultExt.getMessageBinary());
- } else {
- pullResult = new PullResult(status, pullResult.getNextBeginOffset(), pullResult.getMinOffset(), pullResult.getMaxOffset(), pullResult.getMsgFoundList());
- }
- }
- // method PullAPIWrapper#processPullResult will modify queueOffset/nextBeginOffset/minOffset/maxOffset
- return new PullResultWithLogicalQueues(pullResult, this.logicalQueueRouteData);
- }
-
- public void processResponseBody(byte[] responseBody) {
- log.info("LogicalQueueContext.processResponseBody got redirect {}: {}", this.logicalQueueRouteData, responseBody != null ? new String(responseBody, MessageDecoder.CHARSET_UTF8) : null);
- if (responseBody != null) {
- try {
- List<LogicalQueueRouteData> queueRouteDataList = JSON.parseObject(responseBody, MixAll.TYPE_LIST_LOGICAL_QUEUE_ROUTE_DATA);
- this.logicalQueuesInfo.updateLogicalQueueRouteDataList(this.mq.getQueueId(), queueRouteDataList);
- return;
- } catch (Exception e) {
- log.warn("LogicalQueueContext.processResponseBody {} update exception, fallback to updateTopicRouteInfoFromNameServer", this.logicalQueueRouteData, e);
- }
- }
- PullAPIWrapper.this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic(), false, null, Collections.singleton(this.mq.getQueueId()));
- this.buildLogicalQueuesInfo();
- }
- }
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultExt.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultExt.java
index c34a68f..45538ed 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultExt.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultExt.java
@@ -25,11 +25,23 @@ public class PullResultExt extends PullResult {
private final long suggestWhichBrokerId;
private byte[] messageBinary;
+ private final Long offsetDelta;
+
public PullResultExt(PullStatus pullStatus, long nextBeginOffset, long minOffset, long maxOffset,
List<MessageExt> msgFoundList, final long suggestWhichBrokerId, final byte[] messageBinary) {
+ this(pullStatus, nextBeginOffset, minOffset, maxOffset, msgFoundList, suggestWhichBrokerId, messageBinary, 0L);
+ }
+
+ public PullResultExt(PullStatus pullStatus, long nextBeginOffset, long minOffset, long maxOffset,
+ List<MessageExt> msgFoundList, final long suggestWhichBrokerId, final byte[] messageBinary, final Long offsetDelta) {
super(pullStatus, nextBeginOffset, minOffset, maxOffset, msgFoundList);
this.suggestWhichBrokerId = suggestWhichBrokerId;
this.messageBinary = messageBinary;
+ this.offsetDelta = offsetDelta;
+ }
+
+ public Long getOffsetDelta() {
+ return offsetDelta;
}
public byte[] getMessageBinary() {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultWithLogicalQueues.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultWithLogicalQueues.java
deleted file mode 100644
index 8b85e54..0000000
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullResultWithLogicalQueues.java
+++ /dev/null
@@ -1,96 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.client.impl.consumer;
-
-import java.util.List;
-import org.apache.rocketmq.client.consumer.PullResult;
-import org.apache.rocketmq.client.consumer.PullStatus;
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-
-public class PullResultWithLogicalQueues extends PullResultExt {
- private PullResultExt origPullResultExt;
- private final LogicalQueueRouteData queueRouteData;
-
- public PullResultWithLogicalQueues(PullResult pullResult, LogicalQueueRouteData floorQueueRouteData) {
- super(pullResult.getPullStatus(), pullResult.getNextBeginOffset(), pullResult.getMinOffset(), pullResult.getMaxOffset(), pullResult.getMsgFoundList(),
- pullResult instanceof PullResultExt ? ((PullResultExt) pullResult).getSuggestWhichBrokerId() : MixAll.MASTER_ID,
- pullResult instanceof PullResultExt ? ((PullResultExt) pullResult).getMessageBinary() : null);
- if (pullResult instanceof PullResultExt) {
- this.origPullResultExt = (PullResultExt) pullResult;
- } else {
- this.origPullResultExt = new PullResultExt(pullResult.getPullStatus(), pullResult.getNextBeginOffset(), pullResult.getMinOffset(), pullResult.getMaxOffset(), pullResult.getMsgFoundList(), MixAll.MASTER_ID, null);
- }
- this.queueRouteData = floorQueueRouteData;
- }
-
- public PullResult getOrigPullResultExt() {
- return origPullResultExt;
- }
-
- public LogicalQueueRouteData getQueueRouteData() {
- return queueRouteData;
- }
-
- public void setOrigPullResultExt(PullResultExt pullResultExt) {
- this.origPullResultExt = pullResultExt;
- }
-
- @Override public PullStatus getPullStatus() {
- return origPullResultExt.getPullStatus();
- }
-
- @Override public long getNextBeginOffset() {
- return origPullResultExt.getNextBeginOffset();
- }
-
- @Override public long getMinOffset() {
- return origPullResultExt.getMinOffset();
- }
-
- @Override public long getMaxOffset() {
- return origPullResultExt.getMaxOffset();
- }
-
- @Override public List<MessageExt> getMsgFoundList() {
- return origPullResultExt.getMsgFoundList();
- }
-
- @Override public void setMsgFoundList(List<MessageExt> msgFoundList) {
- origPullResultExt.setMsgFoundList(msgFoundList);
- }
-
- @Override public byte[] getMessageBinary() {
- return origPullResultExt.getMessageBinary();
- }
-
- @Override public void setMessageBinary(byte[] messageBinary) {
- origPullResultExt.setMessageBinary(messageBinary);
- }
-
- @Override public long getSuggestWhichBrokerId() {
- return origPullResultExt.getSuggestWhichBrokerId();
- }
-
- @Override public String toString() {
- return "PullResultWithLogicalQueues{" +
- "origPullResultExt=" + origPullResultExt +
- ", queueRouteData=" + queueRouteData +
- '}';
- }
-}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java
index ab0d885..5788d1c 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java
@@ -76,7 +76,7 @@ public abstract class RebalanceImpl {
}
public void unlock(final MessageQueue mq, final boolean oneway) {
- FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(), MixAll.MASTER_ID, true);
+ FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq), MixAll.MASTER_ID, true);
if (findBrokerResult != null) {
UnlockBatchRequestBody requestBody = new UnlockBatchRequestBody();
requestBody.setConsumerGroup(this.consumerGroup);
@@ -141,7 +141,8 @@ public abstract class RebalanceImpl {
continue;
}
- Set<MessageQueue> mqs = result.get(mq.getBrokerName());
+ String destBrokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(mq);
+ Set<MessageQueue> mqs = result.get(destBrokerName);
if (null == mqs) {
mqs = new HashSet<MessageQueue>();
result.put(mq.getBrokerName(), mqs);
@@ -154,7 +155,7 @@ public abstract class RebalanceImpl {
}
public boolean lock(final MessageQueue mq) {
- FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(), MixAll.MASTER_ID, true);
+ FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(this.mQClientFactory.getBrokerNameFromMessageQueue(mq), MixAll.MASTER_ID, true);
if (findBrokerResult != null) {
LockBatchRequestBody requestBody = new LockBatchRequestBody();
requestBody.setConsumerGroup(this.consumerGroup);
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java
index 09d1521..76803dd 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java
@@ -29,6 +29,7 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.ConsumeInitMode;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.heartbeat.ConsumeType;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
@@ -188,7 +189,8 @@ public class RebalancePushImpl extends RebalanceImpl {
}
}
} else {
- result = -1;
+ throw new MQClientException(ResponseCode.QUERY_NOT_FOUND, "Failed to query consume offset from " +
+ "offset store");
}
break;
}
@@ -197,9 +199,11 @@ public class RebalancePushImpl extends RebalanceImpl {
if (lastOffset >= 0) {
result = lastOffset;
} else if (-1 == lastOffset) {
+ //the offset will be fixed by the OFFSET_ILLEGAL process
result = 0L;
} else {
- result = -1;
+ throw new MQClientException(ResponseCode.QUERY_NOT_FOUND, "Failed to query offset from offset " +
+ "store");
}
break;
}
@@ -226,7 +230,8 @@ public class RebalancePushImpl extends RebalanceImpl {
}
}
} else {
- result = -1;
+ throw new MQClientException(ResponseCode.QUERY_NOT_FOUND, "Failed to query offset from offset " +
+ "store");
}
break;
}
@@ -235,6 +240,10 @@ public class RebalancePushImpl extends RebalanceImpl {
break;
}
+ if (result < 0) {
+ throw new MQClientException(ResponseCode.SYSTEM_ERROR, "Found unexpected result " + result);
+ }
+
return result;
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/factory/MQClientInstance.java b/client/src/main/java/org/apache/rocketmq/client/impl/factory/MQClientInstance.java
index 874b1f8..0181951 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/factory/MQClientInstance.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/factory/MQClientInstance.java
@@ -16,26 +16,6 @@
*/
package org.apache.rocketmq.client.impl.factory;
-import java.io.UnsupportedEncodingException;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Random;
-import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ThreadFactory;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.locks.Lock;
-import java.util.concurrent.locks.ReentrantLock;
import org.apache.commons.lang3.StringUtils;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.admin.MQAdminExtInner;
@@ -77,8 +57,6 @@ import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.common.protocol.heartbeat.ProducerData;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
import org.apache.rocketmq.common.protocol.route.BrokerData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
import org.apache.rocketmq.common.protocol.route.QueueData;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
import org.apache.rocketmq.logging.InternalLogger;
@@ -88,10 +66,32 @@ import org.apache.rocketmq.remoting.exception.RemotingException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+import java.io.UnsupportedEncodingException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import static org.apache.rocketmq.common.rpc.ClientMetadata.topicRouteData2EndpointsForStaticTopic;
public class MQClientInstance {
private final static long LOCK_TIMEOUT_MILLIS = 3000;
- private final InternalLogger log = ClientLogger.getLog();
+ private final static InternalLogger log = ClientLogger.getLog();
private final ClientConfig clientConfig;
private final int instanceIndex;
private final String clientId;
@@ -103,6 +103,7 @@ public class MQClientInstance {
private final MQClientAPIImpl mQClientAPIImpl;
private final MQAdminImpl mQAdminImpl;
private final ConcurrentMap<String/* Topic */, TopicRouteData> topicRouteTable = new ConcurrentHashMap<String, TopicRouteData>();
+ private final ConcurrentMap<String/* Topic */, ConcurrentMap<MessageQueue, String/*brokerName*/>> topicEndPointsTable = new ConcurrentHashMap<>();
private final Lock lockNamesrv = new ReentrantLock();
private final Lock lockHeartbeat = new ReentrantLock();
private final ConcurrentMap<String/* Broker Name */, HashMap<Long/* brokerId */, String/* address */>> brokerAddrTable =
@@ -162,8 +163,12 @@ public class MQClientInstance {
MQVersion.getVersionDesc(MQVersion.CURRENT_VERSION), RemotingCommand.getSerializeTypeConfigInThisServer());
}
+
+
+
public static TopicPublishInfo topicRouteData2TopicPublishInfo(final String topic, final TopicRouteData route) {
TopicPublishInfo info = new TopicPublishInfo();
+ // TO DO should check the usage of raw route, it is better to remove such field
info.setTopicRouteData(route);
if (route.getOrderTopicConf() != null && route.getOrderTopicConf().length() > 0) {
String[] brokers = route.getOrderTopicConf().split(";");
@@ -177,28 +182,13 @@ public class MQClientInstance {
}
info.setOrderTopic(true);
- } else if (route.getOrderTopicConf() == null && route.getLogicalQueuesInfo() != null) {
+ } else if (route.getOrderTopicConf() == null
+ && route.getTopicQueueMappingByBroker() != null
+ && !route.getTopicQueueMappingByBroker().isEmpty()) {
info.setOrderTopic(false);
- List<MessageQueue> messageQueueList = info.getMessageQueueList();
- LogicalQueuesInfo logicalQueueInfo = route.getLogicalQueuesInfo();
- for (Map.Entry<Integer, List<LogicalQueueRouteData>> entry : logicalQueueInfo.entrySet()) {
- boolean someWritable = false;
- for (LogicalQueueRouteData logicalQueueRouteData : entry.getValue()) {
- if (logicalQueueRouteData.isWritable()) {
- someWritable = true;
- break;
- }
- }
- if (!someWritable) {
- continue;
- }
- MessageQueue mq = new MessageQueue();
- mq.setQueueId(entry.getKey());
- mq.setBrokerName(MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME);
- mq.setTopic(topic);
- messageQueueList.add(mq);
- }
- Collections.sort(messageQueueList, new Comparator<MessageQueue>() {
+ ConcurrentMap<MessageQueue, String> mqEndPoints = topicRouteData2EndpointsForStaticTopic(topic, route);
+ info.getMessageQueueList().addAll(mqEndPoints.keySet());
+ Collections.sort(info.getMessageQueueList(), new Comparator<MessageQueue>() {
@Override public int compare(MessageQueue o1, MessageQueue o2) {
return MixAll.compareInteger(o1.getQueueId(), o2.getQueueId());
}
@@ -239,26 +229,10 @@ public class MQClientInstance {
public static Set<MessageQueue> topicRouteData2TopicSubscribeInfo(final String topic, final TopicRouteData route) {
Set<MessageQueue> mqList = new HashSet<MessageQueue>();
- if (route.getLogicalQueuesInfo() != null) {
- LogicalQueuesInfo logicalQueueInfo = route.getLogicalQueuesInfo();
- for (Map.Entry<Integer, List<LogicalQueueRouteData>> entry : logicalQueueInfo.entrySet()) {
- boolean someReadable = false;
- for (LogicalQueueRouteData logicalQueueRouteData : entry.getValue()) {
- if (logicalQueueRouteData.isReadable()) {
- someReadable = true;
- break;
- }
- }
- if (!someReadable) {
- continue;
- }
- MessageQueue mq = new MessageQueue();
- mq.setQueueId(entry.getKey());
- mq.setBrokerName(MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME);
- mq.setTopic(topic);
- mqList.add(mq);
- }
- return mqList;
+ if (route.getTopicQueueMappingByBroker() != null
+ && !route.getTopicQueueMappingByBroker().isEmpty()) {
+ ConcurrentMap<MessageQueue, String> mqEndPoints = topicRouteData2EndpointsForStaticTopic(topic, route);
+ return mqEndPoints.keySet();
}
List<QueueData> qds = route.getQueueDatas();
for (QueueData qd : qds) {
@@ -656,11 +630,6 @@ public class MQClientInstance {
public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
DefaultMQProducer defaultMQProducer) {
- return this.updateTopicRouteInfoFromNameServer(topic, isDefault, defaultMQProducer, null);
- }
-
- public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
- DefaultMQProducer defaultMQProducer, Set<Integer> logicalQueueIdsFilter) {
try {
if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
@@ -676,7 +645,7 @@ public class MQClientInstance {
}
}
} else {
- topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, clientConfig.getMqClientApiTimeout(), true, logicalQueueIdsFilter);
+ topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, clientConfig.getMqClientApiTimeout());
}
if (topicRouteData != null) {
TopicRouteData old = this.topicRouteTable.get(topic);
@@ -688,31 +657,19 @@ public class MQClientInstance {
}
if (changed) {
- TopicRouteData cloneTopicRouteData = new TopicRouteData(topicRouteData);
- if (logicalQueueIdsFilter != null && cloneTopicRouteData.getLogicalQueuesInfo() != null) {
- TopicRouteData curTopicRouteData = this.topicRouteTable.get(topic);
- if (curTopicRouteData != null) {
- LogicalQueuesInfo curLogicalQueuesInfo = curTopicRouteData.getLogicalQueuesInfo();
- if (curLogicalQueuesInfo != null) {
- LogicalQueuesInfo cloneLogicalQueuesInfo = cloneTopicRouteData.getLogicalQueuesInfo();
- curLogicalQueuesInfo.readLock().lock();
- try {
- for (Entry<Integer, List<LogicalQueueRouteData>> entry : curLogicalQueuesInfo.entrySet()) {
- if (!cloneLogicalQueuesInfo.containsKey(entry.getKey())) {
- cloneLogicalQueuesInfo.put(entry.getKey(), entry.getValue());
- }
- }
- } finally {
- curLogicalQueuesInfo.readLock().unlock();
- }
- }
- }
- }
for (BrokerData bd : topicRouteData.getBrokerDatas()) {
this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
}
+ // Update endpoint map
+ {
+ ConcurrentMap<MessageQueue, String> mqEndPoints = topicRouteData2EndpointsForStaticTopic(topic, topicRouteData);
+ if (mqEndPoints != null && !mqEndPoints.isEmpty()) {
+ topicEndPointsTable.put(topic, mqEndPoints);
+ }
+ }
+
// Update Pub info
{
TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
@@ -739,6 +696,7 @@ public class MQClientInstance {
}
}
}
+ TopicRouteData cloneTopicRouteData = new TopicRouteData(topicRouteData);
log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
this.topicRouteTable.put(topic, cloneTopicRouteData);
return true;
@@ -865,13 +823,6 @@ public class MQClientInstance {
private boolean topicRouteDataIsChange(TopicRouteData olddata, TopicRouteData nowdata) {
if (olddata == null || nowdata == null)
return true;
- LogicalQueuesInfo oldLogicalQueuesInfo = olddata.getLogicalQueuesInfo();
- LogicalQueuesInfo newLogicalQueuesInfo = nowdata.getLogicalQueuesInfo();
- if (oldLogicalQueuesInfo != null && newLogicalQueuesInfo != null) {
- return oldLogicalQueuesInfo.keySet().equals(newLogicalQueuesInfo.keySet());
- } else if (oldLogicalQueuesInfo != null || newLogicalQueuesInfo != null) {
- return true;
- }
TopicRouteData old = new TopicRouteData(olddata);
TopicRouteData now = new TopicRouteData(nowdata);
Collections.sort(old.getQueueDatas());
@@ -1067,7 +1018,21 @@ public class MQClientInstance {
return this.consumerTable.get(group);
}
+
+ public String getBrokerNameFromMessageQueue(final MessageQueue mq) {
+ if (topicEndPointsTable != null
+ && topicEndPointsTable.get(mq.getTopic()) != null
+ && !topicEndPointsTable.get(mq.getTopic()).isEmpty()) {
+ return topicEndPointsTable.get(mq.getTopic()).get(mq);
+ }
+ return mq.getBrokerName();
+ }
+
+
public FindBrokerResult findBrokerAddressInAdmin(final String brokerName) {
+ if (brokerName == null) {
+ return null;
+ }
String brokerAddr = null;
boolean slave = false;
boolean found = false;
@@ -1097,7 +1062,11 @@ public class MQClientInstance {
return null;
}
+
public String findBrokerAddressInPublish(final String brokerName) {
+ if (brokerName == null) {
+ return null;
+ }
HashMap<Long/* brokerId */, String/* address */> map = this.brokerAddrTable.get(brokerName);
if (map != null && !map.isEmpty()) {
return map.get(MixAll.MASTER_ID);
@@ -1106,11 +1075,15 @@ public class MQClientInstance {
return null;
}
+
public FindBrokerResult findBrokerAddressInSubscribe(
final String brokerName,
final long brokerId,
final boolean onlyThisBroker
) {
+ if (brokerName == null) {
+ return null;
+ }
String brokerAddr = null;
boolean slave = false;
boolean found = false;
@@ -1141,7 +1114,7 @@ public class MQClientInstance {
return null;
}
- public int findBrokerVersion(String brokerName, String brokerAddr) {
+ private int findBrokerVersion(String brokerName, String brokerAddr) {
if (this.brokerVersionTable.containsKey(brokerName)) {
if (this.brokerVersionTable.get(brokerName).containsKey(brokerAddr)) {
return this.brokerVersionTable.get(brokerName).get(brokerAddr);
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
index 00df89c..bf2ca28 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
@@ -16,16 +16,12 @@
*/
package org.apache.rocketmq.client.impl.producer;
-import com.alibaba.fastjson.JSON;
-import com.google.common.base.Objects;
import java.io.IOException;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collections;
import java.util.HashSet;
import java.util.List;
-import java.util.Locale;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
@@ -45,7 +41,6 @@ import org.apache.rocketmq.client.Validators;
import org.apache.rocketmq.client.common.ClientErrorCode;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
-import org.apache.rocketmq.client.exception.MQRedirectException;
import org.apache.rocketmq.client.exception.RequestTimeoutException;
import org.apache.rocketmq.client.hook.CheckForbiddenContext;
import org.apache.rocketmq.client.hook.CheckForbiddenHook;
@@ -67,7 +62,6 @@ import org.apache.rocketmq.client.producer.RequestFutureTable;
import org.apache.rocketmq.client.producer.RequestResponseFuture;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
-import org.apache.rocketmq.client.producer.SendResultForLogicalQueue;
import org.apache.rocketmq.client.producer.SendStatus;
import org.apache.rocketmq.client.producer.TransactionCheckListener;
import org.apache.rocketmq.client.producer.TransactionListener;
@@ -88,13 +82,9 @@ import org.apache.rocketmq.common.message.MessageId;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.message.MessageType;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
-import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.CheckTransactionStateRequestHeader;
import org.apache.rocketmq.common.protocol.header.EndTransactionRequestHeader;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.common.protocol.route.TopicRouteData;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.utils.CorrelationIdUtil;
import org.apache.rocketmq.logging.InternalLogger;
@@ -730,43 +720,13 @@ public class DefaultMQProducerImpl implements MQProducerInner {
final SendCallback sendCallback,
final TopicPublishInfo topicPublishInfo,
final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
- if (MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME.equals(mq.getBrokerName())) {
- LogicalQueueSendContext logicalQueueContext = new LogicalQueueSendContext(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout);
- while (true) {
- try {
- SendResult sendResult = this.sendKernelImplWithoutRetry(msg,
- logicalQueueContext.getModifiedMessageQueue(),
- communicationMode,
- logicalQueueContext.wrapSendCallback(),
- topicPublishInfo,
- timeout);
- return logicalQueueContext.wrapSendResult(sendResult);
- } catch (MQRedirectException e) {
- if (!logicalQueueContext.shouldRetry(e)) {
- throw new MQBrokerException(ResponseCode.SYSTEM_ERROR, "redirect");
- }
- } catch (RemotingException e) {
- if (!logicalQueueContext.shouldRetry(e)) {
- throw e;
- }
- }
- }
- } else {
- return sendKernelImplWithoutRetry(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout);
- }
- }
-
- private SendResult sendKernelImplWithoutRetry(final Message msg,
- final MessageQueue mq,
- final CommunicationMode communicationMode,
- SendCallback sendCallback,
- final TopicPublishInfo topicPublishInfo,
- final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
long beginStartTime = System.currentTimeMillis();
- String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ String brokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(mq);
+ String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(brokerName);
if (null == brokerAddr) {
tryToFindTopicPublishInfo(mq.getTopic());
- brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
+ brokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(mq);
+ brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(brokerName);
}
SendMessageContext context = null;
@@ -798,10 +758,6 @@ public class DefaultMQProducerImpl implements MQProducerInner {
sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
}
- if (!CommunicationMode.ONEWAY.equals(communicationMode)) {
- sysFlag |= MessageSysFlag.LOGICAL_QUEUE_FLAG;
- }
-
if (hasCheckForbiddenHook()) {
CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
@@ -890,7 +846,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
brokerAddr,
- mq.getBrokerName(),
+ brokerName,
tmpMessage,
requestHeader,
timeout - costTimeAsync,
@@ -910,7 +866,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
brokerAddr,
- mq.getBrokerName(),
+ brokerName,
msg,
requestHeader,
timeout - costTimeSync,
@@ -941,7 +897,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
}
- throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
+ throw new MQClientException("The broker[" + brokerName + "] not exist", null);
}
public MQClientInstance getmQClientFactory() {
@@ -1042,7 +998,6 @@ public class DefaultMQProducerImpl implements MQProducerInner {
executeEndTransactionHook(context);
}
}
-
/**
* DEFAULT ONEWAY -------------------------------------------------------
*/
@@ -1377,7 +1332,8 @@ public class DefaultMQProducerImpl implements MQProducerInner {
id = MessageDecoder.decodeMessageId(sendResult.getMsgId());
}
String transactionId = sendResult.getTransactionId();
- final String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(sendResult.getMessageQueue().getBrokerName());
+ final String destBrokerName = this.mQClientFactory.getBrokerNameFromMessageQueue(defaultMQProducer.queueWithNamespace(sendResult.getMessageQueue()));
+ final String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(destBrokerName);
EndTransactionRequestHeader requestHeader = new EndTransactionRequestHeader();
requestHeader.setTransactionId(transactionId);
requestHeader.setCommitLogOffset(id.getOffset());
@@ -1682,178 +1638,4 @@ public class DefaultMQProducerImpl implements MQProducerInner {
public DefaultMQProducer getDefaultMQProducer() {
return defaultMQProducer;
}
-
- private class LogicalQueueSendContext implements SendCallback {
- private final Message msg;
- private final MessageQueue mq;
- private final CommunicationMode communicationMode;
- private final SendCallback sendCallback;
- private final TopicPublishInfo topicPublishInfo;
- private final long timeout;
-
- private volatile LogicalQueuesInfo logicalQueuesInfo;
- private volatile LogicalQueueRouteData writableQueueRouteData;
-
- private final AtomicInteger retry = new AtomicInteger();
-
- public LogicalQueueSendContext(Message msg, MessageQueue mq,
- CommunicationMode communicationMode, SendCallback sendCallback,
- TopicPublishInfo topicPublishInfo, long timeout) {
- this.msg = msg;
- this.mq = mq;
- this.communicationMode = communicationMode;
- this.sendCallback = sendCallback;
- this.topicPublishInfo = topicPublishInfo;
- this.timeout = timeout;
-
- if (topicPublishInfo == null) {
- topicPublishInfo = DefaultMQProducerImpl.this.tryToFindTopicPublishInfo(mq.getTopic());
- }
- if (topicPublishInfo != null) {
- this.logicalQueuesInfo = topicPublishInfo.getTopicRouteData().getLogicalQueuesInfo();
- } else {
- this.logicalQueuesInfo = null;
- }
- }
-
- private boolean notUsingLogicalQueue() {
- return !Objects.equal(mq.getBrokerName(), MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME) || this.logicalQueuesInfo == null;
- }
-
- public MessageQueue getModifiedMessageQueue() throws MQClientException {
- if (this.notUsingLogicalQueue()) {
- return this.mq;
- }
- this.writableQueueRouteData = getWritableQueueRouteData();
- MessageQueue mq = new MessageQueue(this.mq);
- mq.setBrokerName(writableQueueRouteData.getBrokerName());
- mq.setQueueId(writableQueueRouteData.getQueueId());
- return mq;
- }
-
- private LogicalQueueRouteData getWritableQueueRouteData() throws MQClientException {
- this.logicalQueuesInfo.readLock().lock();
- try {
- List<LogicalQueueRouteData> queueRouteDataList = logicalQueuesInfo.get(mq.getQueueId());
- if (queueRouteDataList == null || queueRouteDataList.size() == 0) {
- throw new MQClientException(String.format(Locale.ENGLISH, "send to a logical queue %d but no queue route data found", mq.getQueueId()), null);
- }
- // usually writable queue is placed in the last position, or second last when queue migrating
- for (int i = queueRouteDataList.size() - 1; i >= 0; i--) {
- LogicalQueueRouteData queueRouteData = queueRouteDataList.get(i);
- if (queueRouteData.isWritable()) {
- return queueRouteData;
- }
- }
- throw new MQClientException(String.format(Locale.ENGLISH, "send to a logical queue %d but no writable queue route data found", mq.getQueueId()), null);
- } finally {
- this.logicalQueuesInfo.readLock().unlock();
- }
- }
-
- @Override public void onSuccess(SendResult sendResult) {
- this.sendCallback.onSuccess(this.wrapSendResult(sendResult));
- }
-
- @Override public void onException(Throwable t) {
- if (this.shouldRetry(t)) {
- try {
- DefaultMQProducerImpl.this.sendKernelImplWithoutRetry(msg, this.getModifiedMessageQueue(), communicationMode, this, topicPublishInfo, timeout);
- return;
- } catch (Exception e) {
- t = e;
- }
- }
- if (t instanceof MQRedirectException) {
- t = new MQBrokerException(ResponseCode.SYSTEM_ERROR, "redirect");
- }
- this.sendCallback.onException(t);
- }
-
- private void handleRedirectException(MQRedirectException re) {
- byte[] responseBody = re.getBody();
- log.info("LogicalQueueContext.processResponseBody got redirect {}: {}", this.writableQueueRouteData, responseBody != null ? new String(responseBody, MessageDecoder.CHARSET_UTF8) : null);
-
- try {
- List<LogicalQueueRouteData> newQueueRouteDataList = JSON.parseObject(responseBody, MixAll.TYPE_LIST_LOGICAL_QUEUE_ROUTE_DATA);
- this.logicalQueuesInfo.updateLogicalQueueRouteDataList(this.mq.getQueueId(), newQueueRouteDataList);
- } catch (Exception e) {
- log.warn("LogicalQueueContext.processResponseBody {} update exception, fallback to updateTopicRouteInfoFromNameServer", this.writableQueueRouteData, e);
- DefaultMQProducerImpl.this.mQClientFactory.updateTopicRouteInfoFromNameServer(this.mq.getTopic(), false, null, Collections.singleton(mq.getQueueId()));
- TopicRouteData topicRouteData = DefaultMQProducerImpl.this.mQClientFactory.getAnExistTopicRouteData(mq.getTopic());
- if (topicRouteData != null) {
- this.logicalQueuesInfo = topicRouteData.getLogicalQueuesInfo();
- } else {
- this.logicalQueuesInfo = null;
- }
- }
- }
-
- public SendCallback wrapSendCallback() {
- if (this.notUsingLogicalQueue()) {
- return this.sendCallback;
- }
- if (!CommunicationMode.ASYNC.equals(this.communicationMode)) {
- return this.sendCallback;
- }
- return this;
- }
-
- public boolean shouldRetry(Throwable t) {
- this.incrRetry();
- if (this.exceedMaxRetry()) {
- log.warn("retry {} too many times: {}", this.retry.get(), this.writableQueueRouteData);
- return false;
- }
- if (!this.writableQueueRouteData.isWritable()) {
- log.warn("no writable queue: {}", this.writableQueueRouteData);
- return false;
- }
- if (t instanceof MQRedirectException) {
- this.handleRedirectException((MQRedirectException) t);
- return true;
- }
- return !(t instanceof RemotingException) || this.handleRemotingException((RemotingException) t);
- }
-
- public boolean exceedMaxRetry() {
- return this.retry.get() >= 3;
- }
-
- public void incrRetry() {
- this.retry.incrementAndGet();
- }
-
- public SendResult wrapSendResult(SendResult sendResult) {
- if (sendResult == null) {
- return null;
- }
- SendResultForLogicalQueue newSendResult = new SendResultForLogicalQueue(sendResult, this.writableQueueRouteData.getLogicalQueueIndex());
- long queueOffset = newSendResult.getQueueOffset();
- if (queueOffset >= 0) {
- newSendResult.setQueueOffset(LogicalQueueSendContext.this.writableQueueRouteData.toLogicalQueueOffset(queueOffset));
- }
- return newSendResult;
- }
-
- public boolean handleRemotingException(RemotingException e) {
- if (e instanceof RemotingTooMuchRequestException) {
- return false;
- }
- DefaultMQProducerImpl.this.mQClientFactory.updateTopicRouteInfoFromNameServer(this.mq.getTopic(), false, null, Collections.singleton(mq.getQueueId()));
- this.logicalQueuesInfo = DefaultMQProducerImpl.this.getTopicPublishInfoTable().get(mq.getTopic()).getTopicRouteData().getLogicalQueuesInfo();
- LogicalQueueRouteData writableQueueRouteData;
- try {
- writableQueueRouteData = this.getWritableQueueRouteData();
- } catch (MQClientException ce) {
- log.warn("getWritableQueueRouteData exception: {}", this.logicalQueuesInfo.get(mq.getQueueId()), ce);
- return false;
- }
- if (Objects.equal(this.writableQueueRouteData.getMessageQueue(), writableQueueRouteData.getMessageQueue()) && writableQueueRouteData.isWritable()) {
- // still same MessageQueue and still writable, no need to retry
- return false;
- }
- return true;
- }
- }
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/producer/SendResultForLogicalQueue.java b/client/src/main/java/org/apache/rocketmq/client/producer/SendResultForLogicalQueue.java
deleted file mode 100644
index 09cd469..0000000
--- a/client/src/main/java/org/apache/rocketmq/client/producer/SendResultForLogicalQueue.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.client.producer;
-
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.message.MessageQueue;
-
-public class SendResultForLogicalQueue extends SendResult {
- private final String origBrokerName;
- private final int origQueueId;
-
- public SendResultForLogicalQueue(SendResult sendResult, int logicalQueueIdx) {
- super(sendResult.getSendStatus(), sendResult.getMsgId(), sendResult.getOffsetMsgId(), new MessageQueue(sendResult.getMessageQueue().getTopic(), MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME, logicalQueueIdx), sendResult.getQueueOffset());
- this.origBrokerName = sendResult.getMessageQueue().getBrokerName();
- this.origQueueId = sendResult.getMessageQueue().getQueueId();
- }
-
- public String getOrigBrokerName() {
- return origBrokerName;
- }
-
- public int getOrigQueueId() {
- return origQueueId;
- }
-
- @Override public String toString() {
- return "SendResultForLogicalQueue{" +
- "origBrokerName='" + origBrokerName + '\'' +
- ", origQueueId=" + origQueueId +
- "} " + super.toString();
- }
-}
diff --git a/client/src/test/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumerLogicalQueueTest.java b/client/src/test/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumerLogicalQueueTest.java
deleted file mode 100644
index 15ec564..0000000
--- a/client/src/test/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumerLogicalQueueTest.java
+++ /dev/null
@@ -1,248 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.client.consumer;
-
-import com.alibaba.fastjson.JSON;
-import com.google.common.collect.ImmutableList;
-import com.google.common.util.concurrent.SettableFuture;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.TimeUnit;
-import org.apache.commons.lang3.reflect.FieldUtils;
-import org.apache.rocketmq.client.ClientConfig;
-import org.apache.rocketmq.client.exception.MQRedirectException;
-import org.apache.rocketmq.client.impl.CommunicationMode;
-import org.apache.rocketmq.client.impl.FindBrokerResult;
-import org.apache.rocketmq.client.impl.MQClientAPIImpl;
-import org.apache.rocketmq.client.impl.MQClientManager;
-import org.apache.rocketmq.client.impl.consumer.PullAPIWrapper;
-import org.apache.rocketmq.client.impl.consumer.PullResultExt;
-import org.apache.rocketmq.client.impl.factory.MQClientInstance;
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageQueue;
-import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.route.BrokerData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
-import org.apache.rocketmq.common.protocol.route.QueueData;
-import org.apache.rocketmq.common.protocol.route.TopicRouteData;
-import org.assertj.core.util.Lists;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.invocation.InvocationOnMock;
-import org.mockito.junit.MockitoJUnitRunner;
-import org.mockito.stubbing.Answer;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
-import static org.mockito.ArgumentMatchers.anyLong;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.ArgumentMatchers.isNull;
-import static org.mockito.Mockito.doAnswer;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.when;
-
-@RunWith(MockitoJUnitRunner.class)
-public class DefaultMQPullConsumerLogicalQueueTest {
- private MQClientInstance mQClientFactory;
- @Mock
- private MQClientAPIImpl mQClientAPIImpl;
- private DefaultMQPullConsumer pullConsumer;
- private String topic;
- private static final String cluster = "DefaultCluster";
- private static final String broker1Name = "BrokerA";
- private static final String broker1Addr = "127.0.0.2:10911";
- private static final String broker2Name = "BrokerB";
- private static final String broker2Addr = "127.0.0.3:10911";
-
- @Before
- public void init() throws Exception {
- topic = "FooBar" + System.nanoTime();
-
- mQClientFactory = spy(MQClientManager.getInstance().getOrCreateMQClientInstance(new ClientConfig()));
-
- FieldUtils.writeField(mQClientFactory, "mQClientAPIImpl", mQClientAPIImpl, true);
-
- pullConsumer = new DefaultMQPullConsumer("FooBarGroup" + System.nanoTime());
- pullConsumer.setNamesrvAddr("127.0.0.1:9876");
- pullConsumer.start();
-
- PullAPIWrapper pullAPIWrapper = pullConsumer.getDefaultMQPullConsumerImpl().getPullAPIWrapper();
- FieldUtils.writeDeclaredField(pullAPIWrapper, "mQClientFactory", mQClientFactory, true);
-
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRouteData());
-
- doReturn(new FindBrokerResult(broker1Addr, false)).when(mQClientFactory).findBrokerAddressInSubscribe(eq(broker1Name), anyLong(), anyBoolean());
- doReturn(new FindBrokerResult(broker2Addr, false)).when(mQClientFactory).findBrokerAddressInSubscribe(eq(broker2Name), anyLong(), anyBoolean());
- }
-
- @After
- public void terminate() {
- pullConsumer.shutdown();
- }
-
- @Test
- public void testStart_OffsetShouldNotNUllAfterStart() {
- Assert.assertNotNull(pullConsumer.getOffsetStore());
- }
-
- @Test
- public void testPullMessage_Success() throws Exception {
- doAnswer(new Answer<PullResultExt>() {
- @Override public PullResultExt answer(InvocationOnMock mock) throws Throwable {
- PullMessageRequestHeader requestHeader = mock.getArgument(1);
- return DefaultMQPullConsumerLogicalQueueTest.this.createPullResult(requestHeader, PullStatus.FOUND, Collections.singletonList(new MessageExt()));
- }
- }).when(mQClientAPIImpl).pullMessage(eq(broker1Addr), any(PullMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC), (PullCallback) isNull());
-
- MessageQueue messageQueue = new MessageQueue(topic, MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME, 0);
- PullResult pullResult = pullConsumer.pull(messageQueue, "*", 1024, 3);
- assertThat(pullResult).isNotNull();
- assertThat(pullResult.getPullStatus()).isEqualTo(PullStatus.FOUND);
- assertThat(pullResult.getNextBeginOffset()).isEqualTo(1024 + 1);
- assertThat(pullResult.getMinOffset()).isEqualTo(123);
- assertThat(pullResult.getMaxOffset()).isEqualTo(2048);
- assertThat(pullResult.getMsgFoundList()).isEqualTo(Collections.emptyList());
- }
-
- @Test
- public void testPullMessage_NotFound() throws Exception {
- doAnswer(new Answer<PullResult>() {
- @Override public PullResult answer(InvocationOnMock mock) throws Throwable {
- PullMessageRequestHeader requestHeader = mock.getArgument(1);
- return DefaultMQPullConsumerLogicalQueueTest.this.createPullResult(requestHeader, PullStatus.NO_NEW_MSG, new ArrayList<MessageExt>());
- }
- }).when(mQClientAPIImpl).pullMessage(eq(broker1Addr), any(PullMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC), (PullCallback) isNull());
-
- MessageQueue messageQueue = new MessageQueue(topic, MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME, 0);
- PullResult pullResult = pullConsumer.pull(messageQueue, "*", 1024, 3);
- assertThat(pullResult.getPullStatus()).isEqualTo(PullStatus.NO_NEW_MSG);
- }
-
- @Test
- public void testPullMessageAsync_Success() throws Exception {
- doAnswer(new Answer<PullResult>() {
- @Override public PullResult answer(InvocationOnMock mock) throws Throwable {
- PullMessageRequestHeader requestHeader = mock.getArgument(1);
- PullResult pullResult = DefaultMQPullConsumerLogicalQueueTest.this.createPullResult(requestHeader, PullStatus.FOUND, Collections.singletonList(new MessageExt()));
-
- PullCallback pullCallback = mock.getArgument(4);
- pullCallback.onSuccess(pullResult);
- return null;
- }
- }).when(mQClientAPIImpl).pullMessage(eq(broker1Addr), any(PullMessageRequestHeader.class), anyLong(), eq(CommunicationMode.ASYNC), any(PullCallback.class));
-
- final SettableFuture<PullResult> future = SettableFuture.create();
- MessageQueue messageQueue = new MessageQueue(topic, broker1Name, 0);
- pullConsumer.pull(messageQueue, "*", 1024, 3, new PullCallback() {
- @Override
- public void onSuccess(PullResult pullResult) {
- future.set(pullResult);
- }
-
- @Override
- public void onException(Throwable e) {
- future.setException(e);
- }
- });
- PullResult pullResult = future.get(3, TimeUnit.SECONDS);
- assertThat(pullResult).isNotNull();
- assertThat(pullResult.getPullStatus()).isEqualTo(PullStatus.FOUND);
- assertThat(pullResult.getNextBeginOffset()).isEqualTo(1024 + 1);
- assertThat(pullResult.getMinOffset()).isEqualTo(123);
- assertThat(pullResult.getMaxOffset()).isEqualTo(2048);
- assertThat(pullResult.getMsgFoundList()).isEqualTo(Collections.emptyList());
- }
-
- @Test
- public void testPullMessageSync_Redirect() throws Exception {
- doAnswer(new Answer<PullResult>() {
- @Override public PullResult answer(InvocationOnMock mock) throws Throwable {
- throw new MQRedirectException(JSON.toJSONBytes(ImmutableList.of(
- new LogicalQueueRouteData(0, 0, new MessageQueue(topic, broker1Name, 0), MessageQueueRouteState.Expired, 0, 0, 0, 0, broker1Addr),
- new LogicalQueueRouteData(0, 10, new MessageQueue(topic, broker2Name, 0), MessageQueueRouteState.Normal, 0, -1, -1, -1, broker2Addr)
- )));
- }
- }).when(mQClientAPIImpl).pullMessage(eq(broker1Addr), any(PullMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC), (PullCallback) isNull());
- doAnswer(new Answer<PullResult>() {
- @Override public PullResult answer(InvocationOnMock mock) throws Throwable {
- PullMessageRequestHeader requestHeader = mock.getArgument(1);
- return DefaultMQPullConsumerLogicalQueueTest.this.createPullResult(requestHeader, PullStatus.FOUND, Collections.singletonList(new MessageExt()));
- }
- }).when(mQClientAPIImpl).pullMessage(eq(broker2Addr), any(PullMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC), (PullCallback) isNull());
-
- MessageQueue messageQueue = new MessageQueue(topic, MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME, 0);
- PullResult pullResult = pullConsumer.pull(messageQueue, "*", 1024, 3);
- assertThat(pullResult).isNotNull();
- assertThat(pullResult.getPullStatus()).isEqualTo(PullStatus.FOUND);
- assertThat(pullResult.getNextBeginOffset()).isEqualTo(1024 + 1);
- assertThat(pullResult.getMinOffset()).isEqualTo(123 + 10);
- assertThat(pullResult.getMaxOffset()).isEqualTo(2048 + 10);
- assertThat(pullResult.getMsgFoundList()).isEqualTo(Collections.emptyList());
- }
-
- private TopicRouteData createTopicRouteData() {
- TopicRouteData topicRouteData = new TopicRouteData();
-
- topicRouteData.setFilterServerTable(new HashMap<String, List<String>>());
- topicRouteData.setBrokerDatas(ImmutableList.of(
- new BrokerData(cluster, broker1Name, new HashMap<Long, String>(Collections.singletonMap(MixAll.MASTER_ID, broker1Addr))),
- new BrokerData(cluster, broker2Name, new HashMap<Long, String>(Collections.singletonMap(MixAll.MASTER_ID, broker2Addr)))
- ));
-
- List<QueueData> queueDataList = new ArrayList<QueueData>();
- QueueData queueData;
- queueData = new QueueData();
- queueData.setBrokerName(broker1Name);
- queueData.setPerm(6);
- queueData.setReadQueueNums(3);
- queueData.setWriteQueueNums(4);
- queueData.setTopicSysFlag(0);
- queueDataList.add(queueData);
- queueData = new QueueData();
- queueData.setBrokerName(broker2Name);
- queueData.setPerm(6);
- queueData.setReadQueueNums(3);
- queueData.setWriteQueueNums(4);
- queueData.setTopicSysFlag(0);
- queueDataList.add(queueData);
- topicRouteData.setQueueDatas(queueDataList);
-
- LogicalQueuesInfo info = new LogicalQueuesInfo();
- info.put(0, Lists.newArrayList(new LogicalQueueRouteData(0, 0, new MessageQueue(topic, broker1Name, 0), MessageQueueRouteState.Normal, 0, 0, 0, 0, broker1Addr)));
- topicRouteData.setLogicalQueuesInfo(info);
- return topicRouteData;
- }
-
- private PullResultExt createPullResult(PullMessageRequestHeader requestHeader, PullStatus pullStatus,
- List<MessageExt> messageExtList) throws Exception {
- return new PullResultExt(pullStatus, requestHeader.getQueueOffset() + messageExtList.size(), 123, 2048, messageExtList, 0, new byte[] {});
- }
-}
\ No newline at end of file
diff --git a/client/src/test/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStoreTest.java b/client/src/test/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStoreTest.java
index f762910..73cfefb 100644
--- a/client/src/test/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStoreTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStoreTest.java
@@ -20,10 +20,12 @@ import java.util.Collections;
import java.util.HashSet;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.exception.MQBrokerException;
+import org.apache.rocketmq.client.exception.OffsetNotFoundException;
import org.apache.rocketmq.client.impl.FindBrokerResult;
import org.apache.rocketmq.client.impl.MQClientAPIImpl;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
import org.apache.rocketmq.remoting.exception.RemotingException;
@@ -60,6 +62,7 @@ public class RemoteBrokerOffsetStoreTest {
when(mQClientFactory.getClientId()).thenReturn(clientId);
when(mQClientFactory.findBrokerAddressInAdmin(brokerName)).thenReturn(new FindBrokerResult("127.0.0.1", false));
when(mQClientFactory.getMQClientAPIImpl()).thenReturn(mqClientAPI);
+ when(mQClientFactory.getBrokerNameFromMessageQueue(any())).thenReturn(brokerName);
}
@Test
@@ -84,10 +87,15 @@ public class RemoteBrokerOffsetStoreTest {
offsetStore.updateOffset(messageQueue, 1024, false);
- doThrow(new MQBrokerException(-1, "", null))
+ doThrow(new OffsetNotFoundException(ResponseCode.PULL_NOT_FOUND, "", null))
.when(mqClientAPI).queryConsumerOffset(anyString(), any(QueryConsumerOffsetRequestHeader.class), anyLong());
assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_STORE)).isEqualTo(-1);
+
+ doThrow(new MQBrokerException(-1, "", null))
+ .when(mqClientAPI).queryConsumerOffset(anyString(), any(QueryConsumerOffsetRequestHeader.class), anyLong());
+ assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_STORE)).isEqualTo(-2);
+
doThrow(new RemotingException("", null))
.when(mqClientAPI).queryConsumerOffset(anyString(), any(QueryConsumerOffsetRequestHeader.class), anyLong());
assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_STORE)).isEqualTo(-2);
diff --git a/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerLogicalQueueTest.java b/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerLogicalQueueTest.java
deleted file mode 100644
index 12d5cba..0000000
--- a/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerLogicalQueueTest.java
+++ /dev/null
@@ -1,311 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.client.producer;
-
-import com.alibaba.fastjson.JSON;
-import com.google.common.collect.ImmutableList;
-import com.google.common.util.concurrent.SettableFuture;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.commons.lang3.reflect.FieldUtils;
-import org.apache.rocketmq.client.ClientConfig;
-import org.apache.rocketmq.client.exception.MQBrokerException;
-import org.apache.rocketmq.client.exception.MQClientException;
-import org.apache.rocketmq.client.exception.MQRedirectException;
-import org.apache.rocketmq.client.hook.SendMessageContext;
-import org.apache.rocketmq.client.impl.CommunicationMode;
-import org.apache.rocketmq.client.impl.MQClientAPIImpl;
-import org.apache.rocketmq.client.impl.MQClientManager;
-import org.apache.rocketmq.client.impl.factory.MQClientInstance;
-import org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl;
-import org.apache.rocketmq.client.impl.producer.TopicPublishInfo;
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.message.Message;
-import org.apache.rocketmq.common.message.MessageQueue;
-import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
-import org.apache.rocketmq.common.protocol.route.BrokerData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
-import org.apache.rocketmq.common.protocol.route.QueueData;
-import org.apache.rocketmq.common.protocol.route.TopicRouteData;
-import org.apache.rocketmq.remoting.exception.RemotingConnectException;
-import org.assertj.core.api.ThrowableAssert;
-import org.assertj.core.util.Lists;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
-import org.mockito.Mock;
-import org.mockito.invocation.InvocationOnMock;
-import org.mockito.junit.MockitoJUnitRunner;
-import org.mockito.stubbing.Answer;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import static org.assertj.core.api.Assertions.assertThatThrownBy;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyLong;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.ArgumentMatchers.isNull;
-import static org.mockito.ArgumentMatchers.nullable;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.when;
-
-@RunWith(MockitoJUnitRunner.class)
-public class DefaultMQProducerLogicalQueueTest {
- private MQClientInstance mQClientFactory;
- @Mock
- private MQClientAPIImpl mQClientAPIImpl;
-
- private DefaultMQProducer producer;
- private Message message;
- private String topic;
-
- private MessageQueue messageQueue;
-
- private static final String cluster = "DefaultCluster";
- private static final String broker1Name = "broker1";
- private static final String broker2Name = "broker2";
- private static final String broker1Addr = "127.0.0.2:10911";
- private static final String broker2Addr = "127.0.0.3:10911";
-
- @Before
- public void init() throws Exception {
- topic = "Foobar" + System.nanoTime();
- messageQueue = new MessageQueue(topic, MixAll.LOGICAL_QUEUE_MOCK_BROKER_NAME, 0);
-
- ConcurrentMap<String, MQClientInstance> factoryTable = (ConcurrentMap<String/* clientId */, MQClientInstance>) FieldUtils.readDeclaredField(MQClientManager.getInstance(), "factoryTable", true);
- for (MQClientInstance instance : factoryTable.values()) {
- instance.shutdown();
- }
- factoryTable.clear();
-
- mQClientFactory = spy(MQClientManager.getInstance().getOrCreateMQClientInstance(new ClientConfig()));
- factoryTable.put(new ClientConfig().buildMQClientId(), mQClientFactory);
-
- String producerGroupTemp = "FooBar_PID" + System.nanoTime();
- producer = new DefaultMQProducer(producerGroupTemp);
- producer.setNamesrvAddr("127.0.0.1:9876");
- producer.setCompressMsgBodyOverHowmuch(Integer.MAX_VALUE);
- message = new Message(topic, new byte[] {'a'});
-
- mQClientFactory.registerProducer(producerGroupTemp, producer.getDefaultMQProducerImpl());
-
- producer.start();
-
- FieldUtils.writeDeclaredField(producer.getDefaultMQProducerImpl(), "mQClientFactory", mQClientFactory, true);
- FieldUtils.writeField(mQClientFactory, "mQClientAPIImpl", mQClientAPIImpl, true);
-
- when(mQClientAPIImpl.sendMessage(anyString(), anyString(), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), any(CommunicationMode.class),
- nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class))).thenCallRealMethod();
- when(mQClientAPIImpl.sendMessage(anyString(), anyString(), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), any(CommunicationMode.class),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenReturn(createSendResult(SendStatus.SEND_OK));
- when(mQClientAPIImpl.sendMessage(anyString(), anyString(), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), any(CommunicationMode.class),
- any(SendCallback.class), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenAnswer(new Answer<SendResult>() {
- @Override public SendResult answer(InvocationOnMock invocation) throws Throwable {
- SendCallback sendCallback = invocation.getArgument(6);
- sendCallback.onSuccess(DefaultMQProducerLogicalQueueTest.this.createSendResult(SendStatus.SEND_OK));
- return null;
- }
- });
- }
-
- @After
- public void terminate() {
- producer.shutdown();
- }
-
- @Test
- public void testSendMessageSync_Success() throws Exception {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
- SendResult sendResult = producer.send(message, messageQueue);
-
- assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
- assertThat(sendResult.getOffsetMsgId()).isEqualTo("123");
- assertThat(sendResult.getQueueOffset()).isEqualTo(456L);
- }
-
- @Test
- public void testSendMessageSync_Redirect() throws Exception {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
-
- when(mQClientAPIImpl.sendMessage(eq(broker1Addr), eq(broker1Name), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenThrow(new MQRedirectException(null));
-
- assertThatThrownBy(new ThrowableAssert.ThrowingCallable() {
- @Override public void call() throws Throwable {
- producer.send(message, messageQueue);
- }
- }).isInstanceOf(MQBrokerException.class).hasMessageContaining("redirect");
-
- when(mQClientAPIImpl.sendMessage(eq(broker1Addr), eq(broker1Name), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenThrow(new MQRedirectException(JSON.toJSONBytes(ImmutableList.of(
- new LogicalQueueRouteData(0, 0, new MessageQueue(topic, broker1Name, 0), MessageQueueRouteState.Expired, 0, 0, 0, 0, broker1Addr),
- new LogicalQueueRouteData(0, 10, new MessageQueue(topic, broker2Name, 0), MessageQueueRouteState.Normal, 0, -1, -1, -1, broker2Addr)))));
- when(mQClientAPIImpl.sendMessage(eq(broker2Addr), eq(broker2Name), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenReturn(createSendResult(SendStatus.SEND_OK));
-
- SendResult sendResult = producer.send(message, messageQueue);
- assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
- assertThat(sendResult.getOffsetMsgId()).isEqualTo("123");
- assertThat(sendResult.getQueueOffset()).isEqualTo(466L);
- }
-
- @Test
- public void testSendMessageSync_RemotingException() throws Exception {
- TopicRouteData topicRouteData = createTopicRoute();
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(topicRouteData);
-
- when(mQClientAPIImpl.sendMessage(eq(broker1Addr), eq(broker1Name), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenThrow(new RemotingConnectException(broker1Addr));
- SendResult returnSendResult = createSendResult(SendStatus.SEND_OK);
- when(mQClientAPIImpl.sendMessage(eq(broker2Addr), eq(broker2Name), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), eq(CommunicationMode.SYNC),
- (SendCallback) isNull(), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
- .thenReturn(returnSendResult);
-
- assertThatThrownBy(new ThrowableAssert.ThrowingCallable() {
- @Override public void call() throws Throwable {
- producer.send(message, messageQueue);
- }
- }).isInstanceOf(RemotingConnectException.class).hasMessageContaining(broker1Addr);
-
- topicRouteData.getLogicalQueuesInfo().get(0).add(new LogicalQueueRouteData(0, -1, new MessageQueue(topic, broker2Name, 1), MessageQueueRouteState.WriteOnly, 0, -1, -1, -1, broker2Addr));
-
- SendResult sendResult = producer.send(message, messageQueue);
- assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
- assertThat(sendResult.getOffsetMsgId()).isEqualTo("123");
- assertThat(sendResult.getQueueOffset()).isEqualTo(-1L);
- }
-
- @Test
- public void testSendMessageAsync_Success() throws Exception {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
-
- final SettableFuture<SendResult> future = SettableFuture.create();
- producer.send(message, messageQueue, new SendCallback() {
- @Override
- public void onSuccess(SendResult sendResult) {
- future.set(sendResult);
- }
-
- @Override
- public void onException(Throwable e) {
- future.setException(e);
- }
- });
-
- SendResult sendResult = future.get(3, TimeUnit.SECONDS);
- assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
- assertThat(sendResult.getOffsetMsgId()).isEqualTo("123");
- assertThat(sendResult.getQueueOffset()).isEqualTo(456L);
- }
-
- @Test
- public void testSendMessageAsync() throws Exception {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
-
- final AtomicReference<SettableFuture<SendResult>> future = new AtomicReference<SettableFuture<SendResult>>();
- SendCallback sendCallback = new SendCallback() {
- @Override
- public void onSuccess(SendResult sendResult) {
- future.get().set(sendResult);
- }
-
- @Override
- public void onException(Throwable e) {
- future.get().setException(e);
- }
- };
-
- Message message = new Message();
- message.setTopic("test");
- message.setBody("hello world".getBytes());
- future.set(SettableFuture.<SendResult>create());
- producer.send(new Message(), messageQueue, sendCallback);
- assertThatThrownBy(new ThrowableAssert.ThrowingCallable() {
- @Override public void call() throws Throwable {
- future.get().get(3, TimeUnit.SECONDS);
- }
- }).hasCauseInstanceOf(MQClientException.class).hasMessageContaining("The specified topic is blank");
-
- //this message is send success
- message.setTopic(topic);
- future.set(SettableFuture.<SendResult>create());
- producer.send(message, messageQueue, sendCallback, 1000);
- future.get().get(3, TimeUnit.SECONDS);
- }
-
- public TopicRouteData createTopicRoute() {
- TopicRouteData topicRouteData = new TopicRouteData();
-
- topicRouteData.setFilterServerTable(new HashMap<String, List<String>>());
- topicRouteData.setBrokerDatas(ImmutableList.of(
- new BrokerData(cluster, broker1Name, new HashMap<Long, String>(Collections.singletonMap(MixAll.MASTER_ID, broker1Addr))),
- new BrokerData(cluster, broker2Name, new HashMap<Long, String>(Collections.singletonMap(MixAll.MASTER_ID, broker2Addr)))
- ));
-
- List<QueueData> queueDataList = new ArrayList<QueueData>();
- QueueData queueData;
- queueData = new QueueData();
- queueData.setBrokerName(broker1Name);
- queueData.setPerm(6);
- queueData.setReadQueueNums(3);
- queueData.setWriteQueueNums(4);
- queueData.setTopicSysFlag(0);
- queueDataList.add(queueData);
- queueData = new QueueData();
- queueData.setBrokerName(broker2Name);
- queueData.setPerm(6);
- queueData.setReadQueueNums(3);
- queueData.setWriteQueueNums(4);
- queueData.setTopicSysFlag(0);
- queueDataList.add(queueData);
- topicRouteData.setQueueDatas(queueDataList);
-
- LogicalQueuesInfo info = new LogicalQueuesInfo();
- info.put(0, Lists.newArrayList(new LogicalQueueRouteData(0, 0, new MessageQueue(topic, broker1Name, 0), MessageQueueRouteState.Normal, 0, 0, 0, 0, broker1Addr)));
- topicRouteData.setLogicalQueuesInfo(info);
- return topicRouteData;
- }
-
- private SendResult createSendResult(SendStatus sendStatus) {
- SendResult sendResult = new SendResult();
- sendResult.setMsgId("123");
- sendResult.setOffsetMsgId("123");
- sendResult.setQueueOffset(456);
- sendResult.setSendStatus(sendStatus);
- sendResult.setRegionId("HZ");
- sendResult.setMessageQueue(new MessageQueue(topic, broker1Name, 0));
- return sendResult;
- }
-}
diff --git a/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerTest.java b/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerTest.java
index a8906b3..5f29fe1 100644
--- a/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/producer/DefaultMQProducerTest.java
@@ -22,7 +22,6 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
@@ -49,20 +48,21 @@ import org.apache.rocketmq.common.protocol.route.BrokerData;
import org.apache.rocketmq.common.protocol.route.QueueData;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
import org.apache.rocketmq.remoting.exception.RemotingException;
+import org.apache.rocketmq.remoting.exception.RemotingSendRequestException;
import org.apache.rocketmq.remoting.netty.NettyRemotingClient;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
import org.mockito.Mock;
import org.mockito.Spy;
+import org.mockito.invocation.InvocationOnMock;
import org.mockito.junit.MockitoJUnitRunner;
+import org.mockito.stubbing.Answer;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Fail.failBecauseExceptionWasNotThrown;
import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
@@ -153,7 +153,7 @@ public class DefaultMQProducerTest {
@Test
public void testSendMessageSync_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
SendResult sendResult = producer.send(message);
assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
@@ -163,7 +163,7 @@ public class DefaultMQProducerTest {
@Test
public void testSendMessageSync_WithBodyCompressed() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
SendResult sendResult = producer.send(bigMessage);
assertThat(sendResult.getSendStatus()).isEqualTo(SendStatus.SEND_OK);
@@ -174,7 +174,7 @@ public class DefaultMQProducerTest {
@Test
public void testSendMessageAsync_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
final CountDownLatch countDownLatch = new CountDownLatch(1);
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
producer.send(message, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
@@ -197,7 +197,7 @@ public class DefaultMQProducerTest {
final AtomicInteger cc = new AtomicInteger(0);
final CountDownLatch countDownLatch = new CountDownLatch(6);
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
SendCallback sendCallback = new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
@@ -239,7 +239,7 @@ public class DefaultMQProducerTest {
final AtomicInteger cc = new AtomicInteger(0);
final CountDownLatch countDownLatch = new CountDownLatch(4);
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
SendCallback sendCallback = new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
@@ -260,7 +260,7 @@ public class DefaultMQProducerTest {
}
};
- List<Message> msgs = new ArrayList<Message>();
+ List<Message> msgs = new ArrayList<>();
for (int i = 0; i < 5; i++) {
Message message = new Message();
message.setTopic("test");
@@ -281,7 +281,7 @@ public class DefaultMQProducerTest {
@Test
public void testSendMessageAsync_BodyCompressed() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
final CountDownLatch countDownLatch = new CountDownLatch(1);
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
producer.send(bigMessage, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
@@ -300,7 +300,7 @@ public class DefaultMQProducerTest {
@Test
public void testSendMessageSync_SuccessWithHook() throws Throwable {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
final Throwable[] assertionErrors = new Throwable[1];
final CountDownLatch countDownLatch = new CountDownLatch(2);
producer.getDefaultMQProducerImpl().registerSendMessageHook(new SendMessageHook() {
@@ -368,7 +368,7 @@ public class DefaultMQProducerTest {
@Test
public void testRequestMessage() throws RemotingException, RequestTimeoutException, MQClientException, InterruptedException, MQBrokerException {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
final AtomicBoolean finish = new AtomicBoolean(false);
new Thread(new Runnable() {
@Override public void run() {
@@ -394,13 +394,13 @@ public class DefaultMQProducerTest {
@Test(expected = RequestTimeoutException.class)
public void testRequestMessage_RequestTimeoutException() throws RemotingException, RequestTimeoutException, MQClientException, InterruptedException, MQBrokerException {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
Message result = producer.request(message, 3 * 1000L);
}
@Test
public void testAsyncRequest_OnSuccess() throws Exception {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
final CountDownLatch countDownLatch = new CountDownLatch(1);
RequestCallback requestCallback = new RequestCallback() {
@Override public void onSuccess(Message message) {
diff --git a/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithOpenTracingTest.java b/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithOpenTracingTest.java
index 0a1f685..5d64a93 100644
--- a/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithOpenTracingTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithOpenTracingTest.java
@@ -20,11 +20,6 @@ package org.apache.rocketmq.client.trace;
import io.opentracing.mock.MockSpan;
import io.opentracing.mock.MockTracer;
import io.opentracing.tag.Tags;
-import java.lang.reflect.Field;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
@@ -52,14 +47,17 @@ import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
@@ -115,7 +113,7 @@ public class DefaultMQProducerWithOpenTracingTest {
@Test
public void testSendMessageSync_WithTrace_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
producer.getDefaultMQProducerImpl().getmQClientFactory().registerProducer(producerGroupTraceTemp, producer.getDefaultMQProducerImpl());
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
producer.send(message);
assertThat(tracer.finishedSpans().size()).isEqualTo(1);
MockSpan span = tracer.finishedSpans().get(0);
diff --git a/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithTraceTest.java b/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithTraceTest.java
index 64f63c5..234e32e 100644
--- a/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithTraceTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/trace/DefaultMQProducerWithTraceTest.java
@@ -17,13 +17,6 @@
package org.apache.rocketmq.client.trace;
-import java.lang.reflect.Field;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
@@ -50,17 +43,18 @@ import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyLong;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.nullable;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
@@ -128,7 +122,7 @@ public class DefaultMQProducerWithTraceTest {
@Test
public void testSendMessageSync_WithTrace_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
traceProducer.getDefaultMQProducerImpl().getmQClientFactory().registerProducer(producerGroupTraceTemp, traceProducer.getDefaultMQProducerImpl());
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
final CountDownLatch countDownLatch = new CountDownLatch(1);
try {
producer.send(message);
@@ -140,7 +134,7 @@ public class DefaultMQProducerWithTraceTest {
@Test
public void testSendMessageSync_WithTrace_NoBrokerSet_Exception() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
final CountDownLatch countDownLatch = new CountDownLatch(1);
try {
producer.send(message);
diff --git a/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithOpenTracingTest.java b/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithOpenTracingTest.java
index aca6254..dd6d108 100644
--- a/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithOpenTracingTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithOpenTracingTest.java
@@ -20,12 +20,6 @@ package org.apache.rocketmq.client.trace;
import io.opentracing.mock.MockSpan;
import io.opentracing.mock.MockTracer;
import io.opentracing.tag.Tags;
-import java.lang.reflect.Field;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
@@ -59,14 +53,18 @@ import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
+import java.lang.reflect.Field;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
@@ -133,7 +131,7 @@ public class TransactionMQProducerWithOpenTracingTest {
@Test
public void testSendMessageSync_WithTrace_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
producer.getDefaultMQProducerImpl().getmQClientFactory().registerProducer(producerGroupTraceTemp, producer.getDefaultMQProducerImpl());
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
producer.sendMessageInTransaction(message, null);
assertThat(tracer.finishedSpans().size()).isEqualTo(2);
diff --git a/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithTraceTest.java b/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithTraceTest.java
index b3a4414..f838817 100644
--- a/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithTraceTest.java
+++ b/client/src/test/java/org/apache/rocketmq/client/trace/TransactionMQProducerWithTraceTest.java
@@ -17,13 +17,6 @@
package org.apache.rocketmq.client.trace;
-import java.lang.reflect.Field;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicReference;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
@@ -57,20 +50,19 @@ import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.mockito.ArgumentMatchers;
import org.mockito.Mock;
import org.mockito.Spy;
-import org.mockito.invocation.InvocationOnMock;
import org.mockito.junit.MockitoJUnitRunner;
-import org.mockito.stubbing.Answer;
+
+import java.lang.reflect.Field;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyBoolean;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyLong;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.when;
@@ -135,7 +127,7 @@ public class TransactionMQProducerWithTraceTest {
Field fieldHooks = DefaultMQProducerImpl.class.getDeclaredField("endTransactionHookList");
fieldHooks.setAccessible(true);
- List<EndTransactionHook>hooks = new ArrayList<EndTransactionHook>();
+ List<EndTransactionHook>hooks = new ArrayList<>();
hooks.add(endTransactionHook);
fieldHooks.set(producer.getDefaultMQProducerImpl(), hooks);
@@ -150,14 +142,12 @@ public class TransactionMQProducerWithTraceTest {
@Test
public void testSendMessageSync_WithTrace_Success() throws RemotingException, InterruptedException, MQBrokerException, MQClientException {
traceProducer.getDefaultMQProducerImpl().getmQClientFactory().registerProducer(producerGroupTraceTemp, traceProducer.getDefaultMQProducerImpl());
- when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong(), anyBoolean(), ArgumentMatchers.<Set<Integer>>any())).thenReturn(createTopicRoute());
- final AtomicReference<EndTransactionContext> context = new AtomicReference<EndTransactionContext>();
- doAnswer(new Answer() {
- @Override public Object answer(InvocationOnMock mock) throws Throwable {
- context.set(mock.<EndTransactionContext>getArgument(0));
- return null;
- }
- }).when(endTransactionHook).endTransaction(ArgumentMatchers.<EndTransactionContext>any());
+ when(mQClientAPIImpl.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(createTopicRoute());
+ AtomicReference<EndTransactionContext> context = new AtomicReference<>();
+ doAnswer(mock -> {
+ context.set(mock.getArgument(0));
+ return null;
+ }).when(endTransactionHook).endTransaction(any());
producer.sendMessageInTransaction(message, null);
EndTransactionContext ctx = context.get();
diff --git a/common/src/main/java/org/apache/rocketmq/common/MixAll.java b/common/src/main/java/org/apache/rocketmq/common/MixAll.java
index 8a7183f..cc12077 100644
--- a/common/src/main/java/org/apache/rocketmq/common/MixAll.java
+++ b/common/src/main/java/org/apache/rocketmq/common/MixAll.java
@@ -16,7 +16,12 @@
*/
package org.apache.rocketmq.common;
-import com.alibaba.fastjson.TypeReference;
+import org.apache.rocketmq.common.annotation.ImportantField;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.help.FAQUrl;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
@@ -27,7 +32,6 @@ import java.lang.annotation.Annotation;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
-import java.lang.reflect.Type;
import java.net.Inet6Address;
import java.net.InetAddress;
import java.net.NetworkInterface;
@@ -40,12 +44,6 @@ import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.atomic.AtomicLong;
-import org.apache.rocketmq.common.annotation.ImportantField;
-import org.apache.rocketmq.common.constant.LoggerName;
-import org.apache.rocketmq.common.help.FAQUrl;
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.logging.InternalLogger;
-import org.apache.rocketmq.logging.InternalLoggerFactory;
public class MixAll {
public static final String ROCKETMQ_HOME_ENV = "ROCKETMQ_HOME";
@@ -87,9 +85,9 @@ public class MixAll {
public static final String ACL_CONF_TOOLS_FILE = "/conf/tools.yml";
public static final String REPLY_MESSAGE_FLAG = "reply";
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.COMMON_LOGGER_NAME);
- public static final String LOGICAL_QUEUE_MOCK_BROKER_NAME = "__logical_queue_broker__";
- public static final Type TYPE_LIST_LOGICAL_QUEUE_ROUTE_DATA = new TypeReference<List<LogicalQueueRouteData>>() {
- }.getType();
+ public static final String LOGICAL_QUEUE_MOCK_BROKER_PREFIX = "__syslo__";
+ public static final String METADATA_SCOPE_GLOBAL = "__global__";
+ public static final String LOGICAL_QUEUE_MOCK_BROKER_NAME_NOT_EXIST = "__syslo__none__";
public static String getWSAddr() {
String wsDomainName = System.getProperty("rocketmq.namesrv.domain", DEFAULT_NAMESRV_ADDR_LOOKUP);
diff --git a/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java b/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
index c082ba6..ec4d54b 100644
--- a/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
+++ b/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
@@ -47,6 +47,12 @@ public class TopicConfig {
this.order = other.order;
}
+ public TopicConfig(String topicName, int readQueueNums, int writeQueueNums) {
+ this.topicName = topicName;
+ this.readQueueNums = readQueueNums;
+ this.writeQueueNums = writeQueueNums;
+ }
+
public TopicConfig(String topicName, int readQueueNums, int writeQueueNums, int perm) {
this.topicName = topicName;
this.readQueueNums = readQueueNums;
diff --git a/common/src/main/java/org/apache/rocketmq/common/admin/TopicOffset.java b/common/src/main/java/org/apache/rocketmq/common/admin/TopicOffset.java
index 7e66749..8b52a88 100644
--- a/common/src/main/java/org/apache/rocketmq/common/admin/TopicOffset.java
+++ b/common/src/main/java/org/apache/rocketmq/common/admin/TopicOffset.java
@@ -44,4 +44,13 @@ public class TopicOffset {
public void setLastUpdateTimestamp(long lastUpdateTimestamp) {
this.lastUpdateTimestamp = lastUpdateTimestamp;
}
+
+ @Override
+ public String toString() {
+ return "TopicOffset{" +
+ "minOffset=" + minOffset +
+ ", maxOffset=" + maxOffset +
+ ", lastUpdateTimestamp=" + lastUpdateTimestamp +
+ '}';
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/RequestCode.java b/common/src/main/java/org/apache/rocketmq/common/protocol/RequestCode.java
index 04f126b..534a2d2 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/RequestCode.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/RequestCode.java
@@ -200,13 +200,6 @@ public class RequestCode {
public static final int QUERY_ASSIGNMENT = 400;
public static final int SET_MESSAGE_REQUEST_MODE = 401;
- public static final int UPDATE_TOPIC_LOGICAL_QUEUE_MAPPING = 411;
- public static final int DELETE_TOPIC_LOGICAL_QUEUE_MAPPING = 422;
- public static final int QUERY_TOPIC_LOGICAL_QUEUE_MAPPING = 413;
- public static final int SEAL_TOPIC_LOGICAL_QUEUE = 414;
- public static final int REUSE_TOPIC_LOGICAL_QUEUE = 415;
- public static final int CREATE_MESSAGE_QUEUE_FOR_LOGICAL_QUEUE = 416;
- public static final int MIGRATE_TOPIC_LOGICAL_QUEUE_PREPARE = 417;
- public static final int MIGRATE_TOPIC_LOGICAL_QUEUE_COMMIT = 418;
- public static final int MIGRATE_TOPIC_LOGICAL_QUEUE_NOTIFY = 419;
+ public static final int UPDATE_AND_CREATE_STATIC_TOPIC = 513;
+
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/ResponseCode.java b/common/src/main/java/org/apache/rocketmq/common/protocol/ResponseCode.java
index 42b9c4f..3f691b5 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/ResponseCode.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/ResponseCode.java
@@ -84,4 +84,13 @@ public class ResponseCode extends RemotingSysResponseCode {
public static final int POLLING_FULL = 209;
public static final int POLLING_TIMEOUT = 210;
+
+ public static final int NOT_LEADER_FOR_QUEUE = 501;
+
+ public static final int RPC_UNKNOWN = -1000;
+ public static final int RPC_ADDR_IS_NULL = -1002;
+ public static final int RPC_SEND_TO_CHANNEL_FAILED = -1004;
+ public static final int RPC_TIME_OUT = -1006;
+
+
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/CreateMessageQueueForLogicalQueueRequestBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/CreateMessageQueueForLogicalQueueRequestBody.java
deleted file mode 100644
index e446d9b..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/CreateMessageQueueForLogicalQueueRequestBody.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.body;
-
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-
-public class CreateMessageQueueForLogicalQueueRequestBody extends RemotingSerializable {
- private String topic;
- private int logicalQueueIndex;
- private MessageQueueRouteState messageQueueStatus;
-
- public String getTopic() {
- return topic;
- }
-
- public void setTopic(String topic) {
- this.topic = topic;
- }
-
- public int getLogicalQueueIndex() {
- return logicalQueueIndex;
- }
-
- public void setLogicalQueueIndex(int logicalQueueIndex) {
- this.logicalQueueIndex = logicalQueueIndex;
- }
-
- public MessageQueueRouteState getMessageQueueStatus() {
- return messageQueueStatus;
- }
-
- public void setMessageQueueStatus(MessageQueueRouteState messageQueueStatuses) {
- this.messageQueueStatus = messageQueueStatuses;
- }
-}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/MigrateLogicalQueueBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/MigrateLogicalQueueBody.java
deleted file mode 100644
index 6eb06a5..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/MigrateLogicalQueueBody.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.body;
-
-import org.apache.rocketmq.common.protocol.route.LogicalQueueRouteData;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-
-public class MigrateLogicalQueueBody extends RemotingSerializable {
- private LogicalQueueRouteData fromQueueRouteData;
- private LogicalQueueRouteData toQueueRouteData;
-
- public LogicalQueueRouteData getFromQueueRouteData() {
- return fromQueueRouteData;
- }
-
- public void setFromQueueRouteData(
- LogicalQueueRouteData fromQueueRouteData) {
- this.fromQueueRouteData = fromQueueRouteData;
- }
-
- public LogicalQueueRouteData getToQueueRouteData() {
- return toQueueRouteData;
- }
-
- public void setToQueueRouteData(LogicalQueueRouteData toQueueRouteData) {
- this.toQueueRouteData = toQueueRouteData;
- }
-}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/RegisterBrokerBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/RegisterBrokerBody.java
index 4065c08..1921727 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/RegisterBrokerBody.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/body/RegisterBrokerBody.java
@@ -33,6 +33,7 @@ import java.util.zip.InflaterInputStream;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingInfo;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
@@ -41,7 +42,7 @@ import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
public class RegisterBrokerBody extends RemotingSerializable {
private static final InternalLogger LOGGER = InternalLoggerFactory.getLogger(LoggerName.COMMON_LOGGER_NAME);
- private TopicConfigSerializeWrapper topicConfigSerializeWrapper = new TopicConfigSerializeWrapper();
+ private TopicConfigAndMappingSerializeWrapper topicConfigSerializeWrapper = new TopicConfigAndMappingSerializeWrapper();
private List<String> filterServerList = new ArrayList<String>();
public byte[] encode(boolean compress) {
@@ -82,6 +83,20 @@ public class RegisterBrokerBody extends RemotingSerializable {
// write filter server list json
outputStream.write(buffer);
+ //write the topic queue mapping
+ Map<String, TopicQueueMappingInfo> topicQueueMappingInfoMap = topicConfigSerializeWrapper.getTopicQueueMappingInfoMap();
+ if (topicQueueMappingInfoMap == null) {
+ //as the place holder
+ topicQueueMappingInfoMap = new ConcurrentHashMap<String, TopicQueueMappingInfo>();
+ }
+ outputStream.write(convertIntToByteArray(topicQueueMappingInfoMap.size()));
+ for (TopicQueueMappingInfo info: topicQueueMappingInfoMap.values()) {
+ buffer = JSON.toJSONString(info).getBytes(MixAll.DEFAULT_CHARSET);
+ outputStream.write(convertIntToByteArray(buffer.length));
+ // write filter server list json
+ outputStream.write(buffer);
+ }
+
outputStream.finish();
long interval = System.currentTimeMillis() - start;
if (interval > 50) {
@@ -134,6 +149,17 @@ public class RegisterBrokerBody extends RemotingSerializable {
}
registerBrokerBody.setFilterServerList(filterServerList);
+
+ int topicQueueMappingNum = readInt(inflaterInputStream);
+ Map<String/* topic */, TopicQueueMappingInfo> topicQueueMappingInfoMap = new ConcurrentHashMap<String, TopicQueueMappingInfo>();
+ for (int i = 0; i < topicQueueMappingNum; i++) {
+ int mappingJsonLen = readInt(inflaterInputStream);
+ byte[] buffer = readBytes(inflaterInputStream, mappingJsonLen);
+ TopicQueueMappingInfo info = TopicQueueMappingInfo.decode(buffer, TopicQueueMappingInfo.class);
+ topicQueueMappingInfoMap.put(info.getTopic(), info);
+ }
+ registerBrokerBody.getTopicConfigSerializeWrapper().setTopicQueueMappingInfoMap(topicQueueMappingInfoMap);
+
long interval = System.currentTimeMillis() - start;
if (interval > 50) {
LOGGER.info("Decompressing takes {}ms", interval);
@@ -167,11 +193,11 @@ public class RegisterBrokerBody extends RemotingSerializable {
return byteBuffer.getInt();
}
- public TopicConfigSerializeWrapper getTopicConfigSerializeWrapper() {
+ public TopicConfigAndMappingSerializeWrapper getTopicConfigSerializeWrapper() {
return topicConfigSerializeWrapper;
}
- public void setTopicConfigSerializeWrapper(TopicConfigSerializeWrapper topicConfigSerializeWrapper) {
+ public void setTopicConfigSerializeWrapper(TopicConfigAndMappingSerializeWrapper topicConfigSerializeWrapper) {
this.topicConfigSerializeWrapper = topicConfigSerializeWrapper;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/ReuseTopicLogicalQueueRequestBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/ReuseTopicLogicalQueueRequestBody.java
deleted file mode 100644
index 22ab452..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/ReuseTopicLogicalQueueRequestBody.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.body;
-
-import org.apache.rocketmq.common.protocol.route.MessageQueueRouteState;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-
-public class ReuseTopicLogicalQueueRequestBody extends RemotingSerializable {
- private String topic;
- private int queueId;
- private int logicalQueueIndex;
- private MessageQueueRouteState messageQueueRouteState;
-
- public String getTopic() {
- return topic;
- }
-
- public void setTopic(String topic) {
- this.topic = topic;
- }
-
- public int getQueueId() {
- return queueId;
- }
-
- public void setQueueId(int queueId) {
- this.queueId = queueId;
- }
-
- public int getLogicalQueueIndex() {
- return logicalQueueIndex;
- }
-
- public void setLogicalQueueIndex(int logicalQueueIndex) {
- this.logicalQueueIndex = logicalQueueIndex;
- }
-
- public void setMessageQueueRouteState(MessageQueueRouteState messageQueueRouteState) {
- this.messageQueueRouteState = messageQueueRouteState;
- }
-
- public MessageQueueRouteState getMessageQueueRouteState() {
- return messageQueueRouteState;
- }
-}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/SealTopicLogicalQueueRequestBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/SealTopicLogicalQueueRequestBody.java
deleted file mode 100644
index edb521f..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/SealTopicLogicalQueueRequestBody.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.body;
-
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-
-public class SealTopicLogicalQueueRequestBody extends RemotingSerializable {
- private String topic;
- private int queueId;
- private int logicalQueueIndex;
-
- public String getTopic() {
- return topic;
- }
-
- public void setTopic(String topic) {
- this.topic = topic;
- }
-
- public int getQueueId() {
- return queueId;
- }
-
- public void setQueueId(int queueId) {
- this.queueId = queueId;
- }
-
- public int getLogicalQueueIndex() {
- return logicalQueueIndex;
- }
-
- public void setLogicalQueueIndex(int logicalQueueIndex) {
- this.logicalQueueIndex = logicalQueueIndex;
- }
-}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigAndMappingSerializeWrapper.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigAndMappingSerializeWrapper.java
new file mode 100644
index 0000000..830a8a4
--- /dev/null
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigAndMappingSerializeWrapper.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.common.protocol.body;
+
+import org.apache.rocketmq.common.DataVersion;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingInfo;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class TopicConfigAndMappingSerializeWrapper extends TopicConfigSerializeWrapper {
+ private Map<String/* topic */, TopicQueueMappingInfo> topicQueueMappingInfoMap = new ConcurrentHashMap<String, TopicQueueMappingInfo>();
+
+ private Map<String/* topic */, TopicQueueMappingDetail> topicQueueMappingDetailMap = new ConcurrentHashMap<String, TopicQueueMappingDetail>();
+
+ private DataVersion mappingDataVersion = new DataVersion();
+
+
+ public Map<String, TopicQueueMappingInfo> getTopicQueueMappingInfoMap() {
+ return topicQueueMappingInfoMap;
+ }
+
+ public void setTopicQueueMappingInfoMap(Map<String, TopicQueueMappingInfo> topicQueueMappingInfoMap) {
+ this.topicQueueMappingInfoMap = topicQueueMappingInfoMap;
+ }
+
+ public Map<String, TopicQueueMappingDetail> getTopicQueueMappingDetailMap() {
+ return topicQueueMappingDetailMap;
+ }
+
+ public void setTopicQueueMappingDetailMap(Map<String, TopicQueueMappingDetail> topicQueueMappingDetailMap) {
+ this.topicQueueMappingDetailMap = topicQueueMappingDetailMap;
+ }
+
+ public DataVersion getMappingDataVersion() {
+ return mappingDataVersion;
+ }
+
+ public void setMappingDataVersion(DataVersion mappingDataVersion) {
+ this.mappingDataVersion = mappingDataVersion;
+ }
+
+ public static TopicConfigAndMappingSerializeWrapper from(TopicConfigSerializeWrapper wrapper) {
+ if (wrapper instanceof TopicConfigAndMappingSerializeWrapper) {
+ return (TopicConfigAndMappingSerializeWrapper) wrapper;
+ }
+ TopicConfigAndMappingSerializeWrapper mappingSerializeWrapper = new TopicConfigAndMappingSerializeWrapper();
+ mappingSerializeWrapper.setDataVersion(wrapper.getDataVersion());
+ mappingSerializeWrapper.setTopicConfigTable(wrapper.getTopicConfigTable());
+ return mappingSerializeWrapper;
+ }
+}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigSerializeWrapper.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigSerializeWrapper.java
index 1389663..ce12302 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigSerializeWrapper.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicConfigSerializeWrapper.java
@@ -17,18 +17,15 @@
package org.apache.rocketmq.common.protocol.body;
-import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.TopicConfig;
-import org.apache.rocketmq.common.protocol.route.LogicalQueuesInfo;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
public class TopicConfigSerializeWrapper extends RemotingSerializable {
private ConcurrentMap<String, TopicConfig> topicConfigTable =
new ConcurrentHashMap<String, TopicConfig>();
- private Map<String/* topic */, LogicalQueuesInfo> logicalQueuesInfoMap;
private DataVersion dataVersion = new DataVersion();
public ConcurrentMap<String, TopicConfig> getTopicConfigTable() {
@@ -46,12 +43,4 @@ public class TopicConfigSerializeWrapper extends RemotingSerializable {
public void setDataVersion(DataVersion dataVersion) {
this.dataVersion = dataVersion;
}
-
- public Map<String, LogicalQueuesInfo> getLogicalQueuesInfoMap() {
- return logicalQueuesInfoMap;
- }
-
- public void setLogicalQueuesInfoMap(Map<String, LogicalQueuesInfo> logicalQueuesInfoMap) {
- this.logicalQueuesInfoMap = logicalQueuesInfoMap;
- }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/body/UpdateTopicLogicalQueueMappingRequestBody.java b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicQueueMappingSerializeWrapper.java
similarity index 53%
copy from common/src/main/java/org/apache/rocketmq/common/protocol/body/UpdateTopicLogicalQueueMappingRequestBody.java
copy to common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicQueueMappingSerializeWrapper.java
index 67c6fd2..799f81a 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/body/UpdateTopicLogicalQueueMappingRequestBody.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/body/TopicQueueMappingSerializeWrapper.java
@@ -14,36 +14,32 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
package org.apache.rocketmq.common.protocol.body;
+import org.apache.rocketmq.common.DataVersion;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-public class UpdateTopicLogicalQueueMappingRequestBody extends RemotingSerializable {
- private String topic;
- private int queueId;
- private int logicalQueueIdx;
-
- public int getLogicalQueueIdx() {
- return logicalQueueIdx;
- }
+import java.util.Map;
- public void setLogicalQueueIdx(int logicalQueueIdx) {
- this.logicalQueueIdx = logicalQueueIdx;
- }
+public class TopicQueueMappingSerializeWrapper extends RemotingSerializable {
+ private Map<String/* topic */, TopicQueueMappingDetail> topicQueueMappingInfoMap;
+ private DataVersion dataVersion = new DataVersion();
- public String getTopic() {
- return topic;
+ public Map<String, TopicQueueMappingDetail> getTopicQueueMappingInfoMap() {
+ return topicQueueMappingInfoMap;
}
- public void setTopic(String topic) {
- this.topic = topic;
+ public void setTopicQueueMappingInfoMap(Map<String, TopicQueueMappingDetail> topicQueueMappingInfoMap) {
+ this.topicQueueMappingInfoMap = topicQueueMappingInfoMap;
}
- public int getQueueId() {
- return queueId;
+ public DataVersion getDataVersion() {
+ return dataVersion;
}
- public void setQueueId(int queueId) {
- this.queueId = queueId;
+ public void setDataVersion(DataVersion dataVersion) {
+ this.dataVersion = dataVersion;
}
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
index 8894d0b..290ec4c 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
@@ -23,6 +23,7 @@ package org.apache.rocketmq.common.protocol.header;
import org.apache.rocketmq.common.TopicFilterType;
import org.apache.rocketmq.remoting.CommandCustomHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
+import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
public class CreateTopicRequestHeader implements CommandCustomHeader {
@@ -42,6 +43,9 @@ public class CreateTopicRequestHeader implements CommandCustomHeader {
@CFNotNull
private Boolean order = false;
+ @CFNullable
+ private Boolean force = false;
+
@Override
public void checkFields() throws RemotingCommandException {
try {
@@ -118,4 +122,12 @@ public class CreateTopicRequestHeader implements CommandCustomHeader {
public void setOrder(Boolean order) {
this.order = order;
}
+
+ public Boolean getForce() {
+ return force;
+ }
+
+ public void setForce(Boolean force) {
+ this.force = force;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetEarliestMsgStoretimeRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetEarliestMsgStoretimeRequestHeader.java
index c64381f..acf4497 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetEarliestMsgStoretimeRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetEarliestMsgStoretimeRequestHeader.java
@@ -20,11 +20,11 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class GetEarliestMsgStoretimeRequestHeader implements CommandCustomHeader {
+public class GetEarliestMsgStoretimeRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String topic;
@CFNotNull
@@ -34,18 +34,22 @@ public class GetEarliestMsgStoretimeRequestHeader implements CommandCustomHeader
public void checkFields() throws RemotingCommandException {
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMaxOffsetRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMaxOffsetRequestHeader.java
index 6963195..e961af9 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMaxOffsetRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMaxOffsetRequestHeader.java
@@ -20,51 +20,38 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class GetMaxOffsetRequestHeader implements CommandCustomHeader {
+public class GetMaxOffsetRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String topic;
@CFNotNull
private Integer queueId;
- private boolean committed;
- private boolean logicalQueue;
@Override
public void checkFields() throws RemotingCommandException {
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
- public void setCommitted(boolean committed) {
- this.committed = committed;
- }
-
- public boolean isCommitted() {
- return committed;
- }
-
- public void setLogicalQueue(boolean logicalQueue) {
- this.logicalQueue = logicalQueue;
- }
-
- public boolean getLogicalQueue() {
- return logicalQueue;
- }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMinOffsetRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMinOffsetRequestHeader.java
index 6fb8ed4..70189b7 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMinOffsetRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetMinOffsetRequestHeader.java
@@ -20,11 +20,11 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class GetMinOffsetRequestHeader implements CommandCustomHeader {
+public class GetMinOffsetRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String topic;
@CFNotNull
@@ -34,18 +34,22 @@ public class GetMinOffsetRequestHeader implements CommandCustomHeader {
public void checkFields() throws RemotingCommandException {
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicConfigRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicConfigRequestHeader.java
index ea9d17c..bc7586a 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicConfigRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicConfigRequestHeader.java
@@ -17,11 +17,11 @@
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class GetTopicConfigRequestHeader implements CommandCustomHeader {
+public class GetTopicConfigRequestHeader extends TopicRequestHeader {
@Override
public void checkFields() throws RemotingCommandException {
}
@@ -29,6 +29,7 @@ public class GetTopicConfigRequestHeader implements CommandCustomHeader {
@CFNotNull
private String topic;
+
/**
* @return the topic
*/
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicStatsInfoRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicStatsInfoRequestHeader.java
index c4cf4de..f98e150 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicStatsInfoRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/GetTopicStatsInfoRequestHeader.java
@@ -17,11 +17,11 @@
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class GetTopicStatsInfoRequestHeader implements CommandCustomHeader {
+public class GetTopicStatsInfoRequestHeader extends TopicRequestHeader {
@CFNotNull
private String topic;
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
index 106e89e..e15170f 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
@@ -20,12 +20,12 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class PullMessageRequestHeader implements CommandCustomHeader {
+public class PullMessageRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String consumerGroup;
@CFNotNull
@@ -60,18 +60,22 @@ public class PullMessageRequestHeader implements CommandCustomHeader {
this.consumerGroup = consumerGroup;
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageResponseHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageResponseHeader.java
index 0112f7d..88af984 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageResponseHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageResponseHeader.java
@@ -22,6 +22,7 @@ package org.apache.rocketmq.common.protocol.header;
import org.apache.rocketmq.remoting.CommandCustomHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
+import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
public class PullMessageResponseHeader implements CommandCustomHeader {
@@ -33,6 +34,8 @@ public class PullMessageResponseHeader implements CommandCustomHeader {
private Long minOffset;
@CFNotNull
private Long maxOffset;
+ @CFNullable
+ private Long offsetDelta;
@Override
public void checkFields() throws RemotingCommandException {
@@ -69,4 +72,12 @@ public class PullMessageResponseHeader implements CommandCustomHeader {
public void setSuggestWhichBrokerId(Long suggestWhichBrokerId) {
this.suggestWhichBrokerId = suggestWhichBrokerId;
}
+
+ public Long getOffsetDelta() {
+ return offsetDelta;
+ }
+
+ public void setOffsetDelta(Long offsetDelta) {
+ this.offsetDelta = offsetDelta;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/QueryConsumerOffsetRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/QueryConsumerOffsetRequestHeader.java
index 3b7f627..ebcbe0d 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/QueryConsumerOffsetRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/QueryConsumerOffsetRequestHeader.java
@@ -20,11 +20,11 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class QueryConsumerOffsetRequestHeader implements CommandCustomHeader {
+public class QueryConsumerOffsetRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String consumerGroup;
@CFNotNull
@@ -32,6 +32,8 @@ public class QueryConsumerOffsetRequestHeader implements CommandCustomHeader {
@CFNotNull
private Integer queueId;
+ private Boolean setZeroIfNotFound;
+
@Override
public void checkFields() throws RemotingCommandException {
}
@@ -44,19 +46,31 @@ public class QueryConsumerOffsetRequestHeader implements CommandCustomHeader {
this.consumerGroup = consumerGroup;
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
+
+ public Boolean getSetZeroIfNotFound() {
+ return setZeroIfNotFound;
+ }
+
+ public void setSetZeroIfNotFound(Boolean setZeroIfNotFound) {
+ this.setZeroIfNotFound = setZeroIfNotFound;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SearchOffsetRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SearchOffsetRequestHeader.java
index 5ea2e24..c8291d2 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SearchOffsetRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SearchOffsetRequestHeader.java
@@ -20,11 +20,11 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class SearchOffsetRequestHeader implements CommandCustomHeader {
+public class SearchOffsetRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String topic;
@CFNotNull
@@ -37,18 +37,22 @@ public class SearchOffsetRequestHeader implements CommandCustomHeader {
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageRequestHeader.java
index 2df31e6..808bc2d 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageRequestHeader.java
@@ -20,12 +20,12 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class SendMessageRequestHeader implements CommandCustomHeader {
+public class SendMessageRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String producerGroup;
@CFNotNull
@@ -64,10 +64,12 @@ public class SendMessageRequestHeader implements CommandCustomHeader {
this.producerGroup = producerGroup;
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
@@ -88,10 +90,12 @@ public class SendMessageRequestHeader implements CommandCustomHeader {
this.defaultTopicQueueNums = defaultTopicQueueNums;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/UpdateConsumerOffsetRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/UpdateConsumerOffsetRequestHeader.java
index 3f44db6..11eccd5 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/UpdateConsumerOffsetRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/UpdateConsumerOffsetRequestHeader.java
@@ -20,11 +20,11 @@
*/
package org.apache.rocketmq.common.protocol.header;
-import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-public class UpdateConsumerOffsetRequestHeader implements CommandCustomHeader {
+public class UpdateConsumerOffsetRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private String consumerGroup;
@CFNotNull
@@ -46,18 +46,22 @@ public class UpdateConsumerOffsetRequestHeader implements CommandCustomHeader {
this.consumerGroup = consumerGroup;
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public void setTopic(String topic) {
this.topic = topic;
}
+ @Override
public Integer getQueueId() {
return queueId;
}
+ @Override
public void setQueueId(Integer queueId) {
this.queueId = queueId;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/GetRouteInfoRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/GetRouteInfoRequestHeader.java
index ad776c8..a2806e6 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/GetRouteInfoRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/GetRouteInfoRequestHeader.java
@@ -20,7 +20,6 @@
*/
package org.apache.rocketmq.common.protocol.header.namesrv;
-import java.util.Set;
import org.apache.rocketmq.remoting.CommandCustomHeader;
import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
@@ -29,9 +28,6 @@ public class GetRouteInfoRequestHeader implements CommandCustomHeader {
@CFNotNull
private String topic;
- private int sysFlag;
- private Set<Integer> logicalQueueIdsFilter;
-
@Override
public void checkFields() throws RemotingCommandException {
}
@@ -43,20 +39,4 @@ public class GetRouteInfoRequestHeader implements CommandCustomHeader {
public void setTopic(String topic) {
this.topic = topic;
}
-
- public int getSysFlag() {
- return sysFlag;
- }
-
- public void setSysFlag(int sysFlag) {
- this.sysFlag = sysFlag;
- }
-
- public void setLogicalQueueIdsFilter(Set<Integer> filter) {
- this.logicalQueueIdsFilter = filter;
- }
-
- public Set<Integer> getLogicalQueueIdsFilter() {
- return logicalQueueIdsFilter;
- }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueueRouteData.java b/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueueRouteData.java
deleted file mode 100644
index aed4d5d..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueueRouteData.java
+++ /dev/null
@@ -1,309 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.route;
-
-import com.alibaba.fastjson.annotation.JSONField;
-import com.google.common.base.Objects;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Lists;
-import java.util.List;
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageQueue;
-
-/**
- * logical queue offset -> message queue offset mapping
- */
-public class LogicalQueueRouteData implements Comparable<LogicalQueueRouteData> {
- private volatile int logicalQueueIndex = -1; /* -1 means not set */
- private volatile long logicalQueueDelta = -1; /* inclusive, -1 means not set, occurred in writeOnly state */
-
- private MessageQueue messageQueue;
-
- private volatile MessageQueueRouteState state = MessageQueueRouteState.Normal;
-
- private volatile long offsetDelta = 0; // valid when Normal/WriteOnly/ReadOnly
- private volatile long offsetMax = -1; // exclusive, valid when ReadOnly
-
- private volatile long firstMsgTimeMillis = -1; // valid when ReadOnly
- private volatile long lastMsgTimeMillis = -1; // valid when ReadOnly
-
- private String brokerAddr; /* not always set, only used by high availability forward */
-
- public LogicalQueueRouteData() {
- }
-
- public LogicalQueueRouteData(int logicalQueueIndex, long logicalQueueDelta,
- MessageQueue messageQueue, MessageQueueRouteState state, long offsetDelta, long offsetMax,
- long firstMsgTimeMillis,
- long lastMsgTimeMillis, String brokerAddr) {
- this.logicalQueueIndex = logicalQueueIndex;
- this.logicalQueueDelta = logicalQueueDelta;
- this.messageQueue = messageQueue;
- this.state = state;
- this.offsetDelta = offsetDelta;
- this.offsetMax = offsetMax;
- this.firstMsgTimeMillis = firstMsgTimeMillis;
- this.lastMsgTimeMillis = lastMsgTimeMillis;
- this.brokerAddr = brokerAddr;
- }
-
- public LogicalQueueRouteData(LogicalQueueRouteData queueRouteData) {
- copyFrom(queueRouteData);
- }
-
- public int getLogicalQueueIndex() {
- return logicalQueueIndex;
- }
-
- public void setLogicalQueueIndex(int logicalQueueIndex) {
- this.logicalQueueIndex = logicalQueueIndex;
- }
-
- public long getLogicalQueueDelta() {
- return logicalQueueDelta;
- }
-
- public void setLogicalQueueDelta(long logicalQueueDelta) {
- this.logicalQueueDelta = logicalQueueDelta;
- }
-
- public MessageQueue getMessageQueue() {
- return messageQueue;
- }
-
- public void setMessageQueue(MessageQueue messageQueue) {
- this.messageQueue = messageQueue;
- }
-
- public MessageQueueRouteState getState() {
- return state;
- }
-
- @JSONField(serialize = false)
- public int getStateOrdinal() {
- return state.ordinal();
- }
-
- public void setState(MessageQueueRouteState state) {
- this.state = state;
- }
-
- public String getBrokerAddr() {
- return brokerAddr;
- }
-
- public void setBrokerAddr(String brokerAddr) {
- this.brokerAddr = brokerAddr;
- }
-
- public long getOffsetDelta() {
- return offsetDelta;
- }
-
- public void setOffsetDelta(long offsetDelta) {
- this.offsetDelta = offsetDelta;
- }
-
- public long getOffsetMax() {
- return offsetMax;
- }
-
- public void setOffsetMax(long offsetMax) {
- this.offsetMax = offsetMax;
- }
-
- public long getFirstMsgTimeMillis() {
- return firstMsgTimeMillis;
- }
-
- public void setFirstMsgTimeMillis(long firstMsgTimeMillis) {
- this.firstMsgTimeMillis = firstMsgTimeMillis;
- }
-
- public long getLastMsgTimeMillis() {
- return lastMsgTimeMillis;
- }
-
- public void setLastMsgTimeMillis(long lastMsgTimeMillis) {
- this.lastMsgTimeMillis = lastMsgTimeMillis;
- }
-
- @Override public String toString() {
- return "LogicalQueueRouteData{" +
- "logicalQueueIndex=" + logicalQueueIndex +
- ", logicalQueueDelta=" + logicalQueueDelta +
- ", messageQueue=" + messageQueue +
- ", state=" + state +
- ", offsetDelta=" + offsetDelta +
- ", offsetMax=" + offsetMax +
- ", firstMsgTimeMillis=" + firstMsgTimeMillis +
- ", lastMsgTimeMillis=" + lastMsgTimeMillis +
- ", brokerAddr='" + brokerAddr + '\'' +
- '}';
- }
-
- public void copyFrom(LogicalQueueRouteData queueRouteData) {
- this.logicalQueueIndex = queueRouteData.logicalQueueIndex;
- this.logicalQueueDelta = queueRouteData.logicalQueueDelta;
- this.messageQueue = new MessageQueue(queueRouteData.getMessageQueue());
- this.state = queueRouteData.state;
- this.offsetDelta = queueRouteData.offsetDelta;
- this.offsetMax = queueRouteData.offsetMax;
- this.firstMsgTimeMillis = queueRouteData.firstMsgTimeMillis;
- this.lastMsgTimeMillis = queueRouteData.lastMsgTimeMillis;
- this.brokerAddr = queueRouteData.brokerAddr;
- }
-
- public long toLogicalQueueOffset(long messageQueueOffset) {
- return this.logicalQueueDelta < 0 ? -1 : messageQueueOffset - this.offsetDelta + this.logicalQueueDelta;
- }
-
- public long toMessageQueueOffset(long logicalQueueOffset) {
- return logicalQueueOffset - this.logicalQueueDelta + this.offsetDelta;
- }
-
- @Override public boolean equals(Object o) {
- if (this == o)
- return true;
- if (o == null || getClass() != o.getClass())
- return false;
- LogicalQueueRouteData that = (LogicalQueueRouteData) o;
- return logicalQueueIndex == that.logicalQueueIndex && logicalQueueDelta == that.logicalQueueDelta && offsetDelta == that.offsetDelta && offsetMax == that.offsetMax && firstMsgTimeMillis == that.firstMsgTimeMillis && lastMsgTimeMillis == that.lastMsgTimeMillis && Objects.equal(messageQueue, that.messageQueue) && state == that.state && Objects.equal(brokerAddr, that.brokerAddr);
- }
-
- @Override public int hashCode() {
- return Objects.hashCode(logicalQueueIndex, logicalQueueDelta, messageQueue, state, offsetDelta, offsetMax, firstMsgTimeMillis, lastMsgTimeMillis, brokerAddr);
- }
-
- @JSONField(serialize = false)
- public long getMessagesCount() {
- return this.offsetDelta >= 0 && this.offsetMax >= 0 ? this.offsetMax - this.offsetDelta : 0L;
- }
-
- @JSONField(serialize = false)
- public boolean isWritable() {
- return MessageQueueRouteState.Normal.equals(state) || MessageQueueRouteState.WriteOnly.equals(state);
- }
-
- @JSONField(serialize = false)
- public boolean isReadable() {
- return MessageQueueRouteState.Normal.equals(state) || MessageQueueRouteState.ReadOnly.equals(state);
- }
-
- @JSONField(serialize = false)
- public boolean isExpired() {
- return MessageQueueRouteState.Expired.equals(state);
- }
-
- @JSONField(serialize = false)
- public boolean isWriteOnly() {
- return MessageQueueRouteState.WriteOnly.equals(state);
- }
-
- @JSONField(serialize = false)
- public int getQueueId() {
- return messageQueue.getQueueId();
- }
-
- @JSONField(serialize = false)
- public String getBrokerName() {
- return messageQueue.getBrokerName();
- }
-
- @JSONField(serialize = false)
- public String getTopic() {
- return messageQueue.getTopic();
- }
-
- public boolean isSameTo(LogicalQueueRouteData o) {
- if (o == null) {
- return false;
- }
- return isSameTo(o.getMessageQueue(), o.offsetDelta);
- }
-
- public boolean isSameTo(MessageQueue mq, long offsetDelta) {
- return Objects.equal(this.messageQueue, mq) && this.offsetDelta == offsetDelta;
- }
-
- /**
- * First compare logicalQueueDelta, negative delta must be ordered in the last; then compare state's ordinal; then
- * compare messageQueue, nulls first; then compare offsetDelta.
- */
- @Override
- public int compareTo(LogicalQueueRouteData o) {
- long x = this.getLogicalQueueDelta();
- long y = o.getLogicalQueueDelta();
- int result;
- {
- if (x >= 0 && y >= 0) {
- result = MixAll.compareLong(x, y);
- } else if (x < 0 && y < 0) {
- result = MixAll.compareLong(-x, -y);
- } else if (x < 0) {
- // o1 < 0 && o2 >= 0
- result = 1;
- } else {
- // o1 >= 0 && o2 < 0
- result = -1;
- }
- }
- if (result == 0) {
- result = MixAll.compareInteger(this.state.ordinal(), o.state.ordinal());
- }
- if (result == 0) {
- if (this.messageQueue == null) {
- if (o.messageQueue != null) {
- result = -1;
- }
- } else {
- if (o.messageQueue != null) {
- result = this.messageQueue.compareTo(o.messageQueue);
- } else {
- result = 1;
- }
- }
- }
- if (result == 0) {
- result = MixAll.compareLong(this.offsetDelta, o.offsetDelta);
- }
- return result;
- }
-
- public static final Predicate<LogicalQueueRouteData> READABLE_PREDICT = new Predicate<LogicalQueueRouteData>() {
- @Override
- public boolean apply(LogicalQueueRouteData input) {
- return input != null && input.isReadable();
- }
- };
-
- public List<MessageExt> filterMessages(List<MessageExt> list) {
- if (this.offsetMax < 0 || list == null || list.isEmpty()) {
- return list;
- }
- List<MessageExt> result = Lists.newArrayListWithExpectedSize(list.size());
- for (MessageExt m : list) {
- if (m.getQueueOffset() >= this.offsetMax) {
- break;
- } else {
- result.add(m);
- }
- }
- return result;
- }
-}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueuesInfo.java b/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueuesInfo.java
deleted file mode 100644
index de62b43..0000000
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/route/LogicalQueuesInfo.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.common.protocol.route;
-
-import com.alibaba.fastjson.parser.ParserConfig;
-import com.google.common.base.Objects;
-import com.google.common.collect.Lists;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.locks.Lock;
-import java.util.concurrent.locks.ReadWriteLock;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-import org.apache.rocketmq.common.fastjson.GenericMapSuperclassDeserializer;
-
-public class LogicalQueuesInfo extends HashMap<Integer, List<LogicalQueueRouteData>> {
- // TODO whether here needs more fine-grained locks like per logical queue lock?
- private final ReadWriteLock lock = new ReentrantReadWriteLock();
-
- public LogicalQueuesInfo() {
- super();
- }
-
- public LogicalQueuesInfo(Map<Integer, List<LogicalQueueRouteData>> m) {
- super(m);
- }
-
- public Lock readLock() {
- return lock.readLock();
- }
-
- public Lock writeLock() {
- return lock.writeLock();
- }
-
- public void updateLogicalQueueRouteDataList(int logicalQueueIdx,
- List<LogicalQueueRouteData> logicalQueueRouteDataList) {
- this.writeLock().lock();
- try {
- logicalQueueRouteDataList = Lists.newLinkedList(logicalQueueRouteDataList);
- List<LogicalQueueRouteData> queueRouteDataList = this.get(logicalQueueIdx);
- for (LogicalQueueRouteData logicalQueueRouteData : queueRouteDataList) {
- for (Iterator<LogicalQueueRouteData> it = logicalQueueRouteDataList.iterator(); it.hasNext(); ) {
- LogicalQueueRouteData newQueueRouteData = it.next();
- if (Objects.equal(newQueueRouteData.getMessageQueue(), logicalQueueRouteData.getMessageQueue()) && newQueueRouteData.getOffsetDelta() == logicalQueueRouteData.getOffsetDelta()) {
- logicalQueueRouteData.copyFrom(newQueueRouteData);
... 9342 lines suppressed ...
[rocketmq] 06/17: [acl] Modify unit test to solve the problem of lack of license
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 8bb98c9f41097960bef1467cc09acfb1d54326fb
Author: zhangyang21 <zh...@xiaomi.com>
AuthorDate: Thu Nov 18 19:48:59 2021 +0800
[acl] Modify unit test to solve the problem of lack of license
Signed-off-by: zhangyang21 <zh...@xiaomi.com>
---
.../org/apache/rocketmq/acl/common/AclUtils.java | 39 +++++++++++++++++
.../apache/rocketmq/acl/common/AclUtilsTest.java | 21 +++++++++
.../acl/plain/PlainAccessValidatorTest.java | 51 ++++++++++++++--------
3 files changed, 92 insertions(+), 19 deletions(-)
diff --git a/acl/src/main/java/org/apache/rocketmq/acl/common/AclUtils.java b/acl/src/main/java/org/apache/rocketmq/acl/common/AclUtils.java
index b801c69..cf45432 100644
--- a/acl/src/main/java/org/apache/rocketmq/acl/common/AclUtils.java
+++ b/acl/src/main/java/org/apache/rocketmq/acl/common/AclUtils.java
@@ -20,9 +20,11 @@ import com.alibaba.fastjson.JSONObject;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
+import java.nio.channels.FileChannel;
import java.util.Map;
import java.util.SortedMap;
import org.apache.commons.lang3.StringUtils;
@@ -274,6 +276,43 @@ public class AclUtils {
return true;
}
+ public static boolean copyFile(String from, String to) {
+ FileChannel input = null;
+ FileChannel output = null;
+ try {
+ input = new FileInputStream(new File(from)).getChannel();
+ output = new FileOutputStream(new File(to)).getChannel();
+ output.transferFrom(input, 0, input.size());
+ return true;
+ } catch (Exception e) {
+ log.error("file copy error. from={}, to={}", from, to, e);
+ } finally {
+ closeFileChannel(input);
+ closeFileChannel(output);
+ }
+ return false;
+ }
+
+ public static boolean moveFile(String from, String to) {
+ try {
+ File file = new File(from);
+ return file.renameTo(new File(to));
+ } catch (Exception e) {
+ log.error("file move error. from={}, to={}", from, to, e);
+ }
+ return false;
+ }
+
+ private static void closeFileChannel(FileChannel fileChannel) {
+ if (fileChannel != null) {
+ try {
+ fileChannel.close();
+ } catch (IOException e) {
+ log.error("Close file channel error.", e);
+ }
+ }
+ }
+
public static RPCHook getAclRPCHook(String fileName) {
JSONObject yamlDataObject = null;
try {
diff --git a/acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java b/acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java
index e2a212a..bfbbb42 100644
--- a/acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java
+++ b/acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java
@@ -316,4 +316,25 @@ public class AclUtilsTest {
Assert.assertNull(incompleteContRPCHook);
}
+ @Test
+ public void testCopyAndMoveFile() {
+ String path = "src/test/resources/conf/plain_acl.yml";
+ String backupPath = "src/test/resources/conf/plain_acl.yml_backup";
+ Map<String, Object> aclYamlData = AclUtils.getYamlDataObject(path, Map.class);
+
+ AclUtils.copyFile(path, backupPath);
+ Assert.assertEquals(aclYamlData, AclUtils.getYamlDataObject(backupPath, Map.class));
+
+ Map<String, Object> aclYamlMap = new HashMap<String, Object>();
+ List<String> globalWhiteRemoteAddrs = new ArrayList<String>();
+ globalWhiteRemoteAddrs.add("10.10.103.*");
+ globalWhiteRemoteAddrs.add("192.168.0.*");
+ aclYamlMap.put("globalWhiteRemoteAddrs", globalWhiteRemoteAddrs);
+ AclUtils.writeDataObject(path, aclYamlMap);
+ Assert.assertNotEquals(aclYamlData, AclUtils.getYamlDataObject(path, Map.class));
+
+ AclUtils.moveFile(backupPath, path);
+ Assert.assertEquals(aclYamlData, AclUtils.getYamlDataObject(path, Map.class));
+ }
+
}
diff --git a/acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java b/acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java
index a0eb567..d19916b 100644
--- a/acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java
+++ b/acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java
@@ -328,7 +328,7 @@ public class PlainAccessValidatorTest {
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_update_create.yml");
String targetFileName = "src/test/resources/conf/plain_acl_update_create.yml";
- Map<String, Object> backUpAclConfigMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
+ AclUtils.copyFile(targetFileName, buildBackupPath(targetFileName));
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
plainAccessConfig.setAccessKey("RocketMQ");
@@ -370,8 +370,8 @@ public class PlainAccessValidatorTest {
List<Map<String, Object>> dataVersions = (List<Map<String, Object>>) readableMap.get("dataVersion");
Assert.assertEquals(1,dataVersions.get(0).get("counter"));
- // Restore the backup file and flush to yaml file
- AclUtils.writeDataObject(targetFileName, backUpAclConfigMap);
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
@Test
@@ -380,7 +380,7 @@ public class PlainAccessValidatorTest {
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_update_create.yml");
String targetFileName = "src/test/resources/conf/plain_acl_update_create.yml";
- Map<String, Object> backUpAclConfigMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
+ AclUtils.copyFile(targetFileName, buildBackupPath(targetFileName));
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
plainAccessConfig.setAccessKey("RocketMQ");
@@ -401,18 +401,17 @@ public class PlainAccessValidatorTest {
}
Assert.assertEquals(verifyMap.get(AclConstants.CONFIG_SECRET_KEY),"123456789111");
- // Restore the backup file and flush to yaml file
- AclUtils.writeDataObject(targetFileName, backUpAclConfigMap);
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
-
@Test
public void createAndUpdateAccessAclYamlConfigNormalTest() {
System.setProperty("rocketmq.home.dir", "src/test/resources");
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_update_create.yml");
String targetFileName = "src/test/resources/conf/plain_acl_update_create.yml";
- Map<String, Object> backUpAclConfigMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
+ AclUtils.copyFile(targetFileName, buildBackupPath(targetFileName));
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
plainAccessConfig.setAccessKey("RocketMQ33");
@@ -478,9 +477,8 @@ public class PlainAccessValidatorTest {
Assert.assertEquals(2,dataVersions2.get(0).get(AclConstants.CONFIG_COUNTER));
Assert.assertEquals(verifyMap2.get(AclConstants.CONFIG_SECRET_KEY),"1234567890123");
-
- // Restore the backup file and flush to yaml file
- AclUtils.writeDataObject(targetFileName, backUpAclConfigMap);
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
@Test(expected = AclException.class)
@@ -503,15 +501,14 @@ public class PlainAccessValidatorTest {
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_delete.yml");
String targetFileName = "src/test/resources/conf/plain_acl_delete.yml";
- Map<String, Object> backUpAclConfigMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
-
+ AclUtils.copyFile(targetFileName, buildBackupPath(targetFileName));
String accessKey = "rocketmq2";
PlainAccessValidator plainAccessValidator = new PlainAccessValidator();
plainAccessValidator.deleteAccessConfig(accessKey);
Map<String, Object> readableMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
- List<Map<String, Object>> accounts = (List<Map<String, Object>>)readableMap.get(AclConstants.CONFIG_ACCOUNTS);
+ List<Map<String, Object>> accounts = (List<Map<String, Object>>) readableMap.get(AclConstants.CONFIG_ACCOUNTS);
Map<String, Object> verifyMap = null;
for (Map<String, Object> account : accounts) {
if (account.get(AclConstants.CONFIG_ACCESS_KEY).equals(accessKey)) {
@@ -526,8 +523,8 @@ public class PlainAccessValidatorTest {
List<Map<String, Object>> dataVersions = (List<Map<String, Object>>) readableMap.get(AclConstants.CONFIG_DATA_VERSION);
Assert.assertEquals(1,dataVersions.get(0).get(AclConstants.CONFIG_COUNTER));
- // Restore the backup file and flush to yaml file
- AclUtils.writeDataObject(targetFileName, backUpAclConfigMap);
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
@Test
@@ -583,7 +580,7 @@ public class PlainAccessValidatorTest {
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_global_white_addrs.yml");
String targetFileName = "src/test/resources/conf/plain_acl_global_white_addrs.yml";
- Map<String, Object> backUpAclConfigMap = AclUtils.getYamlDataObject(targetFileName, Map.class);
+ AclUtils.copyFile(targetFileName, buildBackupPath(targetFileName));
PlainAccessValidator plainAccessValidator = new PlainAccessValidator();
// Update global white remote addr value list in the acl access yaml config file
@@ -605,8 +602,8 @@ public class PlainAccessValidatorTest {
List<Map<String, Object>> dataVersions = (List<Map<String, Object>>) readableMap.get(AclConstants.CONFIG_DATA_VERSION);
Assert.assertEquals(1,dataVersions.get(0).get(AclConstants.CONFIG_COUNTER));
- // Restore the backup file and flush to yaml file
- AclUtils.writeDataObject(targetFileName, backUpAclConfigMap);
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
@Test
@@ -620,6 +617,9 @@ public class PlainAccessValidatorTest {
@Test
public void updateAccessConfigEmptyPermListTest(){
+ String targetFileName = "src/test/resources/conf/plain_acl.yml";
+ AclUtils.copyFile(targetFileName, this.buildBackupPath(targetFileName));
+
PlainAccessValidator plainAccessValidator = new PlainAccessValidator();
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
String accessKey = "updateAccessConfigEmptyPerm";
@@ -636,10 +636,16 @@ public class PlainAccessValidatorTest {
Assert.assertEquals(0, result.getTopicPerms().size());
plainAccessValidator.deleteAccessConfig(accessKey);
+
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
}
@Test
public void updateAccessConfigEmptyWhiteRemoteAddressTest(){
+ String targetFileName = "src/test/resources/conf/plain_acl.yml";
+ AclUtils.copyFile(targetFileName, this.buildBackupPath(targetFileName));
+
PlainAccessValidator plainAccessValidator = new PlainAccessValidator();
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
String accessKey = "updateAccessConfigEmptyWhiteRemoteAddress";
@@ -656,5 +662,12 @@ public class PlainAccessValidatorTest {
Assert.assertEquals("", result.getWhiteRemoteAddress());
plainAccessValidator.deleteAccessConfig(accessKey);
+
+ // Restore the backup file
+ AclUtils.moveFile(buildBackupPath(targetFileName), targetFileName);
+ }
+
+ private String buildBackupPath(String path) {
+ return String.format("%s_backup", path);
}
}
[rocketmq] 13/17: [ISSUE #3708] Refactor CQ and BCQ loading process and Fix some unit-tests issue. (#3713)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 6172ad371384619b0200ea5424d725c4ce9400f8
Author: Hongjian Fei <er...@163.com>
AuthorDate: Thu Jan 6 20:07:21 2022 +0800
[ISSUE #3708] Refactor CQ and BCQ loading process and Fix some unit-tests issue. (#3713)
* [RIP-26] Both CQ and BCQ will be supported in DefaultMessageStore.
* [RIP-26] Both CQ and BCQ will be supported in DefaultMessageStore.
* Part of ISSUE #3708. Refactor CQ and BCQ loading process.
* Mock getDiskSpaceWarningLevelRatio and getDiskSpaceCleanForciblyRatio for ut.
* Optimize batch message unit-test.
* Mock getDiskSpaceWarningLevelRatio and getDiskSpaceCleanForciblyRatio for ut.
* Optimize batch message unit-test.
---
.../rocketmq/store/queue/ConsumeQueueStore.java | 127 +++++++++------------
.../store/DefaultMessageStoreCleanFilesTest.java | 28 ++---
.../store/queue/BatchConsumeMessageTest.java | 70 +++++-------
.../store/queue/ConsumeQueueStoreTest.java | 100 ++++++++++++++++
.../apache/rocketmq/store/queue/QueueTestBase.java | 37 ++++--
5 files changed, 214 insertions(+), 148 deletions(-)
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
index e8146ff..d3bfe75 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
@@ -28,7 +28,6 @@ import org.apache.rocketmq.store.DispatchRequest;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.apache.rocketmq.store.config.StorePathConfigHelper;
import java.io.File;
import java.util.HashMap;
@@ -39,6 +38,10 @@ import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import static java.lang.String.format;
+import static org.apache.rocketmq.store.config.StorePathConfigHelper.getStorePathBatchConsumeQueue;
+import static org.apache.rocketmq.store.config.StorePathConfigHelper.getStorePathConsumeQueue;
+
public class ConsumeQueueStore {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
@@ -79,7 +82,8 @@ public class ConsumeQueueStore {
* Apply the dispatched request and build the consume queue.
* This function should be idempotent.
*
- * @param request
+ * @param consumeQueue consume queue
+ * @param request dispatch request
*/
public void putMessagePositionInfoWrapper(ConsumeQueueInterface consumeQueue, DispatchRequest request) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
@@ -97,11 +101,13 @@ public class ConsumeQueueStore {
}
public boolean load() {
- return loadConsumeQueues() && loadBatchConsumeQueues();
+ boolean cqLoadResult = loadConsumeQueues(getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()), CQType.SimpleCQ);
+ boolean bcqLoadResult = loadConsumeQueues(getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()), CQType.BatchCQ);
+ return cqLoadResult && bcqLoadResult;
}
- private boolean loadBatchConsumeQueues() {
- File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ private boolean loadConsumeQueues(String storePath, CQType cqType) {
+ File dirLogic = new File(storePath);
File[] fileTopicList = dirLogic.listFiles();
if (fileTopicList != null) {
@@ -118,24 +124,9 @@ public class ConsumeQueueStore {
continue;
}
- TopicConfig topicConfig = this.topicConfigTable == null ? null : this.topicConfigTable.get(topic);
-
- // For batch consume queue, the topic config must exist
- if (topicConfig == null) {
- log.warn("topic: {} has no topic config.", topic);
- continue;
- }
+ queueTypeShouldBe(topic, cqType);
- if (!Objects.equals(CQType.BatchCQ, QueueTypeUtils.getCQType(Optional.of(topicConfig)))) {
- log.error("[BUG]topic: {} should be BCQ.", topic);
- }
-
- ConsumeQueueInterface logic = new BatchConsumeQueue(
- topic,
- queueId,
- StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
- this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
- this.messageStore);
+ ConsumeQueueInterface logic = createConsumeQueueByType(cqType, topic, queueId, storePath);
this.putConsumeQueue(topic, queueId, logic);
if (!this.load(logic)) {
return false;
@@ -145,46 +136,39 @@ public class ConsumeQueueStore {
}
}
- log.info("load batch consume queue all over, OK");
+ log.info("load {} all over, OK", cqType);
return true;
}
- private boolean loadConsumeQueues() {
- File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
- File[] fileTopicList = dirLogic.listFiles();
- if (fileTopicList != null) {
-
- for (File fileTopic : fileTopicList) {
- String topic = fileTopic.getName();
-
- File[] fileQueueIdList = fileTopic.listFiles();
- if (fileQueueIdList != null) {
- for (File fileQueueId : fileQueueIdList) {
- int queueId;
- try {
- queueId = Integer.parseInt(fileQueueId.getName());
- } catch (NumberFormatException e) {
- continue;
- }
- ConsumeQueueInterface logic = new ConsumeQueue(
- topic,
- queueId,
- StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
- this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
- this.messageStore);
- this.putConsumeQueue(topic, queueId, logic);
- if (!this.load(logic)) {
- return false;
- }
- }
- }
- }
+ private ConsumeQueueInterface createConsumeQueueByType(CQType cqType, String topic, int queueId, String storePath) {
+ if (Objects.equals(CQType.SimpleCQ, cqType)) {
+ return new ConsumeQueue(
+ topic,
+ queueId,
+ storePath,
+ this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
+ this.messageStore);
+ } else if (Objects.equals(CQType.BatchCQ, cqType)) {
+ return new BatchConsumeQueue(
+ topic,
+ queueId,
+ storePath,
+ this.messageStoreConfig.getMapperFileSizeBatchConsumeQueue(),
+ this.messageStore);
+ } else {
+ throw new RuntimeException(format("queue type %s is not supported.", cqType.toString()));
}
+ }
- log.info("load logics queue all over, OK");
+ private void queueTypeShouldBe(String topic, CQType cqTypeExpected) {
+ TopicConfig topicConfig = this.topicConfigTable == null ? null : this.topicConfigTable.get(topic);
- return true;
+ CQType cqTypeActual = QueueTypeUtils.getCQType(Optional.ofNullable(topicConfig));
+
+ if (!Objects.equals(cqTypeExpected, cqTypeActual)) {
+ throw new RuntimeException(format("The queue type of topic: %s should be %s, but is %s", topic, cqTypeExpected, cqTypeActual));
+ }
}
public void recover(ConsumeQueueInterface consumeQueue) {
@@ -212,13 +196,9 @@ public class ConsumeQueueStore {
}
public void checkSelf() {
- Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- Iterator<Map.Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
- while (itNext.hasNext()) {
- Map.Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
- this.checkSelf(cq.getValue());
+ for (Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> topicEntry : this.consumeQueueTable.entrySet()) {
+ for (Map.Entry<Integer, ConsumeQueueInterface> cqEntry : topicEntry.getValue().entrySet()) {
+ this.checkSelf(cqEntry.getValue());
}
}
}
@@ -292,28 +272,23 @@ public class ConsumeQueueStore {
newLogic = new BatchConsumeQueue(
topic,
queueId,
- StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
this.messageStoreConfig.getMapperFileSizeBatchConsumeQueue(),
this.messageStore);
- ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
- if (oldLogic != null) {
- logic = oldLogic;
- } else {
- logic = newLogic;
- }
} else {
newLogic = new ConsumeQueue(
topic,
queueId,
- StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
this.messageStore);
- ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
- if (oldLogic != null) {
- logic = oldLogic;
- } else {
- logic = newLogic;
- }
+ }
+
+ ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
+ if (oldLogic != null) {
+ logic = oldLogic;
+ } else {
+ logic = newLogic;
}
return logic;
diff --git a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
index 88cf181..9dad5ea 100644
--- a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
@@ -47,6 +47,8 @@ import static org.apache.rocketmq.common.message.MessageDecoder.CHARSET_UTF8;
import static org.apache.rocketmq.store.ConsumeQueue.CQ_STORE_UNIT_SIZE;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
/**
* Test case for DefaultMessageStore.CleanCommitLogService and DefaultMessageStore.CleanConsumeQueueService
@@ -334,26 +336,6 @@ public class DefaultMessageStoreCleanFilesTest {
}
}
- private DefaultMessageStore.CleanCommitLogService getCleanCommitLogService(double diskSpaceCleanForciblyRatio)
- throws Exception {
- Field serviceField = messageStore.getClass().getDeclaredField("cleanCommitLogService");
- serviceField.setAccessible(true);
- DefaultMessageStore.CleanCommitLogService cleanCommitLogService =
- (DefaultMessageStore.CleanCommitLogService) serviceField.get(messageStore);
- serviceField.setAccessible(false);
-
- Field warningLevelRatioField = cleanCommitLogService.getClass().getDeclaredField("diskSpaceWarningLevelRatio");
- warningLevelRatioField.setAccessible(true);
- warningLevelRatioField.set(cleanCommitLogService, String.valueOf(diskSpaceCleanForciblyRatio));
- warningLevelRatioField.setAccessible(false);
-
- Field cleanForciblyRatioField = cleanCommitLogService.getClass().getDeclaredField("diskSpaceCleanForciblyRatio");
- cleanForciblyRatioField.setAccessible(true);
- cleanForciblyRatioField.set(cleanCommitLogService, String.valueOf(diskSpaceCleanForciblyRatio));
- cleanForciblyRatioField.setAccessible(false);
- return cleanCommitLogService;
- }
-
private DefaultMessageStore.CleanConsumeQueueService getCleanConsumeQueueService()
throws Exception {
Field serviceField = messageStore.getClass().getDeclaredField("cleanConsumeQueueService");
@@ -490,11 +472,15 @@ public class DefaultMessageStoreCleanFilesTest {
messageStore = new DefaultMessageStore(messageStoreConfig,
new BrokerStatsManager("test"), new MyMessageArrivingListener(), new BrokerConfig());
- cleanCommitLogService = getCleanCommitLogService(diskSpaceCleanForciblyRatio);
cleanConsumeQueueService = getCleanConsumeQueueService();
assertTrue(messageStore.load());
messageStore.start();
+
+ // partially mock a real obj
+ cleanCommitLogService = spy(cleanCommitLogService);
+ when(cleanCommitLogService.getDiskSpaceWarningLevelRatio()).thenReturn(diskSpaceCleanForciblyRatio);
+ when(cleanCommitLogService.getDiskSpaceCleanForciblyRatio()).thenReturn(diskSpaceCleanForciblyRatio);
}
private class MyMessageArrivingListener implements MessageArrivingListener {
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
index 500cd81..bc5f896 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
@@ -17,8 +17,6 @@
package org.apache.rocketmq.store.queue;
-import org.apache.rocketmq.common.TopicAttributes;
-import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.message.MessageAccessor;
@@ -27,7 +25,6 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.utils.QueueTypeUtils;
-import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.GetMessageResult;
import org.apache.rocketmq.store.GetMessageStatus;
import org.apache.rocketmq.store.MessageExtBrokerInner;
@@ -43,16 +40,11 @@ import org.junit.Test;
import java.io.File;
import java.util.ArrayDeque;
import java.util.ArrayList;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
import java.util.Objects;
import java.util.Queue;
import java.util.Random;
import java.util.UUID;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.awaitility.Awaitility.await;
@@ -173,8 +165,9 @@ public class BatchConsumeMessageTest extends QueueTestBase {
long pullOffset = 0L;
int getMessageCount = 0;
+ int atMostMsgNum = 1;
while (true) {
- GetMessageResult getMessageResult = messageStore.getMessage("group", topic, 0, pullOffset, 1, null);
+ GetMessageResult getMessageResult = messageStore.getMessage("group", topic, 0, pullOffset, atMostMsgNum, null);
if (Objects.equals(getMessageResult.getStatus(), GetMessageStatus.OFFSET_OVERFLOW_ONE)) {
break;
}
@@ -229,8 +222,9 @@ public class BatchConsumeMessageTest extends QueueTestBase {
String topic = "TestDispatchBuildConsumeQueue";
createTopic(topic, CQType.SimpleCQ, messageStore);
- long timeStart = System.currentTimeMillis();
+ long timeStart = -1;
long timeMid = -1;
+ long commitLogMid = -1;
for (int i = 0; i < 100; i++) {
MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, -1);
@@ -238,8 +232,14 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
Thread.sleep(2);
- if (i == 49)
- timeMid = System.currentTimeMillis();
+ if (i == 0) {
+ timeStart = putMessageResult.getAppendMessageResult().getStoreTimestamp();
+ }
+ if (i == 50) {
+ timeMid = putMessageResult.getAppendMessageResult().getStoreTimestamp();
+ commitLogMid = putMessageResult.getAppendMessageResult().getWroteOffset();
+ }
+
}
await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
@@ -261,16 +261,14 @@ public class BatchConsumeMessageTest extends QueueTestBase {
}
//check the message time
- long latencyAllowed = 20;
- long earlistMessageTime = messageStore.getEarliestMessageTime(topic, 0);
- Assert.assertTrue(earlistMessageTime > timeStart - latencyAllowed);
- Assert.assertTrue(earlistMessageTime < timeStart + latencyAllowed);
+ long earliestMessageTime = messageStore.getEarliestMessageTime(topic, 0);
+ Assert.assertEquals(timeStart, earliestMessageTime);
long messageStoreTime = messageStore.getMessageStoreTimeStamp(topic, 0, 50);
- Assert.assertTrue(messageStoreTime > timeMid - latencyAllowed);
- Assert.assertTrue(messageStoreTime < timeMid + latencyAllowed);
+ Assert.assertEquals(timeMid, messageStoreTime);
long commitLogOffset = messageStore.getCommitLogOffsetInQueue(topic, 0, 50);
Assert.assertTrue(commitLogOffset >= messageStore.getMinPhyOffset());
Assert.assertTrue(commitLogOffset <= messageStore.getMaxPhyOffset());
+ Assert.assertEquals(commitLogMid, commitLogOffset);
Assert.assertFalse(messageStore.checkInDiskByConsumeOffset(topic, 0, 50));
}
@@ -279,7 +277,7 @@ public class BatchConsumeMessageTest extends QueueTestBase {
public void testDispatchBuildBatchConsumeQueue() throws Exception {
String topic = "testDispatchBuildBatchConsumeQueue";
int batchNum = 10;
- long timeStart = System.currentTimeMillis();
+ long timeStart = -1;
long timeMid = -1;
createTopic(topic, CQType.BatchCQ, messageStore);
@@ -288,8 +286,13 @@ public class BatchConsumeMessageTest extends QueueTestBase {
PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(topic, batchNum));
Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
Thread.sleep(2);
- if (i == 29)
- timeMid = System.currentTimeMillis();
+ if (i == 0) {
+ timeStart = putMessageResult.getAppendMessageResult().getStoreTimestamp();
+ }
+ if (i == 30) {
+ timeMid = putMessageResult.getAppendMessageResult().getStoreTimestamp();;
+ }
+
}
await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
@@ -309,12 +312,10 @@ public class BatchConsumeMessageTest extends QueueTestBase {
}
//check the message time
- long earlistMessageTime = messageStore.getEarliestMessageTime(topic, 0);
- Assert.assertTrue(earlistMessageTime > timeStart - 20);
- Assert.assertTrue(earlistMessageTime < timeStart + 20);
+ long earliestMessageTime = messageStore.getEarliestMessageTime(topic, 0);
+ Assert.assertEquals(earliestMessageTime, timeStart);
long messageStoreTime = messageStore.getMessageStoreTimeStamp(topic, 0, 300);
- Assert.assertTrue(messageStoreTime > timeMid - 20);
- Assert.assertTrue(messageStoreTime < timeMid + 20);
+ Assert.assertEquals(messageStoreTime, timeMid);
long commitLogOffset = messageStore.getCommitLogOffsetInQueue(topic, 0, 300);
Assert.assertTrue(commitLogOffset >= messageStore.getMinPhyOffset());
Assert.assertTrue(commitLogOffset <= messageStore.getMaxPhyOffset());
@@ -450,21 +451,4 @@ public class BatchConsumeMessageTest extends QueueTestBase {
}
}
- private void createTopic(String topic, CQType cqType, MessageStore messageStore) {
- ConcurrentMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<>();
- TopicConfig topicConfigToBeAdded = new TopicConfig();
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName(), cqType.toString());
- topicConfigToBeAdded.setTopicName(topic);
- topicConfigToBeAdded.setAttributes(attributes);
-
- topicConfigTable.put(topic, topicConfigToBeAdded);
- ((DefaultMessageStore)messageStore).setTopicConfigTable(topicConfigTable);
- }
-
- private Callable<Boolean> fullyDispatched(MessageStore messageStore) {
- return () -> messageStore.dispatchBehindBytes() == 0;
- }
-
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueStoreTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueStoreTest.java
new file mode 100644
index 0000000..a8379fc
--- /dev/null
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueStoreTest.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.store.queue;
+
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.attribute.CQType;
+import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.MessageStore;
+import org.apache.rocketmq.store.PutMessageResult;
+import org.apache.rocketmq.store.PutMessageStatus;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.UUID;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.awaitility.Awaitility.await;
+import static org.junit.Assert.assertEquals;
+
+public class ConsumeQueueStoreTest extends QueueTestBase {
+ private MessageStore messageStore;
+
+ @Before
+ public void init() throws Exception {
+ messageStore = createMessageStore(null, true);
+ messageStore.load();
+ messageStore.start();
+ }
+
+ @After
+ public void destroy() {
+ messageStore.shutdown();
+ messageStore.destroy();
+
+ File file = new File(messageStore.getMessageStoreConfig().getStorePathRootDir());
+ UtilAll.deleteFile(file);
+ }
+
+ @Test
+ public void testLoadConsumeQueuesWithWrongAttribute() {
+ String normalTopic = UUID.randomUUID().toString();
+ createTopic(normalTopic, CQType.SimpleCQ, messageStore);
+
+ for (int i = 0; i < 10; i++) {
+ PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(normalTopic, -1));
+ assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+ }
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
+ // simulate delete topic but with files left.
+ ((DefaultMessageStore)messageStore).setTopicConfigTable(null);
+
+ createTopic(normalTopic, CQType.BatchCQ, messageStore);
+ messageStore.shutdown();
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> messageStore.getQueueStore().load());
+ Assert.assertTrue(runtimeException.getMessage().endsWith("should be SimpleCQ, but is BatchCQ"));
+ }
+
+ @Test
+ public void testLoadBatchConsumeQueuesWithWrongAttribute() {
+ String batchTopic = UUID.randomUUID().toString();
+ createTopic(batchTopic, CQType.BatchCQ, messageStore);
+
+ for (int i = 0; i < 10; i++) {
+ PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(batchTopic, 10));
+ assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+ }
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
+ // simulate delete topic but with files left.
+ ((DefaultMessageStore)messageStore).setTopicConfigTable(null);
+
+ createTopic(batchTopic, CQType.SimpleCQ, messageStore);
+ messageStore.shutdown();
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> messageStore.getQueueStore().load());
+ Assert.assertTrue(runtimeException.getMessage().endsWith("should be BatchCQ, but is SimpleCQ"));
+ }
+
+}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
index c16d7cb..506cbd6 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
@@ -17,6 +17,9 @@
package org.apache.rocketmq.store.queue;
import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
@@ -31,10 +34,31 @@ import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
import java.io.File;
+import java.util.HashMap;
import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
public class QueueTestBase extends StoreTestBase {
+ protected void createTopic(String topic, CQType cqType, MessageStore messageStore) {
+ ConcurrentMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<>();
+ TopicConfig topicConfigToBeAdded = new TopicConfig();
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName(), cqType.toString());
+ topicConfigToBeAdded.setTopicName(topic);
+ topicConfigToBeAdded.setAttributes(attributes);
+
+ topicConfigTable.put(topic, topicConfigToBeAdded);
+ ((DefaultMessageStore)messageStore).setTopicConfigTable(topicConfigTable);
+ }
+
+ protected Callable<Boolean> fullyDispatched(MessageStore messageStore) {
+ return () -> messageStore.dispatchBehindBytes() == 0;
+ }
+
protected MessageStore createMessageStore(String baseDir, boolean extent) throws Exception {
if (baseDir == null) {
baseDir = createBaseDir();
@@ -58,14 +82,11 @@ public class QueueTestBase extends StoreTestBase {
messageStoreConfig.setFlushIntervalCommitLog(1);
messageStoreConfig.setFlushCommitLogThoroughInterval(2);
- DefaultMessageStore messageStore = new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MessageArrivingListener() {
- @Override
- public void arriving(String topic, int queueId, long logicOffset, long tagsCode, long msgStoreTime,
- byte[] filterBitMap, Map<String, String> properties) {
-
- }
- }, new BrokerConfig());
- return messageStore;
+ return new DefaultMessageStore(
+ messageStoreConfig,
+ new BrokerStatsManager("simpleTest"),
+ (topic, queueId, logicOffset, tagsCode, msgStoreTime, filterBitMap, properties) -> {},
+ new BrokerConfig());
}
public MessageExtBrokerInner buildMessage(String topic, int batchNum) {
[rocketmq] 14/17: [ISSUE #3708] add CorrectLogicOffsetService to periodically correct min logic offset (#3722)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 78a3ed7b0ef1a2d997dd58168f2a082423fe6301
Author: Hongjian Fei <er...@163.com>
AuthorDate: Mon Jan 10 18:44:12 2022 +0800
[ISSUE #3708] add CorrectLogicOffsetService to periodically correct min logic offset (#3722)
* [ISSUE #3708] add CorrectLogicOffsetService to periodically correct min logic offset; refactor QueueOffsetAssigner.
* Mock getDiskSpaceWarningLevelRatio and getDiskSpaceCleanForciblyRatio to get around configuration protection in unit-test.
* Fix check style.
---
.../apache/rocketmq/common/attribute/CQType.java | 3 +-
.../org/apache/rocketmq/store/ConsumeQueue.java | 9 +-
.../apache/rocketmq/store/DefaultMessageStore.java | 124 +++++++++++++++++++++
.../rocketmq/store/queue/BatchConsumeQueue.java | 9 +-
.../rocketmq/store/queue/ConsumeQueueStore.java | 2 +-
.../rocketmq/store/queue/QueueOffsetAssigner.java | 30 +++--
.../store/DefaultMessageStoreCleanFilesTest.java | 21 ++++
7 files changed, 173 insertions(+), 25 deletions(-)
diff --git a/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java b/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
index 6bd6ad2..73ef218 100644
--- a/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
@@ -19,6 +19,5 @@ package org.apache.rocketmq.common.attribute;
public enum CQType {
SimpleCQ,
- BatchCQ,
- MillionCQ;
+ BatchCQ
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
index 7763a0f..a1fc870 100644
--- a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
@@ -18,7 +18,6 @@ package org.apache.rocketmq.store;
import java.io.File;
import java.nio.ByteBuffer;
-import java.util.HashMap;
import java.util.List;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
@@ -443,12 +442,8 @@ public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
@Override
public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageExtBrokerInner msg, short messageNum) {
String topicQueueKey = getTopic() + "-" + getQueueId();
- HashMap<String, Long> topicQueueTable = queueOffsetAssigner.getTopicQueueTable();
-
- long topicOffset = topicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
- topicQueueTable.put(topicQueueKey, topicOffset + messageNum);
-
- msg.setQueueOffset(topicOffset);
+ long queueOffset = queueOffsetAssigner.assignQueueOffset(topicQueueKey, messageNum);
+ msg.setQueueOffset(queueOffset);
}
private boolean putMessagePositionInfo(final long offset, final int size, final long tagsCode,
diff --git a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
index da47be6..76165d9 100644
--- a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
@@ -32,6 +32,7 @@ import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
@@ -49,6 +50,7 @@ import org.apache.rocketmq.common.SystemClock;
import org.apache.rocketmq.common.ThreadFactoryImpl;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
@@ -94,6 +96,8 @@ public class DefaultMessageStore implements MessageStore {
private final CleanConsumeQueueService cleanConsumeQueueService;
+ private final CorrectLogicOffsetService correctLogicOffsetService;
+
private final IndexService indexService;
private final AllocateMappedFileService allocateMappedFileService;
@@ -156,6 +160,7 @@ public class DefaultMessageStore implements MessageStore {
this.flushConsumeQueueService = new FlushConsumeQueueService();
this.cleanCommitLogService = new CleanCommitLogService();
this.cleanConsumeQueueService = new CleanConsumeQueueService();
+ this.correctLogicOffsetService = new CorrectLogicOffsetService();
this.storeStatsService = new StoreStatsService();
this.indexService = new IndexService(this);
if (!messageStoreConfig.isEnableDLegerCommitLog()) {
@@ -1351,6 +1356,8 @@ public class DefaultMessageStore implements MessageStore {
long deleteCount = 0L;
deleteCount += this.cleanCommitLogService.run();
deleteCount += this.cleanConsumeQueueService.run();
+
+ this.correctLogicOffsetService.run();
return deleteCount;
}
@@ -1879,6 +1886,123 @@ public class DefaultMessageStore implements MessageStore {
}
}
+ class CorrectLogicOffsetService {
+ private long lastForceCorrectTime = -1L;
+
+ public void run() {
+ try {
+ this.correctLogicMinOffset();
+ } catch (Throwable e) {
+ log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ private boolean needCorrect(ConsumeQueueInterface logic, long minPhyOffset, long lastForeCorrectTimeCurRun) {
+ if (logic == null) {
+ return false;
+ }
+ // If first exist and not available, it means first file may destroy failed, delete it.
+ if (DefaultMessageStore.this.consumeQueueStore.isFirstFileExist(logic) && !DefaultMessageStore.this.consumeQueueStore.isFirstFileAvailable(logic)) {
+ log.error("CorrectLogicOffsetService.needCorrect. first file not available, trigger correct." +
+ " topic:{}, queue:{}, maxPhyOffset in queue:{}, minPhyOffset " +
+ "in commit log:{}, minOffset in queue:{}, maxOffset in queue:{}, cqType:{}"
+ , logic.getTopic(), logic.getQueueId(), logic.getMaxPhysicOffset()
+ , minPhyOffset, logic.getMinOffsetInQueue(), logic.getMaxOffsetInQueue(), logic.getCQType());
+ return true;
+ }
+
+ // logic.getMaxPhysicOffset() or minPhyOffset = -1
+ // means there is no message in current queue, so no need to correct.
+ if (logic.getMaxPhysicOffset() == -1 || minPhyOffset == -1) {
+ return false;
+ }
+
+ if (logic.getMaxPhysicOffset() < minPhyOffset) {
+ if (logic.getMinOffsetInQueue() < logic.getMaxOffsetInQueue()) {
+ log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is less than min phy offset: {}, " +
+ "but min offset: {} is less than max offset: {}. topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ } else if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
+ return false;
+ } else {
+ log.error("CorrectLogicOffsetService.needCorrect. It should not happen, logic max phy offset: {} is less than min phy offset: {}," +
+ " but min offset: {} is larger than max offset: {}. topic:{}, queue:{}, cqType:{}"
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return false;
+ }
+ }
+ //the logic.getMaxPhysicOffset() >= minPhyOffset
+ int forceCorrectInterval = DefaultMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetForceInterval();
+ if ((System.currentTimeMillis() - lastForeCorrectTimeCurRun) > forceCorrectInterval) {
+ lastForceCorrectTime = System.currentTimeMillis();
+ CqUnit cqUnit = logic.getEarliestUnit();
+ if (cqUnit == null) {
+ if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
+ return false;
+ } else {
+ log.error("CorrectLogicOffsetService.needCorrect. cqUnit is null, logic max phy offset: {} is greater than min phy offset: {}, " +
+ "but min offset: {} is not equal to max offset: {}. topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ }
+ }
+
+ if (cqUnit.getPos() < minPhyOffset) {
+ log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is greater than min phy offset: {}, " +
+ "but minPhyPos in cq is: {}. min offset in queue: {}, max offset in queue: {}, topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, cqUnit.getPos(), logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ }
+
+ if (cqUnit.getPos() >= minPhyOffset) {
+
+ // Normal case, do not need correct.
+ return false;
+ }
+ }
+
+ return false;
+ }
+
+ private void correctLogicMinOffset() {
+
+ long lastForeCorrectTimeCurRun = lastForceCorrectTime;
+ long minPhyOffset = getMinPhyOffset();
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.getConsumeQueueTable();
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ if (Objects.equals(CQType.SimpleCQ, logic.getCQType())) {
+ // cq is not supported for now.
+ continue;
+ }
+ if (needCorrect(logic, minPhyOffset, lastForeCorrectTimeCurRun)) {
+ doCorrect(logic, minPhyOffset);
+ }
+ }
+ }
+ }
+
+ private void doCorrect(ConsumeQueueInterface logic, long minPhyOffset) {
+ DefaultMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minPhyOffset);
+ int sleepIntervalWhenCorrectMinOffset = DefaultMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetSleepInterval();
+ if (sleepIntervalWhenCorrectMinOffset > 0) {
+ try {
+ Thread.sleep(sleepIntervalWhenCorrectMinOffset);
+ } catch (InterruptedException ignored) {
+ }
+ }
+ }
+
+ public String getServiceName() {
+ return CorrectLogicOffsetService.class.getSimpleName();
+ }
+ }
+
class FlushConsumeQueueService extends ServiceThread {
private static final int RETRY_TIMES_OVER = 3;
private long lastFlushTimestamp = 0;
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
index 648a472..3400120 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
@@ -35,7 +35,6 @@ import org.apache.rocketmq.store.logfile.MappedFile;
import java.io.File;
import java.nio.ByteBuffer;
-import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentSkipListMap;
@@ -482,17 +481,15 @@ public class BatchConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCy
@Override
public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageExtBrokerInner msg, short messageNum) {
- HashMap<String, Long> batchTopicQueueTable = queueOffsetAssigner.getBatchTopicQueueTable();
String topicQueueKey = getTopic() + "-" + getQueueId();
- Long topicOffset = batchTopicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ long queueOffset = queueOffsetAssigner.assignBatchQueueOffset(topicQueueKey, messageNum);
if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
- MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_BASE, String.valueOf(topicOffset));
+ MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_BASE, String.valueOf(queueOffset));
msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
}
- msg.setQueueOffset(topicOffset);
- batchTopicQueueTable.put(topicQueueKey, topicOffset + messageNum);
+ msg.setQueueOffset(queueOffset);
}
boolean putBatchMessagePositionInfo(final long offset, final int size, final long tagsCode, final long storeTime,
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
index d3bfe75..d2d147c 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
@@ -295,7 +295,7 @@ public class ConsumeQueueStore {
}
public Long getMaxOffset(String topic, int queueId) {
- return this.queueOffsetAssigner.getTopicQueueTable().get(topic + "-" + queueId);
+ return this.queueOffsetAssigner.currentQueueOffset(topic + "-" + queueId);
}
public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java b/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java
index 09e18ec..4ca1126 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java
@@ -24,7 +24,7 @@ import org.apache.rocketmq.logging.InternalLoggerFactory;
import java.util.HashMap;
/**
- * QueueOffsetAssigner is a component for assigning queue.
+ * QueueOffsetAssigner is a component for assigning offsets for queues.
*
*/
public class QueueOffsetAssigner {
@@ -33,20 +33,24 @@ public class QueueOffsetAssigner {
private HashMap<String, Long> topicQueueTable = new HashMap<>(1024);
private HashMap<String, Long> batchTopicQueueTable = new HashMap<>(1024);
- public HashMap<String, Long> getTopicQueueTable() {
- return topicQueueTable;
+ public long assignQueueOffset(String topicQueueKey, short messageNum) {
+ long queueOffset = this.topicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ this.topicQueueTable.put(topicQueueKey, queueOffset + messageNum);
+ return queueOffset;
}
- public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
- this.topicQueueTable = topicQueueTable;
+ public long assignBatchQueueOffset(String topicQueueKey, short messageNum) {
+ Long topicOffset = this.batchTopicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ this.batchTopicQueueTable.put(topicQueueKey, topicOffset + messageNum);
+ return topicOffset;
}
- public HashMap<String, Long> getBatchTopicQueueTable() {
- return batchTopicQueueTable;
+ public long currentQueueOffset(String topicQueueKey) {
+ return this.topicQueueTable.get(topicQueueKey);
}
- public void setBatchTopicQueueTable(HashMap<String, Long> batchTopicQueueTable) {
- this.batchTopicQueueTable = batchTopicQueueTable;
+ public long currentBatchQueueOffset(String topicQueueKey) {
+ return this.batchTopicQueueTable.get(topicQueueKey);
}
public synchronized void remove(String topic, Integer queueId) {
@@ -57,4 +61,12 @@ public class QueueOffsetAssigner {
log.info("removeQueueFromTopicQueueTable OK Topic: {} QueueId: {}", topic, queueId);
}
+
+ public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
+ this.topicQueueTable = topicQueueTable;
+ }
+
+ public void setBatchTopicQueueTable(HashMap<String, Long> batchTopicQueueTable) {
+ this.batchTopicQueueTable = batchTopicQueueTable;
+ }
}
\ No newline at end of file
diff --git a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
index 9dad5ea..356e653 100644
--- a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java
@@ -336,6 +336,17 @@ public class DefaultMessageStoreCleanFilesTest {
}
}
+ private DefaultMessageStore.CleanCommitLogService getCleanCommitLogService()
+ throws Exception {
+ Field serviceField = messageStore.getClass().getDeclaredField("cleanCommitLogService");
+ serviceField.setAccessible(true);
+ DefaultMessageStore.CleanCommitLogService cleanCommitLogService =
+ (DefaultMessageStore.CleanCommitLogService) serviceField.get(messageStore);
+ serviceField.setAccessible(false);
+
+ return cleanCommitLogService;
+ }
+
private DefaultMessageStore.CleanConsumeQueueService getCleanConsumeQueueService()
throws Exception {
Field serviceField = messageStore.getClass().getDeclaredField("cleanConsumeQueueService");
@@ -472,6 +483,7 @@ public class DefaultMessageStoreCleanFilesTest {
messageStore = new DefaultMessageStore(messageStoreConfig,
new BrokerStatsManager("test"), new MyMessageArrivingListener(), new BrokerConfig());
+ cleanCommitLogService = getCleanCommitLogService();
cleanConsumeQueueService = getCleanConsumeQueueService();
assertTrue(messageStore.load());
@@ -481,6 +493,15 @@ public class DefaultMessageStoreCleanFilesTest {
cleanCommitLogService = spy(cleanCommitLogService);
when(cleanCommitLogService.getDiskSpaceWarningLevelRatio()).thenReturn(diskSpaceCleanForciblyRatio);
when(cleanCommitLogService.getDiskSpaceCleanForciblyRatio()).thenReturn(diskSpaceCleanForciblyRatio);
+
+ putFiledBackToMessageStore(cleanCommitLogService);
+ }
+
+ private void putFiledBackToMessageStore(DefaultMessageStore.CleanCommitLogService cleanCommitLogService) throws Exception {
+ Field cleanCommitLogServiceField = DefaultMessageStore.class.getDeclaredField("cleanCommitLogService");
+ cleanCommitLogServiceField.setAccessible(true);
+ cleanCommitLogServiceField.set(messageStore, cleanCommitLogService);
+ cleanCommitLogServiceField.setAccessible(false);
}
private class MyMessageArrivingListener implements MessageArrivingListener {
[rocketmq] 02/17: support batch consume-queue.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 821b91f0a25839550fe2fbdd8183ddb389147e78
Author: Erik1288 <er...@gmail.com>
AuthorDate: Thu Dec 23 18:57:43 2021 +0800
support batch consume-queue.
---
.../apache/rocketmq/broker/BrokerController.java | 17 +-
.../broker/plugin/AbstractPluginMessageStore.java | 6 +-
.../broker/processor/AdminBrokerProcessor.java | 27 +-
.../broker/processor/SendMessageProcessor.java | 32 +-
.../broker/pagecache/OneMessageTransferTest.java | 8 +-
.../broker/processor/AdminBrokerProcessorTest.java | 4 +-
.../broker/processor/PopMessageProcessorTest.java | 4 +-
.../client/consumer/DefaultMQPullConsumer.java | 7 +
.../client/consumer/DefaultMQPushConsumer.java | 11 +
.../rocketmq/client/consumer/MQPullConsumer.java | 7 +
.../rocketmq/client/impl/MQClientAPIImpl.java | 3 +-
.../impl/consumer/DefaultMQPullConsumerImpl.java | 34 +
.../impl/consumer/DefaultMQPushConsumerImpl.java | 1 +
.../client/impl/consumer/PullAPIWrapper.java | 58 +
.../client/producer/DefaultMQProducer.java | 1 +
.../java/org/apache/rocketmq/common/UtilAll.java | 13 +
.../rocketmq/common/message/MessageConst.java | 2 +
.../rocketmq/common/message/MessageDecoder.java | 49 +
.../protocol/header/PullMessageRequestHeader.java | 10 +
.../protocol/header/SendMessageResponseHeader.java | 9 +
.../rocketmq/common/sysflag/MessageSysFlag.java | 5 +
.../common/message/MessageDecoderTest.java | 4 +-
.../rocketmq/store/AllocateMappedFileService.java | 12 +-
.../rocketmq/store/AppendMessageCallback.java | 2 -
.../apache/rocketmq/store/AppendMessageResult.java | 12 +
.../java/org/apache/rocketmq/store/CommitLog.java | 730 ++++--
.../org/apache/rocketmq/store/ConsumeQueue.java | 164 +-
.../org/apache/rocketmq/store/ConsumeQueueExt.java | 1 +
.../apache/rocketmq/store/DefaultMessageStore.java | 541 ++--
.../org/apache/rocketmq/store/DispatchRequest.java | 45 +
.../apache/rocketmq/store/FileQueueSnapshot.java | 90 +
.../apache/rocketmq/store/GetMessageResult.java | 23 +-
.../org/apache/rocketmq/store/MappedFileQueue.java | 97 +-
.../apache/rocketmq/store}/MessageExtBatch.java | 17 +-
.../org/apache/rocketmq/store/MessageStore.java | 169 +-
.../rocketmq/store/MultiPathMappedFileQueue.java | 1 +
.../apache/rocketmq/store/PutMessageContext.java | 31 +
.../rocketmq/store/SelectMappedBufferResult.java | 11 +-
.../org/apache/rocketmq/store/StoreCheckpoint.java | 7 +-
.../java/org/apache/rocketmq/store/StoreUtil.java | 48 +
.../apache/rocketmq/store/StreamMessageStore.java | 2573 ++++++++++++++++++++
.../java/org/apache/rocketmq/store/Swappable.java | 33 +-
.../org/apache/rocketmq/store/TopicQueueLock.java | 29 +
.../rocketmq/store/config/MessageStoreConfig.java | 292 ++-
.../store/config/StorePathConfigHelper.java | 3 +
.../rocketmq/store/dledger/DLedgerCommitLog.java | 217 +-
.../org/apache/rocketmq/store/ha/HAService.java | 8 +-
.../org/apache/rocketmq/store/index/IndexFile.java | 5 +-
.../apache/rocketmq/store/index/IndexService.java | 8 +-
.../rocketmq/store/logfile/AbstractMappedFile.java | 28 +-
.../DefaultMappedFile.java} | 202 +-
.../apache/rocketmq/store/logfile/MappedFile.java | 331 +++
.../rocketmq/store/queue/BatchConsumeQueue.java | 952 ++++++++
.../rocketmq/store/queue/BatchOffsetIndex.java | 57 +
.../org/apache/rocketmq/store/queue/CQType.java | 30 +-
.../store/queue/ConsumeQueueInterface.java | 112 +
.../rocketmq/store/queue/ConsumeQueueStore.java | 176 ++
.../org/apache/rocketmq/store/queue/CqUnit.java | 115 +
.../FileQueueLifeCycle.java} | 32 +-
.../rocketmq/store/queue/ReferredIterator.java | 30 +-
.../store/schedule/ScheduleMessageService.java | 65 +-
.../apache/rocketmq/store/stats/BrokerStats.java | 6 +-
.../apache/rocketmq/store/util/PerfCounter.java | 370 +++
.../{StoreUtil.java => util/QueueTypeUtils.java} | 26 +-
.../apache/rocketmq/store/AppendCallbackTest.java | 2 -
.../apache/rocketmq/store/BatchPutMessageTest.java | 1 -
.../apache/rocketmq/store/ConsumeQueueTest.java | 63 +-
.../store/DefaultMessageStoreCleanFilesTest.java | 8 +-
.../store/DefaultMessageStoreShutDownTest.java | 4 +-
.../rocketmq/store/DefaultMessageStoreTest.java | 78 +-
.../apache/rocketmq/store/MappedFileQueueTest.java | 156 +-
.../org/apache/rocketmq/store/MappedFileTest.java | 4 +-
.../store/MultiPathMappedFileQueueTest.java | 1 +
.../org/apache/rocketmq/store/StoreTestBase.java | 14 +-
.../store/dledger/DLedgerCommitlogTest.java | 6 +-
.../store/queue/BatchConsumeMessageTest.java | 306 +++
.../store/queue/BatchConsumeQueueTest.java | 315 +++
.../rocketmq/store/queue/ConsumeQueueTest.java | 85 +
.../apache/rocketmq/store/queue/QueueTestBase.java | 89 +
.../org/apache/rocketmq/test/base/BaseConf.java | 22 +-
.../rocketmq/test/base/IntegrationTestBase.java | 31 +
.../base/dledger/DLedgerProduceAndConsumeIT.java | 2 +
.../test/client/producer/batch/BatchSendIT.java | 162 ++
.../client/producer/oneway/OneWaySendWithMQIT.java | 10 -
.../rocketmq/test/statictopic/StaticTopicIT.java | 2 +-
85 files changed, 8419 insertions(+), 955 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
index 76c8007..6f9563b 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
@@ -96,9 +96,11 @@ import org.apache.rocketmq.srvutil.FileWatchService;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.MessageArrivingListener;
import org.apache.rocketmq.store.MessageStore;
+import org.apache.rocketmq.store.StreamMessageStore;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.dledger.DLedgerCommitLog;
+import org.apache.rocketmq.store.queue.CQType;
import org.apache.rocketmq.store.stats.BrokerStats;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
@@ -296,14 +298,19 @@ public class BrokerController {
if (result) {
try {
- this.messageStore =
- new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener,
- this.brokerConfig);
+ MessageStore messageStore;
+ if (Objects.equals(CQType.BatchCQ.toString(), this.messageStoreConfig.getDefaultCQType())) {
+ messageStore = new StreamMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
+ } else {
+ messageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
+ }
+
+ this.messageStore = messageStore;
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
- ((DLedgerCommitLog) ((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
+ ((DLedgerCommitLog) messageStore.getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
}
- this.brokerStats = new BrokerStats((DefaultMessageStore) this.messageStore);
+ this.brokerStats = new BrokerStats(this.messageStore);
//load plugin
MessageStorePluginContext context = new MessageStorePluginContext(messageStoreConfig, brokerStatsManager, messageArrivingListener, brokerConfig);
this.messageStore = MessageStoreFactory.build(context, this.messageStore);
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/plugin/AbstractPluginMessageStore.java b/broker/src/main/java/org/apache/rocketmq/broker/plugin/AbstractPluginMessageStore.java
index d306d34..44edfe0 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/plugin/AbstractPluginMessageStore.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/plugin/AbstractPluginMessageStore.java
@@ -22,16 +22,16 @@ import java.util.LinkedList;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.store.CommitLogDispatcher;
-import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.GetMessageResult;
+import org.apache.rocketmq.store.MessageExtBatch;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageFilter;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.QueryMessageResult;
import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
public abstract class AbstractPluginMessageStore implements MessageStore {
@@ -261,7 +261,7 @@ public abstract class AbstractPluginMessageStore implements MessageStore {
}
@Override
- public ConsumeQueue getConsumeQueue(String topic, int queueId) {
+ public ConsumeQueueInterface getConsumeQueue(String topic, int queueId) {
return next.getConsumeQueue(topic, queueId);
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index 2891baf..f30953d 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -131,7 +131,6 @@ import org.apache.rocketmq.remoting.protocol.LanguageCode;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
import org.apache.rocketmq.remoting.protocol.RemotingSysResponseCode;
-import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.ConsumeQueueExt;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.MessageExtBrokerInner;
@@ -140,6 +139,9 @@ import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.queue.ReferredIterator;
import java.io.UnsupportedEncodingException;
import java.net.UnknownHostException;
@@ -1799,7 +1801,7 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
RemotingCommand response = RemotingCommand.createResponseCommand(null);
- ConsumeQueue consumeQueue = this.brokerController.getMessageStore().getConsumeQueue(requestHeader.getTopic(),
+ ConsumeQueueInterface consumeQueue = this.brokerController.getMessageStore().getConsumeQueue(requestHeader.getTopic(),
requestHeader.getQueueId());
if (consumeQueue == null) {
response.setCode(ResponseCode.SYSTEM_ERROR);
@@ -1830,26 +1832,31 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
}
}
- SelectMappedBufferResult result = consumeQueue.getIndexBuffer(requestHeader.getIndex());
+ ReferredIterator<CqUnit> result = consumeQueue.iterateFrom(requestHeader.getIndex());
if (result == null) {
response.setRemark(String.format("Index %d of %d@%s is not exist!", requestHeader.getIndex(), requestHeader.getQueueId(), requestHeader.getTopic()));
return response;
}
try {
List<ConsumeQueueData> queues = new ArrayList<>();
- for (int i = 0; i < result.getSize() && i < requestHeader.getCount() * ConsumeQueue.CQ_STORE_UNIT_SIZE; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
+ while (result.hasNext()) {
+ CqUnit cqUnit = result.next();
+ if (cqUnit.getQueueOffset() - requestHeader.getIndex() >= requestHeader.getCount()) {
+ break;
+ }
+
ConsumeQueueData one = new ConsumeQueueData();
- one.setPhysicOffset(result.getByteBuffer().getLong());
- one.setPhysicSize(result.getByteBuffer().getInt());
- one.setTagsCode(result.getByteBuffer().getLong());
+ one.setPhysicOffset(cqUnit.getPos());
+ one.setPhysicSize(cqUnit.getSize());
+ one.setTagsCode(cqUnit.getTagsCode());
- if (!consumeQueue.isExtAddr(one.getTagsCode())) {
+ if (cqUnit.getCqExtUnit() == null && cqUnit.isTagsCodeValid()) {
queues.add(one);
continue;
}
- ConsumeQueueExt.CqExtUnit cqExtUnit = consumeQueue.getExt(one.getTagsCode());
- if (cqExtUnit != null) {
+ if (cqUnit.getCqExtUnit() != null) {
+ ConsumeQueueExt.CqExtUnit cqExtUnit = cqUnit.getCqExtUnit();
one.setExtendDataJson(JSON.toJSONString(cqExtUnit));
if (cqExtUnit.getFilterBitMap() != null) {
one.setBitMap(BitsArray.create(cqExtUnit.getFilterBitMap()).toString());
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
index 17f19cb..3e3173e 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
@@ -17,6 +17,8 @@
package org.apache.rocketmq.broker.processor;
import java.net.SocketAddress;
+import java.nio.ByteBuffer;
+import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
@@ -39,10 +41,11 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.help.FAQUrl;
import org.apache.rocketmq.common.message.MessageAccessor;
+import org.apache.rocketmq.common.message.MessageClientIDSetter;
+import org.apache.rocketmq.common.message.MessageClientIDSetter;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
@@ -57,8 +60,10 @@ import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.RemotingResponseCallback;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+import org.apache.rocketmq.store.MessageExtBatch;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.PutMessageResult;
+import org.apache.rocketmq.store.StoreUtil;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
@@ -677,11 +682,29 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
String clusterName = this.brokerController.getBrokerConfig().getBrokerClusterName();
MessageAccessor.putProperty(messageExtBatch, MessageConst.PROPERTY_CLUSTER, clusterName);
- CompletableFuture<PutMessageResult> putMessageResult = this.brokerController.getMessageStore().asyncPutMessages(messageExtBatch);
- return handlePutMessageResultFuture(putMessageResult, response, request, messageExtBatch, responseHeader, mqtraceContext, ctx, queueIdInt, requestHeader, mappingContext);
- }
+ CompletableFuture<PutMessageResult> putMessageResult;
+
+ if (StoreUtil.isStreamMode(this.brokerController.getMessageStore()) && MessageClientIDSetter.getUniqID(messageExtBatch) != null) {
+ // newly introduced inner-batch message
+ messageExtBatch.setSysFlag(messageExtBatch.getSysFlag() | MessageSysFlag.NEED_UNWRAP_FLAG);
+ messageExtBatch.setSysFlag(messageExtBatch.getSysFlag() | MessageSysFlag.INNER_BATCH_FLAG);
+ messageExtBatch.setInnerBatch(true);
+
+ int innerNum = MessageDecoder.countInnerMsgNum(ByteBuffer.wrap(messageExtBatch.getBody()));
+
+ MessageAccessor.putProperty(messageExtBatch, MessageConst.PROPERTY_INNER_NUM, String.valueOf(innerNum));
+ messageExtBatch.setPropertiesString(MessageDecoder.messageProperties2String(messageExtBatch.getProperties()));
+ // tell the producer that it's an inner-batch message response.
+ responseHeader.setBatchUniqId(MessageClientIDSetter.getUniqID(messageExtBatch));
+ putMessageResult = this.brokerController.getMessageStore().asyncPutMessage(messageExtBatch);
+ } else {
+ // traditional outer-batch message
+ putMessageResult = this.brokerController.getMessageStore().asyncPutMessages(messageExtBatch);
+ }
+ return handlePutMessageResultFuture(putMessageResult, response, request, messageExtBatch, responseHeader, mqtraceContext, ctx, queueIdInt, requestHeader, mappingContext);
+ }
public boolean hasConsumeMessageHook() {
return consumeMessageHookList != null && !this.consumeMessageHookList.isEmpty();
@@ -771,4 +794,5 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
return response;
}
+
}
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/pagecache/OneMessageTransferTest.java b/broker/src/test/java/org/apache/rocketmq/broker/pagecache/OneMessageTransferTest.java
index 2cd4bdc..da70584 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/pagecache/OneMessageTransferTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/pagecache/OneMessageTransferTest.java
@@ -18,8 +18,8 @@
package org.apache.rocketmq.broker.pagecache;
import java.nio.ByteBuffer;
-import org.apache.rocketmq.store.MappedFile;
import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
import org.junit.Assert;
import org.junit.Test;
@@ -29,7 +29,7 @@ public class OneMessageTransferTest {
public void OneMessageTransferTest(){
ByteBuffer byteBuffer = ByteBuffer.allocate(20);
byteBuffer.putInt(20);
- SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new MappedFile());
+ SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new DefaultMappedFile());
OneMessageTransfer manyMessageTransfer = new OneMessageTransfer(byteBuffer,selectMappedBufferResult);
}
@@ -37,7 +37,7 @@ public class OneMessageTransferTest {
public void OneMessageTransferCountTest(){
ByteBuffer byteBuffer = ByteBuffer.allocate(20);
byteBuffer.putInt(20);
- SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new MappedFile());
+ SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new DefaultMappedFile());
OneMessageTransfer manyMessageTransfer = new OneMessageTransfer(byteBuffer,selectMappedBufferResult);
Assert.assertEquals(manyMessageTransfer.count(),40);
}
@@ -46,7 +46,7 @@ public class OneMessageTransferTest {
public void OneMessageTransferPosTest(){
ByteBuffer byteBuffer = ByteBuffer.allocate(20);
byteBuffer.putInt(20);
- SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new MappedFile());
+ SelectMappedBufferResult selectMappedBufferResult = new SelectMappedBufferResult(0,byteBuffer,20,new DefaultMappedFile());
OneMessageTransfer manyMessageTransfer = new OneMessageTransfer(byteBuffer,selectMappedBufferResult);
Assert.assertEquals(manyMessageTransfer.position(),8);
}
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
index 2141f2c..acf05fb 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
@@ -43,13 +43,13 @@ import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.store.AppendMessageResult;
import org.apache.rocketmq.store.AppendMessageStatus;
-import org.apache.rocketmq.store.MappedFile;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -246,7 +246,7 @@ public class AdminBrokerProcessorTest {
}
private SelectMappedBufferResult createSelectMappedBufferResult() {
- SelectMappedBufferResult result = new SelectMappedBufferResult(0, ByteBuffer.allocate(1024), 0, new MappedFile());
+ SelectMappedBufferResult result = new SelectMappedBufferResult(0, ByteBuffer.allocate(1024), 0, new DefaultMappedFile());
return result;
}
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/PopMessageProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/PopMessageProcessorTest.java
index 87d96b2..55ec3e4 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/PopMessageProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/PopMessageProcessorTest.java
@@ -39,11 +39,11 @@ import org.apache.rocketmq.store.AppendMessageStatus;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.GetMessageResult;
import org.apache.rocketmq.store.GetMessageStatus;
-import org.apache.rocketmq.store.MappedFile;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
import org.junit.Before;
import org.junit.Test;
@@ -184,7 +184,7 @@ public class PopMessageProcessorTest {
for (int i = 0; i < msgCnt; i++) {
ByteBuffer bb = ByteBuffer.allocate(64);
bb.putLong(MessageDecoder.MESSAGE_STORE_TIMESTAMP_POSITION, System.currentTimeMillis());
- getMessageResult.addMessage(new SelectMappedBufferResult(200, bb, 64, new MappedFile()));
+ getMessageResult.addMessage(new SelectMappedBufferResult(200, bb, 64, new DefaultMappedFile()));
}
return getMessageResult;
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
index 0876a94..5829f77 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
@@ -349,6 +349,13 @@ public class DefaultMQPullConsumer extends ClientConfig implements MQPullConsume
}
@Override
+ public void pull(MessageQueue mq, String subExpression, long offset, int maxNums, int maxSize, PullCallback pullCallback,
+ long timeout)
+ throws MQClientException, RemotingException, InterruptedException {
+ this.defaultMQPullConsumerImpl.pull(mq, subExpression, offset, maxNums, maxSize, pullCallback, timeout);
+ }
+
+ @Override
public void pull(MessageQueue mq, MessageSelector messageSelector, long offset, int maxNums,
PullCallback pullCallback)
throws MQClientException, RemotingException, InterruptedException {
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
index 8a6340b..fce6b64 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
@@ -231,6 +231,9 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
*/
private int pullBatchSize = 32;
+
+ private int pullBatchSizeInBytes = 256 * 1024;
+
/**
* Whether update subscription relationship when every pull
*/
@@ -942,6 +945,14 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
this.awaitTerminationMillisWhenShutdown = awaitTerminationMillisWhenShutdown;
}
+ public int getPullBatchSizeInBytes() {
+ return pullBatchSizeInBytes;
+ }
+
+ public void setPullBatchSizeInBytes(int pullBatchSizeInBytes) {
+ this.pullBatchSizeInBytes = pullBatchSizeInBytes;
+ }
+
public TraceDispatcher getTraceDispatcher() {
return traceDispatcher;
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/MQPullConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/MQPullConsumer.java
index a8e9628..868ee93 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/MQPullConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/MQPullConsumer.java
@@ -114,6 +114,13 @@ public interface MQPullConsumer extends MQConsumer {
InterruptedException;
/**
+ * Pulling the messages in a async. way
+ */
+ void pull(final MessageQueue mq, final String subExpression, final long offset, final int maxNums, final int maxSize,
+ final PullCallback pullCallback, long timeout) throws MQClientException, RemotingException,
+ InterruptedException;
+
+ /**
* Pulling the messages in a async. way. Support message selection
*/
void pull(final MessageQueue mq, final MessageSelector selector, final long offset, final int maxNums,
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
index 9823e81..20b8f41 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
@@ -727,7 +727,8 @@ public class MQClientAPIImpl {
MessageQueue messageQueue = new MessageQueue(topic, brokerName, responseHeader.getQueueId());
String uniqMsgId = MessageClientIDSetter.getUniqID(msg);
- if (msg instanceof MessageBatch) {
+ if (msg instanceof MessageBatch && responseHeader.getBatchUniqId() == null) {
+ // This means it is not an inner batch
StringBuilder sb = new StringBuilder();
for (Message message : (MessageBatch) msg) {
sb.append(sb.length() == 0 ? "" : ",").append(MessageClientIDSetter.getUniqID(message));
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
index 6d47573..05d3d48 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
@@ -447,6 +447,13 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
this.pullAsyncImpl(mq, subscriptionData, offset, maxNums, pullCallback, false, timeout);
}
+ public void pull(MessageQueue mq, String subExpression, long offset, int maxNums, int maxSize, PullCallback pullCallback,
+ long timeout)
+ throws MQClientException, RemotingException, InterruptedException {
+ SubscriptionData subscriptionData = getSubscriptionData(mq, subExpression);
+ this.pullAsyncImpl(mq, subscriptionData, offset, maxNums, maxSize, pullCallback, false, timeout);
+ }
+
public void pull(MessageQueue mq, MessageSelector messageSelector, long offset, int maxNums,
PullCallback pullCallback)
throws MQClientException, RemotingException, InterruptedException {
@@ -466,6 +473,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
final SubscriptionData subscriptionData,
final long offset,
final int maxNums,
+ final int maxSizeInBytes,
final PullCallback pullCallback,
final boolean block,
final long timeout) throws MQClientException, RemotingException, InterruptedException {
@@ -483,6 +491,11 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
throw new MQClientException("maxNums <= 0", null);
}
+ if (maxSizeInBytes <= 0) {
+ throw new MQClientException("maxSizeInBytes <= 0", null);
+ }
+
+
if (null == pullCallback) {
throw new MQClientException("pullCallback is null", null);
}
@@ -502,6 +515,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
isTagType ? 0L : subscriptionData.getSubVersion(),
offset,
maxNums,
+ maxSizeInBytes,
sysFlag,
0,
this.defaultMQPullConsumer.getBrokerSuspendMaxTimeMillis(),
@@ -526,6 +540,26 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
}
}
+ private void pullAsyncImpl(
+ final MessageQueue mq,
+ final SubscriptionData subscriptionData,
+ final long offset,
+ final int maxNums,
+ final PullCallback pullCallback,
+ final boolean block,
+ final long timeout) throws MQClientException, RemotingException, InterruptedException {
+ pullAsyncImpl(
+ mq,
+ subscriptionData,
+ offset,
+ maxNums,
+ Integer.MAX_VALUE,
+ pullCallback,
+ block,
+ timeout
+ );
+ }
+
public PullResult pullBlockIfNotFound(MessageQueue mq, String subExpression, long offset, int maxNums)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
SubscriptionData subscriptionData = getSubscriptionData(mq, subExpression);
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
index 2b504cc..2fa3830 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
@@ -465,6 +465,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
subscriptionData.getSubVersion(),
pullRequest.getNextOffset(),
this.defaultMQPushConsumer.getPullBatchSize(),
+ this.defaultMQPushConsumer.getPullBatchSizeInBytes(),
sysFlag,
commitOffsetValue,
BROKER_SUSPEND_MAX_TIME_MILLIS,
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
index 30e8439..6ce8e26 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
@@ -40,6 +40,7 @@ import org.apache.rocketmq.common.protocol.header.PopMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
+import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.sysflag.PullSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.exception.RemotingException;
@@ -79,6 +80,31 @@ public class PullAPIWrapper {
ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);
+ boolean needDecodeInnerMessage = false;
+ for (MessageExt messageExt: msgList) {
+ if (MessageSysFlag.check(messageExt.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)
+ && MessageSysFlag.check(messageExt.getSysFlag(), MessageSysFlag.NEED_UNWRAP_FLAG)) {
+ needDecodeInnerMessage = true;
+ break;
+ }
+ }
+ if (needDecodeInnerMessage) {
+ List<MessageExt> innerMsgList = new ArrayList<MessageExt>();
+ try {
+ for (MessageExt messageExt: msgList) {
+ if (MessageSysFlag.check(messageExt.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)
+ && MessageSysFlag.check(messageExt.getSysFlag(), MessageSysFlag.NEED_UNWRAP_FLAG)) {
+ MessageDecoder.decodeMessage(messageExt, innerMsgList);
+ } else {
+ innerMsgList.add(messageExt);
+ }
+ }
+ msgList = innerMsgList;
+ } catch (Throwable t) {
+ log.error("Try to decode the inner batch failed for {}", pullResult.toString(), t);
+ }
+ }
+
List<MessageExt> msgListFilterAgain = msgList;
if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
@@ -154,6 +180,7 @@ public class PullAPIWrapper {
final long subVersion,
final long offset,
final int maxNums,
+ final int maxSizeInBytes,
final int sysFlag,
final long commitOffset,
final long brokerSuspendMaxTimeMillis,
@@ -198,6 +225,7 @@ public class PullAPIWrapper {
requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
requestHeader.setSubscription(subExpression);
requestHeader.setSubVersion(subVersion);
+ requestHeader.setMaxMsgBytes(maxSizeInBytes);
requestHeader.setExpressionType(expressionType);
String brokerAddr = findBrokerResult.getBrokerAddr();
@@ -219,6 +247,36 @@ public class PullAPIWrapper {
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
+ public PullResult pullKernelImpl(
+ MessageQueue mq,
+ final String subExpression,
+ final String expressionType,
+ final long subVersion,
+ long offset,
+ final int maxNums,
+ final int sysFlag,
+ long commitOffset,
+ final long brokerSuspendMaxTimeMillis,
+ final long timeoutMillis,
+ final CommunicationMode communicationMode,
+ PullCallback pullCallback
+ ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
+ return pullKernelImpl(
+ mq,
+ subExpression,
+ expressionType,
+ subVersion, offset,
+ maxNums,
+ Integer.MAX_VALUE,
+ sysFlag,
+ commitOffset,
+ brokerSuspendMaxTimeMillis,
+ timeoutMillis,
+ communicationMode,
+ pullCallback
+ );
+ }
+
public long recalculatePullFromWhichNode(final MessageQueue mq) {
if (this.isConnectBrokerByUser()) {
return this.defaultBrokerId;
diff --git a/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java b/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
index 1af416b..230785c 100644
--- a/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
@@ -989,6 +989,7 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
MessageClientIDSetter.setUniqID(message);
message.setTopic(withNamespace(message.getTopic()));
}
+ MessageClientIDSetter.setUniqID(msgBatch);
msgBatch.setBody(msgBatch.encode());
} catch (Exception e) {
throw new MQClientException("Failed to initiate the MessageBatch", e);
diff --git a/common/src/main/java/org/apache/rocketmq/common/UtilAll.java b/common/src/main/java/org/apache/rocketmq/common/UtilAll.java
index ea22aa7..d3ca237 100644
--- a/common/src/main/java/org/apache/rocketmq/common/UtilAll.java
+++ b/common/src/main/java/org/apache/rocketmq/common/UtilAll.java
@@ -196,6 +196,19 @@ public class UtilAll {
cal.get(Calendar.SECOND));
}
+ public static long getTotalSpace(final String path) {
+ if (null == path || path.isEmpty())
+ return -1;
+ try {
+ File file = new File(path);
+ if (!file.exists())
+ return -1;
+ return file.getTotalSpace();
+ } catch (Exception e) {
+ return -1;
+ }
+ }
+
public static boolean isPathExists(final String path) {
File file = new File(path);
return file.exists();
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageConst.java b/common/src/main/java/org/apache/rocketmq/common/message/MessageConst.java
index 628bf4e..b5c14e9 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageConst.java
+++ b/common/src/main/java/org/apache/rocketmq/common/message/MessageConst.java
@@ -41,6 +41,8 @@ public class MessageConst {
public static final String PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX = "UNIQ_KEY";
public static final String PROPERTY_MAX_RECONSUME_TIMES = "MAX_RECONSUME_TIMES";
public static final String PROPERTY_CONSUME_START_TIMESTAMP = "CONSUME_START_TIME";
+ public static final String PROPERTY_INNER_NUM = "INNER_NUM";
+ public static final String PROPERTY_INNER_BASE = "INNER_BASE";
public static final String PROPERTY_TRANSACTION_PREPARED_QUEUE_OFFSET = "TRAN_PREPARED_QUEUE_OFFSET";
public static final String PROPERTY_TRANSACTION_CHECK_TIMES = "TRANSACTION_CHECK_TIMES";
public static final String PROPERTY_CHECK_IMMUNITY_TIME_IN_SECONDS = "CHECK_IMMUNITY_TIME_IN_SECONDS";
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java b/common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java
index bb023fa..1e5fe44 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java
+++ b/common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java
@@ -395,6 +395,22 @@ public class MessageDecoder {
return decodes(byteBuffer, true);
}
+ public static List<MessageExt> decodesBatch(ByteBuffer byteBuffer,
+ final boolean readBody,
+ final boolean decompressBody,
+ final boolean isClient) {
+ List<MessageExt> msgExts = new ArrayList<MessageExt>();
+ while (byteBuffer.hasRemaining()) {
+ MessageExt msgExt = decode(byteBuffer, readBody, decompressBody, isClient);
+ if (null != msgExt) {
+ msgExts.add(msgExt);
+ } else {
+ break;
+ }
+ }
+ return msgExts;
+ }
+
public static List<MessageExt> decodes(ByteBuffer byteBuffer, final boolean readBody) {
List<MessageExt> msgExts = new ArrayList<MessageExt>();
while (byteBuffer.hasRemaining()) {
@@ -569,4 +585,37 @@ public class MessageDecoder {
}
return msgs;
}
+
+ public static void decodeMessage(MessageExt messageExt, List<MessageExt> list) throws Exception {
+ List<Message> messages = MessageDecoder.decodeMessages(ByteBuffer.wrap(messageExt.getBody()));
+ for (int i = 0; i < messages.size(); i++) {
+ Message message = messages.get(i);
+ MessageClientExt messageClientExt = new MessageClientExt();
+ messageClientExt.setTopic(messageExt.getTopic());
+ messageClientExt.setQueueOffset(messageExt.getQueueOffset() + i);
+ messageClientExt.setQueueId(messageExt.getQueueId());
+ messageClientExt.setFlag(message.getFlag());
+ MessageAccessor.setProperties(messageClientExt, message.getProperties());
+ messageClientExt.setBody(message.getBody());
+ messageClientExt.setStoreHost(messageExt.getStoreHost());
+ messageClientExt.setBornHost(messageExt.getBornHost());
+ messageClientExt.setBornTimestamp(messageExt.getBornTimestamp());
+ messageClientExt.setStoreTimestamp(messageExt.getStoreTimestamp());
+ messageClientExt.setSysFlag(messageExt.getSysFlag());
+ messageClientExt.setCommitLogOffset(messageExt.getCommitLogOffset());
+ messageClientExt.setWaitStoreMsgOK(messageExt.isWaitStoreMsgOK());
+ list.add(messageClientExt);
+ }
+ }
+
+ public static int countInnerMsgNum(ByteBuffer buffer) {
+ int count = 0;
+ while (buffer.hasRemaining()) {
+ count++;
+ int currPos = buffer.position();
+ int size = buffer.getInt();
+ buffer.position(currPos + size);
+ }
+ return count;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
index e15170f..5ac1899 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/PullMessageRequestHeader.java
@@ -35,6 +35,8 @@ public class PullMessageRequestHeader extends TopicQueueRequestHeader {
@CFNotNull
private Long queueOffset;
@CFNotNull
+ private Integer maxMsgBytes;
+ @CFNotNull
private Integer maxMsgNums;
@CFNotNull
private Integer sysFlag;
@@ -143,4 +145,12 @@ public class PullMessageRequestHeader extends TopicQueueRequestHeader {
public void setExpressionType(String expressionType) {
this.expressionType = expressionType;
}
+
+ public Integer getMaxMsgBytes() {
+ return maxMsgBytes;
+ }
+
+ public void setMaxMsgBytes(Integer maxMsgBytes) {
+ this.maxMsgBytes = maxMsgBytes;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageResponseHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageResponseHeader.java
index 6834881..601f720 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageResponseHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageResponseHeader.java
@@ -32,6 +32,7 @@ public class SendMessageResponseHeader implements CommandCustomHeader {
@CFNotNull
private Long queueOffset;
private String transactionId;
+ private String batchUniqId;
@Override
public void checkFields() throws RemotingCommandException {
@@ -68,4 +69,12 @@ public class SendMessageResponseHeader implements CommandCustomHeader {
public void setTransactionId(String transactionId) {
this.transactionId = transactionId;
}
+
+ public String getBatchUniqId() {
+ return batchUniqId;
+ }
+
+ public void setBatchUniqId(String batchUniqId) {
+ this.batchUniqId = batchUniqId;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/sysflag/MessageSysFlag.java b/common/src/main/java/org/apache/rocketmq/common/sysflag/MessageSysFlag.java
index d534571..d28ac62 100644
--- a/common/src/main/java/org/apache/rocketmq/common/sysflag/MessageSysFlag.java
+++ b/common/src/main/java/org/apache/rocketmq/common/sysflag/MessageSysFlag.java
@@ -25,6 +25,8 @@ public class MessageSysFlag {
public final static int TRANSACTION_ROLLBACK_TYPE = 0x3 << 2;
public final static int BORNHOST_V6_FLAG = 0x1 << 4;
public final static int STOREHOSTADDRESS_V6_FLAG = 0x1 << 5;
+ public final static int NEED_UNWRAP_FLAG = 0x1 << 6;
+ public final static int INNER_BATCH_FLAG = 0x1 << 7;
public static int getTransactionValue(final int flag) {
return flag & TRANSACTION_ROLLBACK_TYPE;
@@ -38,4 +40,7 @@ public class MessageSysFlag {
return flag & (~COMPRESSED_FLAG);
}
+ public static boolean check(int flag, int expectedFlag) {
+ return (flag & expectedFlag) != 0;
+ }
}
diff --git a/common/src/test/java/org/apache/rocketmq/common/message/MessageDecoderTest.java b/common/src/test/java/org/apache/rocketmq/common/message/MessageDecoderTest.java
index b27f246..5af7345 100644
--- a/common/src/test/java/org/apache/rocketmq/common/message/MessageDecoderTest.java
+++ b/common/src/test/java/org/apache/rocketmq/common/message/MessageDecoderTest.java
@@ -17,6 +17,7 @@
package org.apache.rocketmq.common.message;
+import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.junit.Test;
import java.net.InetAddress;
@@ -239,7 +240,8 @@ public class MessageDecoderTest {
assertThat(1).isEqualTo(decodedMsg.getQueueId());
assertThat(123456L).isEqualTo(decodedMsg.getCommitLogOffset());
assertThat("hello!q!".getBytes()).isEqualTo(decodedMsg.getBody());
- assertThat(48).isEqualTo(decodedMsg.getSysFlag());
+ // assertThat(48).isEqualTo(decodedMsg.getSysFlag());
+ assertThat(MessageSysFlag.check(messageExt.getSysFlag(), MessageSysFlag.STOREHOSTADDRESS_V6_FLAG)).isTrue();
int msgIDLength = 16 + 4 + 8;
ByteBuffer byteBufferMsgId = ByteBuffer.allocate(msgIDLength);
diff --git a/store/src/main/java/org/apache/rocketmq/store/AllocateMappedFileService.java b/store/src/main/java/org/apache/rocketmq/store/AllocateMappedFileService.java
index acb1d54..847e99d 100644
--- a/store/src/main/java/org/apache/rocketmq/store/AllocateMappedFileService.java
+++ b/store/src/main/java/org/apache/rocketmq/store/AllocateMappedFileService.java
@@ -30,6 +30,8 @@ import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
+import org.apache.rocketmq.store.logfile.MappedFile;
/**
* Create MappedFile in advance
@@ -42,9 +44,9 @@ public class AllocateMappedFileService extends ServiceThread {
private PriorityBlockingQueue<AllocateRequest> requestQueue =
new PriorityBlockingQueue<AllocateRequest>();
private volatile boolean hasException = false;
- private DefaultMessageStore messageStore;
+ private MessageStore messageStore;
- public AllocateMappedFileService(DefaultMessageStore messageStore) {
+ public AllocateMappedFileService(MessageStore messageStore) {
this.messageStore = messageStore;
}
@@ -97,7 +99,9 @@ public class AllocateMappedFileService extends ServiceThread {
AllocateRequest result = this.requestTable.get(nextFilePath);
try {
if (result != null) {
+ messageStore.getPerfCounter().startTick("WAIT_MAPFILE_TIME_MS");
boolean waitOK = result.getCountDownLatch().await(waitTimeOut, TimeUnit.MILLISECONDS);
+ messageStore.getPerfCounter().endTick("WAIT_MAPFILE_TIME_MS");
if (!waitOK) {
log.warn("create mmap timeout " + result.getFilePath() + " " + result.getFileSize());
return null;
@@ -170,10 +174,10 @@ public class AllocateMappedFileService extends ServiceThread {
mappedFile.init(req.getFilePath(), req.getFileSize(), messageStore.getTransientStorePool());
} catch (RuntimeException e) {
log.warn("Use default implementation.");
- mappedFile = new MappedFile(req.getFilePath(), req.getFileSize(), messageStore.getTransientStorePool());
+ mappedFile = new DefaultMappedFile(req.getFilePath(), req.getFileSize(), messageStore.getTransientStorePool());
}
} else {
- mappedFile = new MappedFile(req.getFilePath(), req.getFileSize());
+ mappedFile = new DefaultMappedFile(req.getFilePath(), req.getFileSize());
}
long elapsedTime = UtilAll.computeElapsedTimeMilliseconds(beginTime);
diff --git a/store/src/main/java/org/apache/rocketmq/store/AppendMessageCallback.java b/store/src/main/java/org/apache/rocketmq/store/AppendMessageCallback.java
index 5499c90..accd4e2 100644
--- a/store/src/main/java/org/apache/rocketmq/store/AppendMessageCallback.java
+++ b/store/src/main/java/org/apache/rocketmq/store/AppendMessageCallback.java
@@ -17,8 +17,6 @@
package org.apache.rocketmq.store;
import java.nio.ByteBuffer;
-import org.apache.rocketmq.common.message.MessageExtBatch;
-import org.apache.rocketmq.store.CommitLog.PutMessageContext;
/**
* Write messages callback interface
diff --git a/store/src/main/java/org/apache/rocketmq/store/AppendMessageResult.java b/store/src/main/java/org/apache/rocketmq/store/AppendMessageResult.java
index de3c03b..3cfb85f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/AppendMessageResult.java
+++ b/store/src/main/java/org/apache/rocketmq/store/AppendMessageResult.java
@@ -65,6 +65,18 @@ public class AppendMessageResult {
this.pagecacheRT = pagecacheRT;
}
+ public AppendMessageResult(AppendMessageStatus status, long wroteOffset, int wroteBytes, Supplier<String> msgIdSupplier,
+ long storeTimestamp, long logicsOffset, long pagecacheRT, int msgNum) {
+ this.status = status;
+ this.wroteOffset = wroteOffset;
+ this.wroteBytes = wroteBytes;
+ this.msgIdSupplier = msgIdSupplier;
+ this.storeTimestamp = storeTimestamp;
+ this.logicsOffset = logicsOffset;
+ this.pagecacheRT = pagecacheRT;
+ this.msgNum = msgNum;
+ }
+
public long getPagecacheRT() {
return pagecacheRT;
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/CommitLog.java b/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
index 604d9c2..062c269 100644
--- a/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
+++ b/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
@@ -22,6 +22,7 @@ import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedList;
@@ -29,6 +30,9 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
import java.util.function.Supplier;
import org.apache.rocketmq.common.ServiceThread;
@@ -38,7 +42,6 @@ import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
@@ -47,66 +50,68 @@ import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.ha.HAService;
+import org.apache.rocketmq.store.logfile.MappedFile;
+import org.apache.rocketmq.store.queue.CQType;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
+import org.apache.rocketmq.store.util.QueueTypeUtils;
/**
* Store all metadata downtime for recovery, data protection reliability
*/
-public class CommitLog {
+public class CommitLog implements Swappable {
// Message's MAGIC CODE daa320a7
public final static int MESSAGE_MAGIC_CODE = -626843481;
protected static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
// End of file empty MAGIC CODE cbd43194
protected final static int BLANK_MAGIC_CODE = -875286124;
protected final MappedFileQueue mappedFileQueue;
- protected final DefaultMessageStore defaultMessageStore;
- private final FlushCommitLogService flushCommitLogService;
+ protected final MessageStore defaultMessageStore;
- //If TransientStorePool enabled, we must flush message to FileChannel at fixed periods
- private final FlushCommitLogService commitLogService;
+ private final FlushManager flushManager;
private final AppendMessageCallback appendMessageCallback;
private final ThreadLocal<PutMessageThreadLocal> putMessageThreadLocal;
- protected HashMap<String/* topic-queueid */, Long/* offset */> topicQueueTable = new HashMap<String, Long>(1024);
+
protected volatile long confirmOffset = -1L;
private volatile long beginTimeInLock = 0;
protected final PutMessageLock putMessageLock;
+ protected final TopicQueueLock topicQueueLock;
+
private volatile Set<String> fullStorePaths = Collections.emptySet();
- public CommitLog(final DefaultMessageStore defaultMessageStore) {
- String storePath = defaultMessageStore.getMessageStoreConfig().getStorePathCommitLog();
+ protected int commitLogSize;
+
+ public CommitLog(final MessageStore messageStore) {
+ String storePath = messageStore.getMessageStoreConfig().getStorePathCommitLog();
if (storePath.contains(MessageStoreConfig.MULTI_PATH_SPLITTER)) {
- this.mappedFileQueue = new MultiPathMappedFileQueue(defaultMessageStore.getMessageStoreConfig(),
- defaultMessageStore.getMessageStoreConfig().getMappedFileSizeCommitLog(),
- defaultMessageStore.getAllocateMappedFileService(), this::getFullStorePaths);
+ this.mappedFileQueue = new MultiPathMappedFileQueue(messageStore.getMessageStoreConfig(),
+ messageStore.getMessageStoreConfig().getMappedFileSizeCommitLog(),
+ messageStore.getAllocateMappedFileService(), this::getFullStorePaths);
} else {
this.mappedFileQueue = new MappedFileQueue(storePath,
- defaultMessageStore.getMessageStoreConfig().getMappedFileSizeCommitLog(),
- defaultMessageStore.getAllocateMappedFileService());
+ messageStore.getMessageStoreConfig().getMappedFileSizeCommitLog(),
+ messageStore.getAllocateMappedFileService());
}
- this.defaultMessageStore = defaultMessageStore;
+ this.defaultMessageStore = messageStore;
- if (FlushDiskType.SYNC_FLUSH == defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
- this.flushCommitLogService = new GroupCommitService();
- } else {
- this.flushCommitLogService = new FlushRealTimeService();
- }
+ this.flushManager = new DefaultFlushManager();
- this.commitLogService = new CommitRealTimeService();
-
- this.appendMessageCallback = new DefaultAppendMessageCallback(defaultMessageStore.getMessageStoreConfig().getMaxMessageSize());
+ this.appendMessageCallback = new DefaultAppendMessageCallback(messageStore.getMessageStoreConfig().getMaxMessageSize());
putMessageThreadLocal = new ThreadLocal<PutMessageThreadLocal>() {
@Override
protected PutMessageThreadLocal initialValue() {
return new PutMessageThreadLocal(defaultMessageStore.getMessageStoreConfig().getMaxMessageSize());
}
};
- this.putMessageLock = defaultMessageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage() ? new PutMessageReentrantLock() : new PutMessageSpinLock();
+ this.putMessageLock = messageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage() ? new PutMessageReentrantLock() : new PutMessageSpinLock();
+
+ this.topicQueueLock = new TopicQueueLock();
+ this.commitLogSize = messageStore.getMessageStoreConfig().getMappedFileSizeCommitLog();
}
public void setFullStorePaths(Set<String> fullStorePaths) {
@@ -119,24 +124,19 @@ public class CommitLog {
public boolean load() {
boolean result = this.mappedFileQueue.load();
+ this.mappedFileQueue.checkSelf();
log.info("load commit log " + (result ? "OK" : "Failed"));
return result;
}
public void start() {
- this.flushCommitLogService.start();
-
- if (defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
- this.commitLogService.start();
- }
+ this.flushManager.start();
+ log.info("start commitLog successfully. storeRoot: {}", this.defaultMessageStore.getMessageStoreConfig().getStorePathRootDir());
}
public void shutdown() {
- if (defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
- this.commitLogService.shutdown();
- }
-
- this.flushCommitLogService.shutdown();
+ this.flushManager.shutdown();
+ log.info("shutdown commitLog successfully. storeRoot: {}", this.defaultMessageStore.getMessageStoreConfig().getStorePathRootDir());
}
public long flush() {
@@ -211,17 +211,21 @@ public class CommitLog {
ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
long processOffset = mappedFile.getFileFromOffset();
long mappedFileOffset = 0;
+ // normal recover doesn't require dispatching
+ boolean doDispatch = false;
while (true) {
DispatchRequest dispatchRequest = this.checkMessageAndReturnSize(byteBuffer, checkCRCOnRecover);
int size = dispatchRequest.getMsgSize();
// Normal data
if (dispatchRequest.isSuccess() && size > 0) {
mappedFileOffset += size;
+ this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, false);
}
// Come the end of the file, switch to the next file Since the
// return 0 representatives met last hole,
// this can not be included in truncate offset
else if (dispatchRequest.isSuccess() && size == 0) {
+ this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, true);
index++;
if (index >= mappedFiles.size()) {
// Current branch can not happen
@@ -402,26 +406,38 @@ public class CommitLog {
return new DispatchRequest(totalSize, false/* success */);
}
- return new DispatchRequest(
- topic,
- queueId,
- physicOffset,
- totalSize,
- tagsCode,
- storeTimestamp,
- queueOffset,
- keys,
- uniqKey,
- sysFlag,
- preparedTransactionOffset,
- propertiesMap
+ DispatchRequest dispatchRequest = new DispatchRequest(
+ topic,
+ queueId,
+ physicOffset,
+ totalSize,
+ tagsCode,
+ storeTimestamp,
+ queueOffset,
+ keys,
+ uniqKey,
+ sysFlag,
+ preparedTransactionOffset,
+ propertiesMap
);
+
+ setBatchSizeIfNeeded(propertiesMap, dispatchRequest);
+
+ return dispatchRequest;
} catch (Exception e) {
+ log.error("CheckMessageAndReturnSizeOld", e);
}
return new DispatchRequest(-1, false /* success */);
}
+ private void setBatchSizeIfNeeded(Map<String, String> propertiesMap, DispatchRequest dispatchRequest) {
+ if (null != propertiesMap && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_NUM) && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_BASE)) {
+ dispatchRequest.setMsgBaseOffset(Long.valueOf(propertiesMap.get(MessageConst.PROPERTY_INNER_BASE)));
+ dispatchRequest.setBatchSize(Short.valueOf(propertiesMap.get(MessageConst.PROPERTY_INNER_NUM)));
+ }
+ }
+
protected static int calMsgLength(int sysFlag, int bodyLength, int topicLength, int propertiesLength) {
int bornhostLength = (sysFlag & MessageSysFlag.BORNHOST_V6_FLAG) == 0 ? 8 : 20;
int storehostAddressLength = (sysFlag & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 8 : 20;
@@ -479,6 +495,8 @@ public class CommitLog {
ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
long processOffset = mappedFile.getFileFromOffset();
long mappedFileOffset = 0;
+ // abnormal recover require dispatching
+ boolean doDispatch = true;
while (true) {
DispatchRequest dispatchRequest = this.checkMessageAndReturnSize(byteBuffer, checkCRCOnRecover);
int size = dispatchRequest.getMsgSize();
@@ -490,16 +508,17 @@ public class CommitLog {
if (this.defaultMessageStore.getMessageStoreConfig().isDuplicationEnable()) {
if (dispatchRequest.getCommitLogOffset() < this.defaultMessageStore.getConfirmOffset()) {
- this.defaultMessageStore.doDispatch(dispatchRequest);
+ this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, false);
}
} else {
- this.defaultMessageStore.doDispatch(dispatchRequest);
+ this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, false);
}
}
// Come the end of the file, switch to the next file
// Since the return 0 representatives met last hole, this can
// not be included in truncate offset
else if (size == 0) {
+ this.getMessageStore().onCommitLogDispatch(dispatchRequest, doDispatch, mappedFile, true, true);
index++;
if (index >= mappedFiles.size()) {
// The current branch under normal circumstances should
@@ -540,6 +559,10 @@ public class CommitLog {
}
}
+ protected void onCommitLogAppend(MessageExtBrokerInner msg, AppendMessageResult result, MappedFile commitLogFile) {
+ this.getMessageStore().onCommitLogAppend(msg, result, commitLogFile);
+ }
+
private boolean isMappedFileMatchedRecover(final MappedFile mappedFile) {
ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
@@ -643,67 +666,80 @@ public class CommitLog {
}
PutMessageThreadLocal putMessageThreadLocal = this.putMessageThreadLocal.get();
- PutMessageResult encodeResult = putMessageThreadLocal.getEncoder().encode(msg);
- if (encodeResult != null) {
- return CompletableFuture.completedFuture(encodeResult);
- }
- msg.setEncodedBuff(putMessageThreadLocal.getEncoder().encoderBuffer);
- PutMessageContext putMessageContext = new PutMessageContext(generateKey(putMessageThreadLocal.getKeyBuilder(), msg));
-
+ String topicQueueKey = generateKey(putMessageThreadLocal.getKeyBuilder(), msg);
long elapsedTimeInLock = 0;
MappedFile unlockMappedFile = null;
- putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
+ topicQueueLock.lock(topicQueueKey);
try {
- MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
- long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
- this.beginTimeInLock = beginLockTimestamp;
-
- // Here settings are stored timestamp, in order to ensure an orderly
- // global
- msg.setStoreTimestamp(beginLockTimestamp);
+ defaultMessageStore.assignOffset(topicQueueKey, msg, getBatchNum(msg));
- if (null == mappedFile || mappedFile.isFull()) {
- mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
- }
- if (null == mappedFile) {
- log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null));
+ PutMessageResult encodeResult = putMessageThreadLocal.getEncoder().encode(msg);
+ if (encodeResult != null) {
+ return CompletableFuture.completedFuture(encodeResult);
}
+ msg.setEncodedBuff(putMessageThreadLocal.getEncoder().encoderBuffer);
+ PutMessageContext putMessageContext = new PutMessageContext(topicQueueKey);
- result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
- switch (result.getStatus()) {
- case PUT_OK:
- break;
- case END_OF_FILE:
- unlockMappedFile = mappedFile;
- // Create a new file, re-write the message
- mappedFile = this.mappedFileQueue.getLastMappedFile(0);
- if (null == mappedFile) {
- // XXX: warn and notify me
- log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result));
- }
- result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
- break;
- case MESSAGE_SIZE_EXCEEDED:
- case PROPERTIES_SIZE_EXCEEDED:
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result));
- case UNKNOWN_ERROR:
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
- default:
+ putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
+ try {
+ MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
+ long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
+ this.beginTimeInLock = beginLockTimestamp;
+
+ // Here settings are stored timestamp, in order to ensure an orderly
+ // global
+ msg.setStoreTimestamp(beginLockTimestamp);
+
+ if (null == mappedFile || mappedFile.isFull()) {
+ mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
+ }
+ if (null == mappedFile) {
+ log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
- }
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null));
+ }
- elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
- beginTimeInLock = 0;
+ result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
+ switch (result.getStatus()) {
+ case PUT_OK:
+ onCommitLogAppend(msg, result, mappedFile);
+ break;
+ case END_OF_FILE:
+ onCommitLogAppend(msg, result, mappedFile);
+ unlockMappedFile = mappedFile;
+ // Create a new file, re-write the message
+ mappedFile = this.mappedFileQueue.getLastMappedFile(0);
+ if (null == mappedFile) {
+ // XXX: warn and notify me
+ log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result));
+ }
+ result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
+ if (AppendMessageStatus.PUT_OK.equals(result.getStatus())) {
+ onCommitLogAppend(msg, result, mappedFile);
+ }
+ break;
+ case MESSAGE_SIZE_EXCEEDED:
+ case PROPERTIES_SIZE_EXCEEDED:
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result));
+ case UNKNOWN_ERROR:
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
+ default:
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
+ }
+
+ elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
+ beginTimeInLock = 0;
+ } finally {
+ putMessageLock.unlock();
+ }
} finally {
- putMessageLock.unlock();
+ topicQueueLock.unlock(topicQueueKey);
}
if (elapsedTimeInLock > 500) {
@@ -770,57 +806,66 @@ public class CommitLog {
PutMessageThreadLocal pmThreadLocal = this.putMessageThreadLocal.get();
MessageExtEncoder batchEncoder = pmThreadLocal.getEncoder();
- PutMessageContext putMessageContext = new PutMessageContext(generateKey(pmThreadLocal.getKeyBuilder(), messageExtBatch));
+ String topicQueueKey = generateKey(pmThreadLocal.getKeyBuilder(), messageExtBatch);
+
+ PutMessageContext putMessageContext = new PutMessageContext(topicQueueKey);
messageExtBatch.setEncodedBuff(batchEncoder.encode(messageExtBatch, putMessageContext));
- putMessageLock.lock();
+ topicQueueLock.lock(topicQueueKey);
try {
- long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
- this.beginTimeInLock = beginLockTimestamp;
+ defaultMessageStore.assignOffset(topicQueueKey, messageExtBatch, (short) putMessageContext.getBatchSize());
- // Here settings are stored timestamp, in order to ensure an orderly
- // global
- messageExtBatch.setStoreTimestamp(beginLockTimestamp);
+ putMessageLock.lock();
+ try {
+ long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
+ this.beginTimeInLock = beginLockTimestamp;
- if (null == mappedFile || mappedFile.isFull()) {
- mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
- }
- if (null == mappedFile) {
- log.error("Create mapped file1 error, topic: {} clientAddr: {}", messageExtBatch.getTopic(), messageExtBatch.getBornHostString());
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null));
- }
+ // Here settings are stored timestamp, in order to ensure an orderly
+ // global
+ messageExtBatch.setStoreTimestamp(beginLockTimestamp);
- result = mappedFile.appendMessages(messageExtBatch, this.appendMessageCallback, putMessageContext);
- switch (result.getStatus()) {
- case PUT_OK:
- break;
- case END_OF_FILE:
- unlockMappedFile = mappedFile;
- // Create a new file, re-write the message
- mappedFile = this.mappedFileQueue.getLastMappedFile(0);
- if (null == mappedFile) {
- // XXX: warn and notify me
- log.error("Create mapped file2 error, topic: {} clientAddr: {}", messageExtBatch.getTopic(), messageExtBatch.getBornHostString());
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result));
- }
- result = mappedFile.appendMessages(messageExtBatch, this.appendMessageCallback, putMessageContext);
- break;
- case MESSAGE_SIZE_EXCEEDED:
- case PROPERTIES_SIZE_EXCEEDED:
- beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result));
- case UNKNOWN_ERROR:
- default:
+ if (null == mappedFile || mappedFile.isFull()) {
+ mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
+ }
+ if (null == mappedFile) {
+ log.error("Create mapped file1 error, topic: {} clientAddr: {}", messageExtBatch.getTopic(), messageExtBatch.getBornHostString());
beginTimeInLock = 0;
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
- }
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null));
+ }
+
+ result = mappedFile.appendMessages(messageExtBatch, this.appendMessageCallback, putMessageContext);
+ switch (result.getStatus()) {
+ case PUT_OK:
+ break;
+ case END_OF_FILE:
+ unlockMappedFile = mappedFile;
+ // Create a new file, re-write the message
+ mappedFile = this.mappedFileQueue.getLastMappedFile(0);
+ if (null == mappedFile) {
+ // XXX: warn and notify me
+ log.error("Create mapped file2 error, topic: {} clientAddr: {}", messageExtBatch.getTopic(), messageExtBatch.getBornHostString());
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result));
+ }
+ result = mappedFile.appendMessages(messageExtBatch, this.appendMessageCallback, putMessageContext);
+ break;
+ case MESSAGE_SIZE_EXCEEDED:
+ case PROPERTIES_SIZE_EXCEEDED:
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result));
+ case UNKNOWN_ERROR:
+ default:
+ beginTimeInLock = 0;
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
+ }
- elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
- beginTimeInLock = 0;
+ elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
+ beginTimeInLock = 0;
+ } finally {
+ putMessageLock.unlock();
+ }
} finally {
- putMessageLock.unlock();
+ topicQueueLock.unlock(topicQueueKey);
}
if (elapsedTimeInLock > 500) {
@@ -856,28 +901,7 @@ public class CommitLog {
}
public CompletableFuture<PutMessageStatus> submitFlushRequest(AppendMessageResult result, MessageExt messageExt) {
- // Synchronization flush
- if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
- final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
- if (messageExt.isWaitStoreMsgOK()) {
- GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes(),
- this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
- service.putRequest(request);
- return request.future();
- } else {
- service.wakeup();
- return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
- }
- }
- // Asynchronous flush
- else {
- if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
- flushCommitLogService.wakeup();
- } else {
- commitLogService.wakeup();
- }
- return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
- }
+ return this.flushManager.handleDiskFlush(result, messageExt);
}
public CompletableFuture<PutMessageStatus> submitReplicaRequest(AppendMessageResult result, MessageExt messageExt) {
@@ -899,6 +923,42 @@ public class CommitLog {
return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
}
+
+ public void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {
+ this.flushManager.handleDiskFlush(result, putMessageResult, messageExt);
+ }
+
+ public void handleHA(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {
+ if (BrokerRole.SYNC_MASTER == this.defaultMessageStore.getMessageStoreConfig().getBrokerRole()) {
+ HAService service = this.defaultMessageStore.getHaService();
+ if (messageExt.isWaitStoreMsgOK()) {
+ // Determine whether to wait
+ if (service.isSlaveOK(result.getWroteOffset() + result.getWroteBytes())) {
+ GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());
+ service.putRequest(request);
+ service.getWaitNotifyObject().wakeupAll();
+ PutMessageStatus replicaStatus = null;
+ try {
+ replicaStatus = request.future().get(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout(),
+ TimeUnit.MILLISECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ }
+ if (replicaStatus != PutMessageStatus.PUT_OK) {
+ log.error("do sync transfer other node, wait return, but failed, topic: " + messageExt.getTopic() + " tags: "
+ + messageExt.getTags() + " client address: " + messageExt.getBornHostNameString());
+ putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_SLAVE_TIMEOUT);
+ }
+ }
+ // Slave problem
+ else {
+ // Tell the producer, slave not available
+ putMessageResult.setPutMessageStatus(PutMessageStatus.SLAVE_NOT_AVAILABLE);
+ }
+ }
+ }
+
+ }
+
/**
* According to receive certain message or offset storage time if an error occurs, it returns -1
*/
@@ -948,14 +1008,6 @@ public class CommitLog {
return offset + mappedFileSize - offset % mappedFileSize;
}
- public HashMap<String, Long> getTopicQueueTable() {
- return topicQueueTable;
- }
-
- public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
- this.topicQueueTable = topicQueueTable;
- }
-
public void destroy() {
this.mappedFileQueue.destroy();
}
@@ -982,7 +1034,7 @@ public class CommitLog {
public void removeQueueFromTopicQueueTable(final String topic, final int queueId) {
String key = topic + "-" + queueId;
synchronized (this) {
- this.topicQueueTable.remove(key);
+ this.defaultMessageStore.removeOffsetTable(key);
}
log.info("removeQueueFromTopicQueueTable OK Topic: {} QueueId: {}", topic, queueId);
@@ -1006,6 +1058,21 @@ public class CommitLog {
return diff;
}
+ protected short getBatchNum(MessageExtBrokerInner msgInner) {
+ short batchNum = 1;
+ // IF inner batch, build batchQueueOffset and batchNum property.
+ CQType cqType = QueueTypeUtils.getCQType(defaultMessageStore);
+ if (MessageSysFlag.check(msgInner.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG) || CQType.BatchCQ.equals(cqType)) {
+
+ if (msgInner.getProperty(MessageConst.PROPERTY_INNER_NUM) != null) {
+ batchNum = Short.parseShort(msgInner.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ batchNum = batchNum >= 1 ? batchNum : 1;
+ }
+ }
+
+ return batchNum;
+ }
+
abstract class FlushCommitLogService extends ServiceThread {
protected static final int RETRY_TIMES_OVER = 10;
}
@@ -1041,10 +1108,9 @@ public class CommitLog {
long end = System.currentTimeMillis();
if (!result) {
this.lastCommitTimestamp = end; // result = false means some data committed.
- //now wake up flush thread.
- flushCommitLogService.wakeup();
+ CommitLog.this.flushManager.wakeUpFlush();
}
-
+ CommitLog.this.getMessageStore().getPerfCounter().flowOnce("COMMIT_DATA_TIME_MS", (int)(end - begin));
if (end - begin > 500) {
log.info("Commit data to file costs {} ms", end - begin);
}
@@ -1107,6 +1173,7 @@ public class CommitLog {
CommitLog.this.defaultMessageStore.getStoreCheckpoint().setPhysicMsgTimestamp(storeTimestamp);
}
long past = System.currentTimeMillis() - begin;
+ CommitLog.this.getMessageStore().getPerfCounter().flowOnce("FLUSH_DATA_TIME_MS", (int)past);
if (past > 500) {
log.info("Flush data to disk costs {} ms", past);
}
@@ -1275,6 +1342,114 @@ public class CommitLog {
}
}
+ class GroupCheckService extends FlushCommitLogService {
+ private volatile List<GroupCommitRequest> requestsWrite = new ArrayList<GroupCommitRequest>();
+ private volatile List<GroupCommitRequest> requestsRead = new ArrayList<GroupCommitRequest>();
+
+ public boolean isAynscRequestsFull() {
+ return requestsWrite.size() > CommitLog.this.defaultMessageStore.getMessageStoreConfig().getMaxAsyncPutMessageRequests() * 2;
+ }
+
+ public synchronized boolean putRequest(final GroupCommitRequest request) {
+ synchronized (this.requestsWrite) {
+ this.requestsWrite.add(request);
+ }
+ if (hasNotified.compareAndSet(false, true)) {
+ waitPoint.countDown(); // notify
+ }
+ boolean flag = this.requestsWrite.size() >
+ CommitLog.this.defaultMessageStore.getMessageStoreConfig().getMaxAsyncPutMessageRequests();
+ if (flag) {
+ log.info("Async requests {} exceeded the threshold {}", requestsWrite.size(),
+ CommitLog.this.defaultMessageStore.getMessageStoreConfig().getMaxAsyncPutMessageRequests());
+ }
+
+ return flag;
+ }
+
+ private void swapRequests() {
+ List<GroupCommitRequest> tmp = this.requestsWrite;
+ this.requestsWrite = this.requestsRead;
+ this.requestsRead = tmp;
+ }
+
+ private void doCommit() {
+ synchronized (this.requestsRead) {
+ if (!this.requestsRead.isEmpty()) {
+ for (GroupCommitRequest req : this.requestsRead) {
+ // There may be a message in the next file, so a maximum of
+ // two times the flush
+ boolean flushOK = false;
+ for (int i = 0; i < 1000; i++) {
+ flushOK = CommitLog.this.mappedFileQueue.getFlushedWhere() >= req.getNextOffset();
+ if (flushOK) {
+ break;
+ } else {
+ try {
+ Thread.sleep(1);
+ } catch (Throwable ignored) {
+
+ }
+ }
+ }
+ req.wakeupCustomer(flushOK ? PutMessageStatus.PUT_OK : PutMessageStatus.FLUSH_DISK_TIMEOUT);
+ }
+
+ long storeTimestamp = CommitLog.this.mappedFileQueue.getStoreTimestamp();
+ if (storeTimestamp > 0) {
+ CommitLog.this.defaultMessageStore.getStoreCheckpoint().setPhysicMsgTimestamp(storeTimestamp);
+ }
+
+ this.requestsRead.clear();
+ }
+ }
+ }
+
+ public void run() {
+ CommitLog.log.info(this.getServiceName() + " service started");
+
+ while (!this.isStopped()) {
+ try {
+ this.waitForRunning(1);
+ this.doCommit();
+ } catch (Exception e) {
+ CommitLog.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ // Under normal circumstances shutdown, wait for the arrival of the
+ // request, and then flush
+ try {
+ Thread.sleep(10);
+ } catch (InterruptedException e) {
+ CommitLog.log.warn("GroupCommitService Exception, ", e);
+ }
+
+ synchronized (this) {
+ this.swapRequests();
+ }
+
+ this.doCommit();
+
+ CommitLog.log.info(this.getServiceName() + " service end");
+ }
+
+ @Override
+ protected void onWaitEnd() {
+ this.swapRequests();
+ }
+
+ @Override
+ public String getServiceName() {
+ return CommitLog.GroupCheckService.class.getSimpleName();
+ }
+
+ @Override
+ public long getJointime() {
+ return 1000 * 60 * 5;
+ }
+ }
+
class DefaultAppendMessageCallback implements AppendMessageCallback {
// File at the end of the minimum fixed length empty
private static final int END_FILE_MIN_BLANK_LENGTH = 4 + 4;
@@ -1310,12 +1485,10 @@ public class CommitLog {
};
// Record ConsumeQueue information
- String key = putMessageContext.getTopicQueueTableKey();
- Long queueOffset = CommitLog.this.topicQueueTable.get(key);
- if (null == queueOffset) {
- queueOffset = 0L;
- CommitLog.this.topicQueueTable.put(key, queueOffset);
- }
+ Long queueOffset = msgInner.getQueueOffset();
+
+ // this msg maybe a inner-batch msg.
+ short batchNum = getBatchNum(msgInner);
// Transaction messages that require special handling
final int tranType = MessageSysFlag.getTransactionValue(msgInner.getSysFlag());
@@ -1366,25 +1539,13 @@ public class CommitLog {
final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
+ CommitLog.this.getMessageStore().getPerfCounter().startTick("WRITE_MEMORY_TIME_MS");
// Write messages to the queue buffer
byteBuffer.put(preEncodeBuffer);
+ CommitLog.this.getMessageStore().getPerfCounter().endTick("WRITE_MEMORY_TIME_MS");
msgInner.setEncodedBuff(null);
- AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgIdSupplier,
- msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills);
-
- switch (tranType) {
- case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
- case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
- break;
- case MessageSysFlag.TRANSACTION_NOT_TYPE:
- case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
- // The next update ConsumeQueue information
- CommitLog.this.topicQueueTable.put(key, ++queueOffset);
- break;
- default:
- break;
- }
- return result;
+ return new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgIdSupplier,
+ msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills, batchNum);
}
public AppendMessageResult doAppend(final long fileFromOffset, final ByteBuffer byteBuffer, final int maxBlank,
@@ -1393,12 +1554,7 @@ public class CommitLog {
//physical offset
long wroteOffset = fileFromOffset + byteBuffer.position();
// Record ConsumeQueue information
- String key = putMessageContext.getTopicQueueTableKey();
- Long queueOffset = CommitLog.this.topicQueueTable.get(key);
- if (null == queueOffset) {
- queueOffset = 0L;
- CommitLog.this.topicQueueTable.put(key, queueOffset);
- }
+ Long queueOffset = messageExtBatch.getQueueOffset();
long beginQueueOffset = queueOffset;
int totalMsgLen = 0;
int msgNum = 0;
@@ -1482,7 +1638,6 @@ public class CommitLog {
AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, totalMsgLen, msgIdSupplier,
messageExtBatch.getStoreTimestamp(), beginQueueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills);
result.setMsgNum(msgNum);
- CommitLog.this.topicQueueTable.put(key, queueOffset);
return result;
}
@@ -1537,6 +1692,8 @@ public class CommitLog {
final int msgLen = calMsgLength(msgInner.getSysFlag(), bodyLength, topicLength, propertiesLength);
+ final long queueOffset = msgInner.getQueueOffset();
+
// Exceeds the maximum message
if (msgLen > this.maxMessageSize) {
CommitLog.log.warn("message size exceeded, msg total size: " + msgLen + ", msg body size: " + bodyLength
@@ -1556,8 +1713,8 @@ public class CommitLog {
this.encoderBuffer.putInt(msgInner.getQueueId());
// 5 FLAG
this.encoderBuffer.putInt(msgInner.getFlag());
- // 6 QUEUEOFFSET, need update later
- this.encoderBuffer.putLong(0);
+ // 6 QUEUEOFFSET
+ this.encoderBuffer.putLong(queueOffset);
// 7 PHYSICALOFFSET, need update later
this.encoderBuffer.putLong(0);
// 8 SYSFLAG
@@ -1706,50 +1863,153 @@ public class CommitLog {
}
- static class PutMessageThreadLocal {
- private MessageExtEncoder encoder;
- private StringBuilder keyBuilder;
- PutMessageThreadLocal(int size) {
- encoder = new MessageExtEncoder(size);
- keyBuilder = new StringBuilder();
- }
+ interface FlushManager {
+ void start();
+ void shutdown();
+ void wakeUpFlush();
+ void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt);
+ CompletableFuture<PutMessageStatus> handleDiskFlush(AppendMessageResult result, MessageExt messageExt);
+ }
- public MessageExtEncoder getEncoder() {
- return encoder;
+ class DefaultFlushManager implements FlushManager {
+
+ private final FlushCommitLogService flushCommitLogService;
+
+ //If TransientStorePool enabled, we must flush message to FileChannel at fixed periods
+ private final FlushCommitLogService commitLogService;
+
+ public DefaultFlushManager() {
+ if (FlushDiskType.SYNC_FLUSH == CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
+ this.flushCommitLogService = new CommitLog.GroupCommitService();
+ } else {
+ this.flushCommitLogService = new CommitLog.FlushRealTimeService();
+ }
+
+ this.commitLogService = new CommitLog.CommitRealTimeService();
}
- public StringBuilder getKeyBuilder() {
- return keyBuilder;
+ @Override
+ public void start() {
+ this.flushCommitLogService.start();
+
+ if (defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
+ this.commitLogService.start();
+ }
}
- }
- static class PutMessageContext {
- private String topicQueueTableKey;
- private long[] phyPos;
- private int batchSize;
+ public void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {
+ // Synchronization flush
+ if (FlushDiskType.SYNC_FLUSH == CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
+ final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
+ if (messageExt.isWaitStoreMsgOK()) {
+ GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());
+ service.putRequest(request);
+ CompletableFuture<PutMessageStatus> flushOkFuture = request.future();
+ PutMessageStatus flushStatus = null;
+ try {
+ flushStatus = flushOkFuture.get(CommitLog.this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout(),
+ TimeUnit.MILLISECONDS);
+ } catch (InterruptedException | ExecutionException | TimeoutException e) {
+ //flushOK=false;
+ }
+ if (flushStatus != PutMessageStatus.PUT_OK) {
+ log.error("do groupcommit, wait for flush failed, topic: " + messageExt.getTopic() + " tags: " + messageExt.getTags()
+ + " client address: " + messageExt.getBornHostString());
+ putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_DISK_TIMEOUT);
+ }
+ } else {
+ service.wakeup();
+ }
+ }
+ // Asynchronous flush
+ else {
+ if (!CommitLog.this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
+ flushCommitLogService.wakeup();
+ } else {
+ commitLogService.wakeup();
+ }
+ }
+ }
- public PutMessageContext(String topicQueueTableKey) {
- this.topicQueueTableKey = topicQueueTableKey;
+ @Override
+ public CompletableFuture<PutMessageStatus> handleDiskFlush(AppendMessageResult result, MessageExt messageExt) {
+ // Synchronization flush
+ if (FlushDiskType.SYNC_FLUSH == CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
+ final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
+ if (messageExt.isWaitStoreMsgOK()) {
+ GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes(),
+ CommitLog.this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
+ service.putRequest(request);
+ return request.future();
+ } else {
+ service.wakeup();
+ return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
+ }
+ }
+ // Asynchronous flush
+ else {
+ if (!CommitLog.this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
+ flushCommitLogService.wakeup();
+ } else {
+ commitLogService.wakeup();
+ }
+ return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
+ }
}
- public String getTopicQueueTableKey() {
- return topicQueueTableKey;
+ @Override
+ public void wakeUpFlush() {
+ // now wake up flush thread.
+ flushCommitLogService.wakeup();
}
- public long[] getPhyPos() {
- return phyPos;
+ @Override
+ public void shutdown() {
+ if (defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
+ this.commitLogService.shutdown();
+ }
+
+ this.flushCommitLogService.shutdown();
}
- public void setPhyPos(long[] phyPos) {
- this.phyPos = phyPos;
+ }
+
+ public int getCommitLogSize() {
+ return commitLogSize;
+ }
+
+ public MappedFileQueue getMappedFileQueue() {
+ return mappedFileQueue;
+ }
+
+ public MessageStore getMessageStore() {
+ return defaultMessageStore;
+ }
+
+ @Override
+ public void swapMap(int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs) {
+ this.getMappedFileQueue().swapMap(reserveNum, forceSwapIntervalMs, normalSwapIntervalMs);
+ }
+
+ @Override
+ public void cleanSwappedMap(long forceCleanSwapIntervalMs) {
+ this.getMappedFileQueue().cleanSwappedMap(forceCleanSwapIntervalMs);
+ }
+
+ static class PutMessageThreadLocal {
+ private MessageExtEncoder encoder;
+ private StringBuilder keyBuilder;
+ PutMessageThreadLocal(int size) {
+ encoder = new MessageExtEncoder(size);
+ keyBuilder = new StringBuilder();
}
- public int getBatchSize() {
- return batchSize;
+ public MessageExtEncoder getEncoder() {
+ return encoder;
}
- public void setBatchSize(int batchSize) {
- this.batchSize = batchSize;
+ public StringBuilder getKeyBuilder() {
+ return keyBuilder;
}
}
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
index 87ff0a0..0efe74c 100644
--- a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
@@ -24,8 +24,14 @@ import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
-
-public class ConsumeQueue {
+import org.apache.rocketmq.store.logfile.MappedFile;
+import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.queue.FileQueueLifeCycle;
+import org.apache.rocketmq.store.queue.ReferredIterator;
+
+public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
public static final int CQ_STORE_UNIT_SIZE = 20;
@@ -76,6 +82,7 @@ public class ConsumeQueue {
}
}
+ @Override
public boolean load() {
boolean result = this.mappedFileQueue.load();
log.info("load consume queue " + this.topic + "-" + this.queueId + " " + (result ? "OK" : "Failed"));
@@ -85,6 +92,7 @@ public class ConsumeQueue {
return result;
}
+ @Override
public void recover() {
final List<MappedFile> mappedFiles = this.mappedFileQueue.getMappedFiles();
if (!mappedFiles.isEmpty()) {
@@ -152,6 +160,7 @@ public class ConsumeQueue {
}
}
+ @Override
public long getOffsetInQueueByTime(final long timestamp) {
MappedFile mappedFile = this.mappedFileQueue.getMappedFileByTime(timestamp);
if (mappedFile != null) {
@@ -221,6 +230,7 @@ public class ConsumeQueue {
return 0;
}
+ @Override
public void truncateDirtyLogicFiles(long phyOffet) {
int logicFileSize = this.mappedFileSize;
@@ -291,6 +301,7 @@ public class ConsumeQueue {
}
}
+ @Override
public long getLastOffset() {
long lastOffset = -1;
@@ -321,6 +332,7 @@ public class ConsumeQueue {
return lastOffset;
}
+ @Override
public boolean flush(final int flushLeastPages) {
boolean result = this.mappedFileQueue.flush(flushLeastPages);
if (isExtReadEnable()) {
@@ -330,12 +342,14 @@ public class ConsumeQueue {
return result;
}
+ @Override
public int deleteExpiredFile(long offset) {
int cnt = this.mappedFileQueue.deleteExpiredFileByOffset(offset, CQ_STORE_UNIT_SIZE);
this.correctMinOffset(offset);
return cnt;
}
+ @Override
public void correctMinOffset(long phyMinOffset) {
MappedFile mappedFile = this.mappedFileQueue.getFirstMappedFile();
long minExtAddr = 1;
@@ -372,10 +386,12 @@ public class ConsumeQueue {
}
}
+ @Override
public long getMinOffsetInQueue() {
return this.minLogicOffset / CQ_STORE_UNIT_SIZE;
}
+ @Override
public void putMessagePositionInfoWrapper(DispatchRequest request) {
final int maxRetries = 30;
boolean canWrite = this.defaultMessageStore.getRunningFlags().isCQWriteable();
@@ -488,7 +504,7 @@ public class ConsumeQueue {
}
}
- public SelectMappedBufferResult getIndexBuffer(final long startIndex) {
+ private SelectMappedBufferResult getIndexBuffer(final long startIndex) {
int mappedFileSize = this.mappedFileSize;
long offset = startIndex * CQ_STORE_UNIT_SIZE;
if (offset >= this.getMinLogicOffset()) {
@@ -501,6 +517,124 @@ public class ConsumeQueue {
return null;
}
+ @Override
+ public ReferredIterator<CqUnit> iterateFrom(long startOffset) {
+ SelectMappedBufferResult sbr = getIndexBuffer(startOffset);
+ if (sbr == null) {
+ return null;
+ }
+ return new ConsumeQueueIterator(sbr);
+ }
+
+ @Override
+ public CqUnit get(long offset) {
+ ReferredIterator<CqUnit> it = iterateFrom(offset);
+ if (it == null) {
+ return null;
+ }
+ return it.nextAndRelease();
+ }
+
+ @Override
+ public CqUnit getEarliestUnit() {
+ /**
+ * here maybe should not return null
+ */
+ ReferredIterator<CqUnit> it = iterateFrom(minLogicOffset / CQ_STORE_UNIT_SIZE);
+ if (it == null) {
+ return null;
+ }
+ return it.nextAndRelease();
+ }
+
+ @Override
+ public CqUnit getLatestUnit() {
+ ReferredIterator<CqUnit> it = iterateFrom((mappedFileQueue.getMaxOffset() / CQ_STORE_UNIT_SIZE) - 1);
+ if (it == null) {
+ return null;
+ }
+ return it.nextAndRelease();
+ }
+
+ @Override
+ public boolean isFirstFileAvailable() {
+ return false;
+ }
+
+ @Override
+ public boolean isFirstFileExist() {
+ return false;
+ }
+
+ private class ConsumeQueueIterator implements ReferredIterator<CqUnit> {
+ private SelectMappedBufferResult sbr;
+ private int relativePos = 0;
+
+ public ConsumeQueueIterator(SelectMappedBufferResult sbr) {
+ this.sbr = sbr;
+ if (sbr != null && sbr.getByteBuffer() != null) {
+ relativePos = sbr.getByteBuffer().position();
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (sbr == null || sbr.getByteBuffer() == null) {
+ return false;
+ }
+
+ return sbr.getByteBuffer().hasRemaining();
+ }
+
+ @Override
+ public CqUnit next() {
+ if (!hasNext()) {
+ return null;
+ }
+ long queueOffset = (sbr.getStartOffset() + sbr.getByteBuffer().position() - relativePos) / CQ_STORE_UNIT_SIZE;
+ CqUnit cqUnit = new CqUnit(queueOffset,
+ sbr.getByteBuffer().getLong(),
+ sbr.getByteBuffer().getInt(),
+ sbr.getByteBuffer().getLong());
+
+ if (isExtAddr(cqUnit.getTagsCode())) {
+ ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit();
+ boolean extRet = getExt(cqUnit.getTagsCode(), cqExtUnit);
+ if (extRet) {
+ cqUnit.setTagsCode(cqExtUnit.getTagsCode());
+ cqUnit.setCqExtUnit(cqExtUnit);
+ } else {
+ // can't find ext content.Client will filter messages by tag also.
+ log.error("[BUG] can't find consume queue extend file content! addr={}, offsetPy={}, sizePy={}, topic={}",
+ cqUnit.getTagsCode(), cqUnit.getPos(), cqUnit.getPos(), getTopic());
+ }
+ }
+ return cqUnit;
+ }
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException("remove");
+ }
+
+ @Override
+ public void release() {
+ if (sbr != null) {
+ sbr.release();
+ sbr = null;
+ }
+ }
+
+ @Override
+ public CqUnit nextAndRelease() {
+ try {
+ return next();
+ } finally {
+ release();
+ }
+ }
+ }
+
public ConsumeQueueExt.CqExtUnit getExt(final long offset) {
if (isExtReadEnable()) {
return this.consumeQueueExt.get(offset);
@@ -515,6 +649,7 @@ public class ConsumeQueue {
return false;
}
+ @Override
public long getMinLogicOffset() {
return minLogicOffset;
}
@@ -523,20 +658,29 @@ public class ConsumeQueue {
this.minLogicOffset = minLogicOffset;
}
+ @Override
public long rollNextFile(final long index) {
int mappedFileSize = this.mappedFileSize;
int totalUnitsInFile = mappedFileSize / CQ_STORE_UNIT_SIZE;
return index + totalUnitsInFile - index % totalUnitsInFile;
}
+ @Override
public String getTopic() {
return topic;
}
+ @Override
public int getQueueId() {
return queueId;
}
+ @Override
+ public CQType getCQType() {
+ return CQType.SimpleCQ;
+ }
+
+ @Override
public long getMaxPhysicOffset() {
return maxPhysicOffset;
}
@@ -545,6 +689,7 @@ public class ConsumeQueue {
this.maxPhysicOffset = maxPhysicOffset;
}
+ @Override
public void destroy() {
this.maxPhysicOffset = -1;
this.minLogicOffset = 0;
@@ -554,14 +699,17 @@ public class ConsumeQueue {
}
}
+ @Override
public long getMessageTotalInQueue() {
return this.getMaxOffsetInQueue() - this.getMinOffsetInQueue();
}
+ @Override
public long getMaxOffsetInQueue() {
return this.mappedFileQueue.getMaxOffset() / CQ_STORE_UNIT_SIZE;
}
+ @Override
public void checkSelf() {
mappedFileQueue.checkSelf();
if (isExtReadEnable()) {
@@ -584,4 +732,14 @@ public class ConsumeQueue {
public boolean isExtAddr(long tagsCode) {
return ConsumeQueueExt.isExtAddr(tagsCode);
}
+
+ @Override
+ public void swapMap(int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs) {
+ mappedFileQueue.swapMap(reserveNum, forceSwapIntervalMs, normalSwapIntervalMs);
+ }
+
+ @Override
+ public void cleanSwappedMap(long forceCleanSwapIntervalMs) {
+ mappedFileQueue.cleanSwappedMap(forceCleanSwapIntervalMs);
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueueExt.java b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueueExt.java
index 117a70b..19c7992 100644
--- a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueueExt.java
+++ b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueueExt.java
@@ -26,6 +26,7 @@ import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
+import org.apache.rocketmq.store.logfile.MappedFile;
/**
* Extend of consume queue, to store something not important,
diff --git a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
index 0061369..d25b9eb 100644
--- a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
@@ -51,30 +51,45 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.common.running.RunningStats;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
+import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.dledger.DLedgerCommitLog;
import org.apache.rocketmq.store.ha.HAService;
import org.apache.rocketmq.store.index.IndexService;
import org.apache.rocketmq.store.index.QueryOffsetResult;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
+import org.apache.rocketmq.store.logfile.MappedFile;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.ConsumeQueueStore;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.queue.ReferredIterator;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import org.apache.rocketmq.store.util.PerfCounter;
+
+import static java.lang.String.format;
public class DefaultMessageStore implements MessageStore {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
+ public final PerfCounter.Ticks perfs = new PerfCounter.Ticks(log);
+
private final MessageStoreConfig messageStoreConfig;
// CommitLog
private final CommitLog commitLog;
- private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueue>> consumeQueueTable;
+ private final ConsumeQueueStore consumeQueueStore;
+
+ private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
+
+ protected HashMap<String/* topic-queueid */, Long/* offset */> topicQueueTable = new HashMap<>(1024);
private final FlushConsumeQueueService flushConsumeQueueService;
@@ -124,6 +139,9 @@ public class DefaultMessageStore implements MessageStore {
private final List<CleanFilesHook> cleanFilesHooks = new CopyOnWriteArrayList<>();
+ // Max pull msg size
+ private final static int MAX_PULL_MSG_SIZE = 128 * 1024 * 1024;
+
public DefaultMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager,
final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException {
this.messageArrivingListener = messageArrivingListener;
@@ -137,6 +155,7 @@ public class DefaultMessageStore implements MessageStore {
this.commitLog = new CommitLog(this);
}
this.consumeQueueTable = new ConcurrentHashMap<>(32);
+ this.consumeQueueStore = new ConsumeQueueStore(this, this.messageStoreConfig, this.consumeQueueTable);
this.flushConsumeQueueService = new FlushConsumeQueueService();
this.cleanCommitLogService = new CleanCommitLogService();
@@ -167,18 +186,19 @@ public class DefaultMessageStore implements MessageStore {
this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex());
File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir()));
- MappedFile.ensureDirOK(file.getParent());
- MappedFile.ensureDirOK(getStorePathPhysic());
- MappedFile.ensureDirOK(getStorePathLogic());
+ DefaultMappedFile.ensureDirOK(file.getParent());
+ DefaultMappedFile.ensureDirOK(getStorePathPhysic());
+ DefaultMappedFile.ensureDirOK(getStorePathLogic());
lockFile = new RandomAccessFile(file, "rw");
}
+ @Override
public void truncateDirtyLogicFiles(long phyOffset) {
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueue>> tables = DefaultMessageStore.this.consumeQueueTable;
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
- for (ConcurrentMap<Integer, ConsumeQueue> maps : tables.values()) {
- for (ConsumeQueue logic : maps.values()) {
- logic.truncateDirtyLogicFiles(phyOffset);
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.truncateDirtyLogicFiles(logic, phyOffset);
}
}
}
@@ -186,12 +206,14 @@ public class DefaultMessageStore implements MessageStore {
/**
* @throws IOException
*/
+ @Override
public boolean load() {
boolean result = true;
try {
+ long start = System.currentTimeMillis();
boolean lastExitOK = !this.isTempFileExist();
- log.info("last shutdown {}", lastExitOK ? "normally" : "abnormally");
+ log.info("last shutdown {}, root dir: {}", lastExitOK ? "normally" : "abnormally", messageStoreConfig.getStorePathRootDir());
// load Commit Log
result = result && this.commitLog.load();
@@ -229,6 +251,7 @@ public class DefaultMessageStore implements MessageStore {
/**
* @throws Exception
*/
+ @Override
public void start() throws Exception {
lock = lockFile.getChannel().tryLock(0, 1, false);
@@ -246,8 +269,8 @@ public class DefaultMessageStore implements MessageStore {
* 4. Make sure the fall-behind messages to be dispatched before starting the commitlog, especially when the broker role are automatically changed.
*/
long maxPhysicalPosInLogicQueue = commitLog.getMinOffset();
- for (ConcurrentMap<Integer, ConsumeQueue> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueue logic : maps.values()) {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
if (logic.getMaxPhysicOffset() > maxPhysicalPosInLogicQueue) {
maxPhysicalPosInLogicQueue = logic.getMaxPhysicOffset();
}
@@ -298,9 +321,11 @@ public class DefaultMessageStore implements MessageStore {
this.createTempFile();
this.addScheduleTask();
+ this.perfs.start();
this.shutdown = false;
}
+ @Override
public void shutdown() {
if (!this.shutdown) {
this.shutdown = true;
@@ -330,6 +355,8 @@ public class DefaultMessageStore implements MessageStore {
this.storeCheckpoint.flush();
this.storeCheckpoint.shutdown();
+ this.perfs.shutdown();
+
if (this.runningFlags.isWriteable() && dispatchBehindBytes() == 0) {
this.deleteFile(StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
shutDownNormal = true;
@@ -349,6 +376,7 @@ public class DefaultMessageStore implements MessageStore {
}
}
+ @Override
public void destroy() {
this.destroyLogics();
this.commitLog.destroy();
@@ -357,10 +385,11 @@ public class DefaultMessageStore implements MessageStore {
this.deleteFile(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
}
+ @Override
public void destroyLogics() {
- for (ConcurrentMap<Integer, ConsumeQueue> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueue logic : maps.values()) {
- logic.destroy();
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.destroy(logic);
}
}
}
@@ -453,6 +482,7 @@ public class DefaultMessageStore implements MessageStore {
return putResultFuture;
}
+ @Override
public CompletableFuture<PutMessageResult> asyncPutMessages(MessageExtBatch messageExtBatch) {
PutMessageStatus checkStoreStatus = this.checkStoreStatus();
if (checkStoreStatus != PutMessageStatus.PUT_OK) {
@@ -515,16 +545,27 @@ public class DefaultMessageStore implements MessageStore {
return this.commitLog.lockTimeMills();
}
+ @Override
public SystemClock getSystemClock() {
return systemClock;
}
+ @Override
public CommitLog getCommitLog() {
return commitLog;
}
+ @Override
+ public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
+ final int maxMsgNums,
+ final MessageFilter messageFilter) {
+ return getMessage(group, topic, queueId, offset, maxMsgNums, MAX_PULL_MSG_SIZE, messageFilter);
+ }
+
+ @Override
public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
final int maxMsgNums,
+ final int maxTotalMsgSize,
final MessageFilter messageFilter) {
if (this.shutdown) {
log.warn("message store has shutdown, so getMessage is forbidden");
@@ -543,12 +584,11 @@ public class DefaultMessageStore implements MessageStore {
long minOffset = 0;
long maxOffset = 0;
- // lazy init when find msg.
- GetMessageResult getResult = null;
+ GetMessageResult getResult = new GetMessageResult();
final long maxOffsetPy = this.commitLog.getMaxOffset();
- ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface consumeQueue = findConsumeQueue(topic, queueId);
if (consumeQueue != null) {
minOffset = consumeQueue.getMinOffsetInQueue();
maxOffset = consumeQueue.getMaxOffsetInQueue();
@@ -570,55 +610,67 @@ public class DefaultMessageStore implements MessageStore {
nextBeginOffset = nextOffsetCorrection(offset, maxOffset);
}
} else {
- SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset);
- if (bufferConsumeQueue != null) {
- try {
- status = GetMessageStatus.NO_MATCHED_MESSAGE;
-
- long nextPhyFileStartOffset = Long.MIN_VALUE;
- long maxPhyOffsetPulling = 0;
-
- int i = 0;
- final int maxFilterMessageCount = Math.max(16000, maxMsgNums * ConsumeQueue.CQ_STORE_UNIT_SIZE);
- final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
+ final int maxFilterMessageCount = Math.max(16000, maxMsgNums * ConsumeQueue.CQ_STORE_UNIT_SIZE);
+ final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
- getResult = new GetMessageResult(maxMsgNums);
+ long maxPullSize = Math.max(maxTotalMsgSize, 100);
+ if (maxPullSize > MAX_PULL_MSG_SIZE) {
+ log.warn("The max pull size is too large maxPullSize={} topic={} queueId={}", maxPullSize, topic, queueId);
+ maxPullSize = MAX_PULL_MSG_SIZE;
+ }
+ status = GetMessageStatus.NO_MATCHED_MESSAGE;
+ long maxPhyOffsetPulling = 0;
+ int cqFileNum = 0;
+
+ while (getResult.getBufferTotalSize() <= 0
+ && nextBeginOffset < maxOffset
+ && cqFileNum++ < this.messageStoreConfig.getTravelCqFileNumWhenGetMessage()) {
+ ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(nextBeginOffset);
+
+ if (bufferConsumeQueue == null) {
+ status = GetMessageStatus.OFFSET_FOUND_NULL;
+ nextBeginOffset = nextOffsetCorrection(nextBeginOffset, this.consumeQueueStore.rollNextFile(consumeQueue, nextBeginOffset));
+ log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
+ + maxOffset + ", but access logic queue failed. Correct nextBeginOffset to " + nextBeginOffset);
+ break;
+ }
- ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit();
- for (; i < bufferConsumeQueue.getSize() && i < maxFilterMessageCount; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
- long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
- int sizePy = bufferConsumeQueue.getByteBuffer().getInt();
- long tagsCode = bufferConsumeQueue.getByteBuffer().getLong();
+ try {
+ long nextPhyFileStartOffset = Long.MIN_VALUE;
+ while (bufferConsumeQueue.hasNext()
+ && nextBeginOffset < maxOffset) {
+ CqUnit cqUnit = bufferConsumeQueue.next();
+ long offsetPy = cqUnit.getPos();
+ int sizePy = cqUnit.getSize();
- maxPhyOffsetPulling = offsetPy;
+ boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
- if (nextPhyFileStartOffset != Long.MIN_VALUE) {
- if (offsetPy < nextPhyFileStartOffset)
- continue;
+ if (cqUnit.getQueueOffset() - offset > maxFilterMessageCount) {
+ break;
}
- boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
+ if (this.isTheBatchFull(sizePy, maxMsgNums, maxPullSize, getResult.getBufferTotalSize(), getResult.getMessageCount(),
+ isInDisk)) {
+ break;
+ }
- if (this.isTheBatchFull(sizePy, maxMsgNums, getResult.getBufferTotalSize(), getResult.getMessageCount(),
- isInDisk)) {
+ if (getResult.getBufferTotalSize() >= maxPullSize) {
break;
}
- boolean extRet = false, isTagsCodeLegal = true;
- if (consumeQueue.isExtAddr(tagsCode)) {
- extRet = consumeQueue.getExt(tagsCode, cqExtUnit);
- if (extRet) {
- tagsCode = cqExtUnit.getTagsCode();
- } else {
- // can't find ext content.Client will filter messages by tag also.
- log.error("[BUG] can't find consume queue extend file content!addr={}, offsetPy={}, sizePy={}, topic={}, group={}",
- tagsCode, offsetPy, sizePy, topic, group);
- isTagsCodeLegal = false;
+ maxPhyOffsetPulling = offsetPy;
+
+ //Be careful, here should before the isTheBatchFull
+ nextBeginOffset = cqUnit.getQueueOffset() + cqUnit.getBatchNum();
+
+ if (nextPhyFileStartOffset != Long.MIN_VALUE) {
+ if (offsetPy < nextPhyFileStartOffset) {
+ continue;
}
}
if (messageFilter != null
- && !messageFilter.isMatchedByConsumeQueue(isTagsCodeLegal ? tagsCode : null, extRet ? cqExtUnit : null)) {
+ && !messageFilter.isMatchedByConsumeQueue(cqUnit.getValidTagsCodeAsLong(), cqUnit.getCqExtUnit())) {
if (getResult.getBufferTotalSize() == 0) {
status = GetMessageStatus.NO_MATCHED_MESSAGE;
}
@@ -637,7 +689,7 @@ public class DefaultMessageStore implements MessageStore {
}
if (messageFilter != null
- && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
+ && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
if (getResult.getBufferTotalSize() == 0) {
status = GetMessageStatus.NO_MATCHED_MESSAGE;
}
@@ -647,32 +699,24 @@ public class DefaultMessageStore implements MessageStore {
}
this.storeStatsService.getGetMessageTransferedMsgCount().add(1);
- getResult.addMessage(selectResult, offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE));
+ getResult.addMessage(selectResult, cqUnit.getQueueOffset(), cqUnit.getBatchNum());
status = GetMessageStatus.FOUND;
nextPhyFileStartOffset = Long.MIN_VALUE;
}
-
- if (diskFallRecorded) {
- long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
- brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
- }
-
- nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
-
- long diff = maxOffsetPy - maxPhyOffsetPulling;
- long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE
- * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
- getResult.setSuggestPullingFromSlave(diff > memory);
} finally {
-
bufferConsumeQueue.release();
}
- } else {
- status = GetMessageStatus.OFFSET_FOUND_NULL;
- nextBeginOffset = nextOffsetCorrection(offset, consumeQueue.rollNextFile(offset));
- log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
- + maxOffset + ", but access logic queue failed.");
}
+
+ if (diskFallRecorded) {
+ long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
+ brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
+ }
+
+ long diff = maxOffsetPy - maxPhyOffsetPulling;
+ long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE
+ * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
+ getResult.setSuggestPullingFromSlave(diff > memory);
}
} else {
status = GetMessageStatus.NO_MATCHED_LOGIC_QUEUE;
@@ -699,18 +743,20 @@ public class DefaultMessageStore implements MessageStore {
return getResult;
}
+ @Override
public long getMaxOffsetInQueue(String topic, int queueId) {
return getMaxOffsetInQueue(topic, queueId, true);
}
+ @Override
public long getMaxOffsetInQueue(String topic, int queueId, boolean committed) {
if (committed) {
- ConsumeQueue logic = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
return logic.getMaxOffsetInQueue();
}
} else {
- Long offset = this.commitLog.getTopicQueueTable().get(topic + "-" + queueId);
+ Long offset = this.topicQueueTable.get(topic + "-" + queueId);
if (offset != null) {
return offset;
}
@@ -719,8 +765,9 @@ public class DefaultMessageStore implements MessageStore {
return 0;
}
+ @Override
public long getMinOffsetInQueue(String topic, int queueId) {
- ConsumeQueue logic = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
return logic.getMinOffsetInQueue();
}
@@ -730,13 +777,16 @@ public class DefaultMessageStore implements MessageStore {
@Override
public long getCommitLogOffsetInQueue(String topic, int queueId, long consumeQueueOffset) {
- ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface consumeQueue = findConsumeQueue(topic, queueId);
if (consumeQueue != null) {
- SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(consumeQueueOffset);
+
+ ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(consumeQueueOffset);
if (bufferConsumeQueue != null) {
try {
- long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
- return offsetPy;
+ if (bufferConsumeQueue.hasNext()) {
+ long offsetPy = bufferConsumeQueue.next().getPos();
+ return offsetPy;
+ }
} finally {
bufferConsumeQueue.release();
}
@@ -746,8 +796,9 @@ public class DefaultMessageStore implements MessageStore {
return 0;
}
+ @Override
public long getOffsetInQueueByTime(String topic, int queueId, long timestamp) {
- ConsumeQueue logic = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
return logic.getOffsetInQueueByTime(timestamp);
}
@@ -755,6 +806,7 @@ public class DefaultMessageStore implements MessageStore {
return 0;
}
+ @Override
public MessageExt lookMessageByOffset(long commitLogOffset) {
SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, 4);
if (null != sbr) {
@@ -791,6 +843,7 @@ public class DefaultMessageStore implements MessageStore {
return this.commitLog.getMessage(commitLogOffset, msgSize);
}
+ @Override
public String getRunningDataInfo() {
return this.storeStatsService.toString();
}
@@ -855,27 +908,22 @@ public class DefaultMessageStore implements MessageStore {
@Override
public long getEarliestMessageTime(String topic, int queueId) {
- ConsumeQueue logicQueue = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logicQueue = this.findConsumeQueue(topic, queueId);
if (logicQueue != null) {
- long minLogicOffset = logicQueue.getMinLogicOffset();
-
- SelectMappedBufferResult result = logicQueue.getIndexBuffer(minLogicOffset / ConsumeQueue.CQ_STORE_UNIT_SIZE);
- return getStoreTime(result);
+ return getStoreTime(logicQueue.getEarliestUnit());
}
return -1;
}
- private long getStoreTime(SelectMappedBufferResult result) {
+ protected long getStoreTime(CqUnit result) {
if (result != null) {
try {
- final long phyOffset = result.getByteBuffer().getLong();
- final int size = result.getByteBuffer().getInt();
+ final long phyOffset = result.getPos();
+ final int size = result.getSize();
long storeTime = this.getCommitLog().pickupStoreTimestamp(phyOffset, size);
return storeTime;
} catch (Exception e) {
- } finally {
- result.release();
}
}
return -1;
@@ -890,10 +938,9 @@ public class DefaultMessageStore implements MessageStore {
@Override
public long getMessageStoreTimeStamp(String topic, int queueId, long consumeQueueOffset) {
- ConsumeQueue logicQueue = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logicQueue = this.findConsumeQueue(topic, queueId);
if (logicQueue != null) {
- SelectMappedBufferResult result = logicQueue.getIndexBuffer(consumeQueueOffset);
- return getStoreTime(result);
+ return getStoreTime(logicQueue.get(consumeQueueOffset));
}
return -1;
@@ -901,7 +948,7 @@ public class DefaultMessageStore implements MessageStore {
@Override
public long getMessageTotalInQueue(String topic, int queueId) {
- ConsumeQueue logicQueue = this.findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface logicQueue = this.findConsumeQueue(topic, queueId);
if (logicQueue != null) {
return logicQueue.getMessageTotalInQueue();
}
@@ -1024,16 +1071,16 @@ public class DefaultMessageStore implements MessageStore {
@Override
public int cleanUnusedTopic(Set<String> topics) {
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueue>>> it = this.consumeQueueTable.entrySet().iterator();
+ Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueue>> next = it.next();
+ Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
String topic = next.getKey();
if (!topics.contains(topic) && !topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)
&& !topic.equals(TopicValidator.RMQ_SYS_TRANS_OP_HALF_TOPIC)) {
- ConcurrentMap<Integer, ConsumeQueue> queueTable = next.getValue();
- for (ConsumeQueue cq : queueTable.values()) {
- cq.destroy();
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
+ for (ConsumeQueueInterface cq : queueTable.values()) {
+ this.consumeQueueStore.destroy(cq);
log.info("cleanUnusedTopic: {} {} ConsumeQueue cleaned",
cq.getTopic(),
cq.getQueueId()
@@ -1054,18 +1101,19 @@ public class DefaultMessageStore implements MessageStore {
return 0;
}
+ @Override
public void cleanExpiredConsumerQueue() {
long minCommitLogOffset = this.commitLog.getMinOffset();
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueue>>> it = this.consumeQueueTable.entrySet().iterator();
+ Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueue>> next = it.next();
+ Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
String topic = next.getKey();
if (!topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
- ConcurrentMap<Integer, ConsumeQueue> queueTable = next.getValue();
- Iterator<Entry<Integer, ConsumeQueue>> itQT = queueTable.entrySet().iterator();
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
+ Iterator<Entry<Integer, ConsumeQueueInterface>> itQT = queueTable.entrySet().iterator();
while (itQT.hasNext()) {
- Entry<Integer, ConsumeQueue> nextQT = itQT.next();
+ Entry<Integer, ConsumeQueueInterface> nextQT = itQT.next();
long maxCLOffsetInConsumeQueue = nextQT.getValue().getLastOffset();
if (maxCLOffsetInConsumeQueue == -1) {
@@ -1085,7 +1133,7 @@ public class DefaultMessageStore implements MessageStore {
DefaultMessageStore.this.commitLog.removeQueueFromTopicQueueTable(nextQT.getValue().getTopic(),
nextQT.getValue().getQueueId());
- nextQT.getValue().destroy();
+ this.consumeQueueStore.destroy(nextQT.getValue());
itQT.remove();
}
}
@@ -1105,7 +1153,7 @@ public class DefaultMessageStore implements MessageStore {
return messageIds;
}
- ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface consumeQueue = findConsumeQueue(topic, queueId);
if (consumeQueue != null) {
minOffset = Math.max(minOffset, consumeQueue.getMinOffsetInQueue());
maxOffset = Math.min(maxOffset, consumeQueue.getMaxOffsetInQueue());
@@ -1116,28 +1164,30 @@ public class DefaultMessageStore implements MessageStore {
long nextOffset = minOffset;
while (nextOffset < maxOffset) {
- SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(nextOffset);
- if (bufferConsumeQueue != null) {
- try {
- int i = 0;
- for (; i < bufferConsumeQueue.getSize(); i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
- long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
+ ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(nextOffset);
+ try {
+ if (bufferConsumeQueue != null && bufferConsumeQueue.hasNext()) {
+ while (bufferConsumeQueue.hasNext()) {
+ CqUnit cqUnit = bufferConsumeQueue.next();
+ long offsetPy = cqUnit.getPos();
InetSocketAddress inetSocketAddress = (InetSocketAddress) storeHost;
int msgIdLength = (inetSocketAddress.getAddress() instanceof Inet6Address) ? 16 + 4 + 8 : 4 + 4 + 8;
final ByteBuffer msgIdMemory = ByteBuffer.allocate(msgIdLength);
String msgId =
- MessageDecoder.createMessageId(msgIdMemory, MessageExt.socketAddress2ByteBuffer(storeHost), offsetPy);
- messageIds.put(msgId, nextOffset++);
- if (nextOffset > maxOffset) {
+ MessageDecoder.createMessageId(msgIdMemory, MessageExt.socketAddress2ByteBuffer(storeHost), offsetPy);
+ messageIds.put(msgId, cqUnit.getQueueOffset());
+ nextOffset = cqUnit.getQueueOffset() + cqUnit.getBatchNum();
+ if (nextOffset >= maxOffset) {
return messageIds;
}
}
- } finally {
-
+ } else {
+ return messageIds;
+ }
+ } finally {
+ if (bufferConsumeQueue != null) {
bufferConsumeQueue.release();
}
- } else {
- return messageIds;
}
}
}
@@ -1149,20 +1199,13 @@ public class DefaultMessageStore implements MessageStore {
final long maxOffsetPy = this.commitLog.getMaxOffset();
- ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
+ ConsumeQueueInterface consumeQueue = findConsumeQueue(topic, queueId);
if (consumeQueue != null) {
- SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(consumeOffset);
- if (bufferConsumeQueue != null) {
- try {
- for (int i = 0; i < bufferConsumeQueue.getSize(); ) {
- i += ConsumeQueue.CQ_STORE_UNIT_SIZE;
- long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
- return checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
- }
- } finally {
+ CqUnit cqUnit = consumeQueue.get(consumeOffset);
- bufferConsumeQueue.release();
- }
+ if (cqUnit != null) {
+ long offsetPy = cqUnit.getPos();
+ return checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
} else {
return false;
}
@@ -1195,6 +1238,7 @@ public class DefaultMessageStore implements MessageStore {
this.commitLog.setConfirmOffset(phyOffset);
}
+ @Override
public MessageExt lookMessageByOffset(long commitLogOffset, int size) {
SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, size);
if (null != sbr) {
@@ -1208,35 +1252,8 @@ public class DefaultMessageStore implements MessageStore {
return null;
}
- public ConsumeQueue findConsumeQueue(String topic, int queueId) {
- ConcurrentMap<Integer, ConsumeQueue> map = consumeQueueTable.get(topic);
- if (null == map) {
- ConcurrentMap<Integer, ConsumeQueue> newMap = new ConcurrentHashMap<Integer, ConsumeQueue>(128);
- ConcurrentMap<Integer, ConsumeQueue> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
- if (oldMap != null) {
- map = oldMap;
- } else {
- map = newMap;
- }
- }
-
- ConsumeQueue logic = map.get(queueId);
- if (null == logic) {
- ConsumeQueue newLogic = new ConsumeQueue(
- topic,
- queueId,
- StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
- this.getMessageStoreConfig().getMappedFileSizeConsumeQueue(),
- this);
- ConsumeQueue oldLogic = map.putIfAbsent(queueId, newLogic);
- if (oldLogic != null) {
- logic = oldLogic;
- } else {
- logic = newLogic;
- }
- }
-
- return logic;
+ public ConsumeQueueInterface findConsumeQueue(String topic, int queueId) {
+ return this.consumeQueueStore.findOrCreateConsumeQueue(topic, queueId);
}
private long nextOffsetCorrection(long oldOffset, long newOffset) {
@@ -1252,7 +1269,7 @@ public class DefaultMessageStore implements MessageStore {
return (maxOffsetPy - offsetPy) > memory;
}
- private boolean isTheBatchFull(int sizePy, int maxMsgNums, int bufferTotal, int messageTotal, boolean isInDisk) {
+ private boolean isTheBatchFull(int sizePy, int maxMsgNums, long maxMsgSize, int bufferTotal, int messageTotal, boolean isInDisk) {
if (0 == bufferTotal || 0 == messageTotal) {
return false;
@@ -1262,6 +1279,10 @@ public class DefaultMessageStore implements MessageStore {
return true;
}
+ if (bufferTotal + sizePy > maxMsgSize) {
+ return true;
+ }
+
if (isInDisk) {
if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInDisk()) {
return true;
@@ -1295,11 +1316,12 @@ public class DefaultMessageStore implements MessageStore {
private void createTempFile() throws IOException {
String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
File file = new File(fileName);
- MappedFile.ensureDirOK(file.getParent());
+ DefaultMappedFile.ensureDirOK(file.getParent());
boolean result = file.createNewFile();
log.info(fileName + (result ? " create OK" : " already exists"));
}
+ @Override
public void registerCleanFileHook(CleanFilesHook hook) {
this.cleanFilesHooks.add(hook);
}
@@ -1371,13 +1393,13 @@ public class DefaultMessageStore implements MessageStore {
private void checkSelf() {
this.commitLog.checkSelf();
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueue>>> it = this.consumeQueueTable.entrySet().iterator();
+ Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueue>> next = it.next();
- Iterator<Entry<Integer, ConsumeQueue>> itNext = next.getValue().entrySet().iterator();
+ Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ Iterator<Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
while (itNext.hasNext()) {
- Entry<Integer, ConsumeQueue> cq = itNext.next();
- cq.getValue().checkSelf();
+ Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
+ this.consumeQueueStore.checkSelf(cq.getValue());
}
}
}
@@ -1389,6 +1411,8 @@ public class DefaultMessageStore implements MessageStore {
}
private boolean loadConsumeQueue() {
+ checkOtherConsumeQueue();
+
File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
File[] fileTopicList = dirLogic.listFiles();
if (fileTopicList != null) {
@@ -1405,14 +1429,14 @@ public class DefaultMessageStore implements MessageStore {
} catch (NumberFormatException e) {
continue;
}
- ConsumeQueue logic = new ConsumeQueue(
+ ConsumeQueueInterface logic = new ConsumeQueue(
topic,
queueId,
StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
this.getMessageStoreConfig().getMappedFileSizeConsumeQueue(),
this);
this.putConsumeQueue(topic, queueId, logic);
- if (!logic.load()) {
+ if (!this.consumeQueueStore.load(logic)) {
return false;
}
}
@@ -1425,30 +1449,46 @@ public class DefaultMessageStore implements MessageStore {
return true;
}
+ private void checkOtherConsumeQueue() {
+ File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ if (dirLogic.exists()) {
+ throw new RuntimeException(format("Batch consume queue directory: [%s] exist. Can not load consume queue while batch consume queue exists.",
+ StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir())));
+ }
+ }
+
private void recover(final boolean lastExitOK) {
+ long recoverCqStart = System.currentTimeMillis();
long maxPhyOffsetOfConsumeQueue = this.recoverConsumeQueue();
+ long recoverCqEnd = System.currentTimeMillis();
if (lastExitOK) {
this.commitLog.recoverNormally(maxPhyOffsetOfConsumeQueue);
} else {
this.commitLog.recoverAbnormally(maxPhyOffsetOfConsumeQueue);
}
-
+ long recoverClogEnd = System.currentTimeMillis();
this.recoverTopicQueueTable();
+ long recoverOffsetEnd = System.currentTimeMillis();
+
+ log.info("Recover end total:{} recoverCq:{} recoverClog:{} recoverOffset:{}",
+ recoverOffsetEnd - recoverCqStart, recoverCqEnd - recoverCqStart, recoverClogEnd - recoverCqEnd, recoverOffsetEnd - recoverClogEnd);
}
+ @Override
public MessageStoreConfig getMessageStoreConfig() {
return messageStoreConfig;
}
+ @Override
public TransientStorePool getTransientStorePool() {
return transientStorePool;
}
- private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueue consumeQueue) {
- ConcurrentMap<Integer/* queueId */, ConsumeQueue> map = this.consumeQueueTable.get(topic);
+ private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueueInterface consumeQueue) {
+ ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface> map = this.consumeQueueTable.get(topic);
if (null == map) {
- map = new ConcurrentHashMap<Integer/* queueId */, ConsumeQueue>();
+ map = new ConcurrentHashMap<Integer/* queueId */, ConsumeQueueInterface>();
map.put(queueId, consumeQueue);
this.consumeQueueTable.put(topic, map);
} else {
@@ -1458,9 +1498,9 @@ public class DefaultMessageStore implements MessageStore {
private long recoverConsumeQueue() {
long maxPhysicOffset = -1;
- for (ConcurrentMap<Integer, ConsumeQueue> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueue logic : maps.values()) {
- logic.recover();
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.recover(logic);
if (logic.getMaxPhysicOffset() > maxPhysicOffset) {
maxPhysicOffset = logic.getMaxPhysicOffset();
}
@@ -1473,21 +1513,23 @@ public class DefaultMessageStore implements MessageStore {
public void recoverTopicQueueTable() {
HashMap<String/* topic-queueid */, Long/* offset */> table = new HashMap<String, Long>(1024);
long minPhyOffset = this.commitLog.getMinOffset();
- for (ConcurrentMap<Integer, ConsumeQueue> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueue logic : maps.values()) {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
String key = logic.getTopic() + "-" + logic.getQueueId();
table.put(key, logic.getMaxOffsetInQueue());
- logic.correctMinOffset(minPhyOffset);
+ this.consumeQueueStore.correctMinOffset(logic, minPhyOffset);
}
}
- this.commitLog.setTopicQueueTable(table);
+ this.topicQueueTable = table;
}
+ @Override
public AllocateMappedFileService getAllocateMappedFileService() {
return allocateMappedFileService;
}
+ @Override
public StoreStatsService getStoreStatsService() {
return storeStatsService;
}
@@ -1496,14 +1538,16 @@ public class DefaultMessageStore implements MessageStore {
return runningFlags;
}
- public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueue>> getConsumeQueueTable() {
+ public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> getConsumeQueueTable() {
return consumeQueueTable;
}
+ @Override
public StoreCheckpoint getStoreCheckpoint() {
return storeCheckpoint;
}
+ @Override
public HAService getHaService() {
return haService;
}
@@ -1513,6 +1557,7 @@ public class DefaultMessageStore implements MessageStore {
return scheduleMessageService;
}
+ @Override
public RunningFlags getRunningFlags() {
return runningFlags;
}
@@ -1524,8 +1569,8 @@ public class DefaultMessageStore implements MessageStore {
}
public void putMessagePositionInfo(DispatchRequest dispatchRequest) {
- ConsumeQueue cq = this.findConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
- cq.putMessagePositionInfoWrapper(dispatchRequest);
+ ConsumeQueueInterface cq = this.findConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
+ this.consumeQueueStore.putMessagePositionInfoWrapper(cq, dispatchRequest);
}
@Override
@@ -1560,14 +1605,15 @@ public class DefaultMessageStore implements MessageStore {
}
@Override
- public ConsumeQueue getConsumeQueue(String topic, int queueId) {
- ConcurrentMap<Integer, ConsumeQueue> map = consumeQueueTable.get(topic);
+ public ConsumeQueueInterface getConsumeQueue(String topic, int queueId) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
if (map == null) {
return null;
}
return map.get(queueId);
}
+ @Override
public void unlockMappedFile(final MappedFile mappedFile) {
this.scheduledExecutorService.schedule(new Runnable() {
@Override
@@ -1577,6 +1623,54 @@ public class DefaultMessageStore implements MessageStore {
}, 6, TimeUnit.SECONDS);
}
+ @Override
+ public PerfCounter.Ticks getPerfCounter() {
+ return perfs;
+ }
+
+ @Override
+ public ConsumeQueueStore getQueueStore() {
+ return consumeQueueStore;
+ }
+
+ @Override
+ public void onCommitLogAppend(MessageExtBrokerInner msg, AppendMessageResult result, MappedFile commitLogFile) {
+ // empty
+ }
+
+ @Override
+ public void onCommitLogDispatch(DispatchRequest dispatchRequest, boolean doDispatch, MappedFile commitLogFile, boolean isRecover, boolean isFileEnd) {
+ if (doDispatch && !isFileEnd) {
+ this.doDispatch(dispatchRequest);
+ }
+ }
+
+ @Override
+ public boolean isSyncDiskFlush() {
+ return FlushDiskType.SYNC_FLUSH == this.getMessageStoreConfig().getFlushDiskType();
+ }
+
+ @Override
+ public boolean isSyncMaster() {
+ return BrokerRole.SYNC_MASTER == this.getMessageStoreConfig().getBrokerRole();
+ }
+
+ @Override
+ public void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
+ final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
+
+ if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
+ long topicOffset = this.topicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ msg.setQueueOffset(topicOffset);
+ this.topicQueueTable.put(topicQueueKey, topicOffset + batchNum);
+ }
+ }
+
+ @Override
+ public void removeOffsetTable(String topicQueueKey) {
+ this.topicQueueTable.remove(topicQueueKey);
+ }
+
class CommitLogDispatcherBuildConsumeQueue implements CommitLogDispatcher {
@Override
@@ -1607,17 +1701,53 @@ public class DefaultMessageStore implements MessageStore {
class CleanCommitLogService {
private final static int MAX_MANUAL_DELETE_FILE_TIMES = 20;
- private final double diskSpaceWarningLevelRatio =
- Double.parseDouble(System.getProperty("rocketmq.broker.diskSpaceWarningLevelRatio", "0.90"));
+ private final String diskSpaceWarningLevelRatio =
+ System.getProperty("rocketmq.broker.diskSpaceWarningLevelRatio", "");
- private final double diskSpaceCleanForciblyRatio =
- Double.parseDouble(System.getProperty("rocketmq.broker.diskSpaceCleanForciblyRatio", "0.85"));
+ private final String diskSpaceCleanForciblyRatio =
+ System.getProperty("rocketmq.broker.diskSpaceCleanForciblyRatio", "");
private long lastRedeleteTimestamp = 0;
private volatile int manualDeleteFileSeveralTimes = 0;
private volatile boolean cleanImmediately = false;
+ double getDiskSpaceWarningLevelRatio() {
+ double finalDiskSpaceWarningLevelRatio;
+ if ("".equals(diskSpaceWarningLevelRatio)) {
+ finalDiskSpaceWarningLevelRatio = DefaultMessageStore.this.getMessageStoreConfig().getDiskSpaceWarningLevelRatio() / 100.0;
+ } else {
+ finalDiskSpaceWarningLevelRatio = Double.parseDouble(diskSpaceWarningLevelRatio);
+ }
+
+ if (finalDiskSpaceWarningLevelRatio > 0.90) {
+ finalDiskSpaceWarningLevelRatio = 0.90;
+ }
+ if (finalDiskSpaceWarningLevelRatio < 0.35) {
+ finalDiskSpaceWarningLevelRatio = 0.35;
+ }
+
+ return finalDiskSpaceWarningLevelRatio;
+ }
+
+ double getDiskSpaceCleanForciblyRatio() {
+ double finalDiskSpaceCleanForciblyRatio;
+ if ("".equals(diskSpaceCleanForciblyRatio)) {
+ finalDiskSpaceCleanForciblyRatio = DefaultMessageStore.this.getMessageStoreConfig().getDiskSpaceCleanForciblyRatio() / 100.0;
+ } else {
+ finalDiskSpaceCleanForciblyRatio = Double.parseDouble(diskSpaceCleanForciblyRatio);
+ }
+
+ if (finalDiskSpaceCleanForciblyRatio > 0.85) {
+ finalDiskSpaceCleanForciblyRatio = 0.85;
+ }
+ if (finalDiskSpaceCleanForciblyRatio < 0.30) {
+ finalDiskSpaceCleanForciblyRatio = 0.30;
+ }
+
+ return finalDiskSpaceCleanForciblyRatio;
+ }
+
public void excuteDeleteFilesManualy() {
this.manualDeleteFileSeveralTimes = MAX_MANUAL_DELETE_FILE_TIMES;
DefaultMessageStore.log.info("executeDeleteFilesManually was invoked");
@@ -1714,12 +1844,12 @@ public class DefaultMessageStore implements MessageStore {
minPhysicRatio = physicRatio;
minStorePath = storePathPhysic;
}
- if (physicRatio > diskSpaceCleanForciblyRatio) {
+ if (physicRatio > getDiskSpaceCleanForciblyRatio()) {
fullStorePath.add(storePathPhysic);
}
}
DefaultMessageStore.this.commitLog.setFullStorePaths(fullStorePath);
- if (minPhysicRatio > diskSpaceWarningLevelRatio) {
+ if (minPhysicRatio > getDiskSpaceWarningLevelRatio()) {
boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
if (diskok) {
DefaultMessageStore.log.error("physic disk maybe full soon " + minPhysicRatio +
@@ -1727,7 +1857,7 @@ public class DefaultMessageStore implements MessageStore {
}
cleanImmediately = true;
- } else if (minPhysicRatio > diskSpaceCleanForciblyRatio) {
+ } else if (minPhysicRatio > getDiskSpaceCleanForciblyRatio()) {
cleanImmediately = true;
} else {
boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskOK();
@@ -1748,14 +1878,14 @@ public class DefaultMessageStore implements MessageStore {
String storePathLogics = StorePathConfigHelper
.getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir());
double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
- if (logicsRatio > diskSpaceWarningLevelRatio) {
+ if (logicsRatio > getDiskSpaceWarningLevelRatio()) {
boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
if (diskok) {
DefaultMessageStore.log.error("logics disk maybe full soon " + logicsRatio + ", so mark disk full");
}
cleanImmediately = true;
- } else if (logicsRatio > diskSpaceCleanForciblyRatio) {
+ } else if (logicsRatio > getDiskSpaceCleanForciblyRatio()) {
cleanImmediately = true;
} else {
boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskOK();
@@ -1790,7 +1920,7 @@ public class DefaultMessageStore implements MessageStore {
double physicRatio = UtilAll.isPathExists(path) ?
UtilAll.getDiskPartitionSpaceUsedPercent(path) : -1;
minPhysicRatio = Math.min(minPhysicRatio, physicRatio);
- if (physicRatio > diskSpaceCleanForciblyRatio) {
+ if (physicRatio > getDiskSpaceCleanForciblyRatio()) {
fullStorePath.add(path);
}
}
@@ -1805,7 +1935,7 @@ public class DefaultMessageStore implements MessageStore {
if (physicRatio > ratio) {
DefaultMessageStore.log.info("physic disk of commitLog used: " + physicRatio);
}
- if (physicRatio > this.diskSpaceWarningLevelRatio) {
+ if (physicRatio > this.getDiskSpaceWarningLevelRatio()) {
boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
if (diskok) {
DefaultMessageStore.log.error("physic disk of commitLog maybe full soon, used " + physicRatio + ", so mark disk full");
@@ -1845,11 +1975,11 @@ public class DefaultMessageStore implements MessageStore {
if (minOffset > this.lastPhysicalMinOffset) {
this.lastPhysicalMinOffset = minOffset;
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueue>> tables = DefaultMessageStore.this.consumeQueueTable;
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
- for (ConcurrentMap<Integer, ConsumeQueue> maps : tables.values()) {
- for (ConsumeQueue logic : maps.values()) {
- int deleteCount = logic.deleteExpiredFile(minOffset);
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ int deleteCount = DefaultMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minOffset);
deleteCountSum += deleteCount;
if (deleteCount > 0 && deleteLogicsFilesInterval > 0) {
try {
@@ -1891,13 +2021,13 @@ public class DefaultMessageStore implements MessageStore {
logicsMsgTimestamp = DefaultMessageStore.this.getStoreCheckpoint().getLogicsMsgTimestamp();
}
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueue>> tables = DefaultMessageStore.this.consumeQueueTable;
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
- for (ConcurrentMap<Integer, ConsumeQueue> maps : tables.values()) {
- for (ConsumeQueue cq : maps.values()) {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface cq : maps.values()) {
boolean result = false;
for (int i = 0; i < retryTimes && !result; i++) {
- result = cq.flush(flushConsumeQueueLeastPages);
+ result = DefaultMessageStore.this.consumeQueueStore.flush(cq, flushConsumeQueueLeastPages);
}
}
}
@@ -1910,6 +2040,7 @@ public class DefaultMessageStore implements MessageStore {
}
}
+ @Override
public void run() {
DefaultMessageStore.log.info(this.getServiceName() + " service started");
diff --git a/store/src/main/java/org/apache/rocketmq/store/DispatchRequest.java b/store/src/main/java/org/apache/rocketmq/store/DispatchRequest.java
index 89d47ce..bfe54ba 100644
--- a/store/src/main/java/org/apache/rocketmq/store/DispatchRequest.java
+++ b/store/src/main/java/org/apache/rocketmq/store/DispatchRequest.java
@@ -37,6 +37,12 @@ public class DispatchRequest {
private int bufferSize = -1;//the buffer size maybe larger than the msg size if the message is wrapped by something
+ // for batch consume queue
+ private long msgBaseOffset = -1;
+ private short batchSize = 1;
+
+ private long nextReputFromOffset = -1;
+
public DispatchRequest(
final String topic,
final int queueId,
@@ -58,6 +64,7 @@ public class DispatchRequest {
this.tagsCode = tagsCode;
this.storeTimestamp = storeTimestamp;
this.consumeQueueOffset = consumeQueueOffset;
+ this.msgBaseOffset = consumeQueueOffset;
this.keys = keys;
this.uniqKey = uniqKey;
@@ -159,10 +166,26 @@ public class DispatchRequest {
this.bitMap = bitMap;
}
+ public short getBatchSize() {
+ return batchSize;
+ }
+
+ public void setBatchSize(short batchSize) {
+ this.batchSize = batchSize;
+ }
+
public void setMsgSize(int msgSize) {
this.msgSize = msgSize;
}
+ public long getMsgBaseOffset() {
+ return msgBaseOffset;
+ }
+
+ public void setMsgBaseOffset(long msgBaseOffset) {
+ this.msgBaseOffset = msgBaseOffset;
+ }
+
public int getBufferSize() {
return bufferSize;
}
@@ -170,4 +193,26 @@ public class DispatchRequest {
public void setBufferSize(int bufferSize) {
this.bufferSize = bufferSize;
}
+
+ public long getNextReputFromOffset() {
+ return nextReputFromOffset;
+ }
+
+ public void setNextReputFromOffset(long nextReputFromOffset) {
+ this.nextReputFromOffset = nextReputFromOffset;
+ }
+
+ @Override
+ public String toString() {
+ return "DispatchRequest{" +
+ "topic='" + topic + '\'' +
+ ", queueId=" + queueId +
+ ", commitLogOffset=" + commitLogOffset +
+ ", msgSize=" + msgSize +
+ ", success=" + success +
+ ", msgBaseOffset=" + msgBaseOffset +
+ ", batchSize=" + batchSize +
+ ", nextReputFromOffset=" + nextReputFromOffset +
+ '}';
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/FileQueueSnapshot.java b/store/src/main/java/org/apache/rocketmq/store/FileQueueSnapshot.java
new file mode 100644
index 0000000..6521a76
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/FileQueueSnapshot.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.store;
+
+import org.apache.rocketmq.store.logfile.MappedFile;
+
+public class FileQueueSnapshot {
+ private MappedFile firstFile;
+ private long firstFileIndex;
+ private MappedFile lastFile;
+ private long lastFileIndex;
+ private long currentFile;
+ private long currentFileIndex;
+ private long behindCount;
+ private boolean exist;
+
+ public FileQueueSnapshot() {
+ }
+
+ public FileQueueSnapshot(MappedFile firstFile, long firstFileIndex, MappedFile lastFile, long lastFileIndex, long currentFile, long currentFileIndex, long behindCount, boolean exist) {
+ this.firstFile = firstFile;
+ this.firstFileIndex = firstFileIndex;
+ this.lastFile = lastFile;
+ this.lastFileIndex = lastFileIndex;
+ this.currentFile = currentFile;
+ this.currentFileIndex = currentFileIndex;
+ this.behindCount = behindCount;
+ this.exist = exist;
+ }
+
+ public MappedFile getFirstFile() {
+ return firstFile;
+ }
+
+ public long getFirstFileIndex() {
+ return firstFileIndex;
+ }
+
+ public MappedFile getLastFile() {
+ return lastFile;
+ }
+
+ public long getLastFileIndex() {
+ return lastFileIndex;
+ }
+
+ public long getCurrentFile() {
+ return currentFile;
+ }
+
+ public long getCurrentFileIndex() {
+ return currentFileIndex;
+ }
+
+ public long getBehindCount() {
+ return behindCount;
+ }
+
+ public boolean isExist() {
+ return exist;
+ }
+
+ @Override
+ public String toString() {
+ return "FileQueueSnapshot{" +
+ "firstFile=" + firstFile +
+ ", firstFileIndex=" + firstFileIndex +
+ ", lastFile=" + lastFile +
+ ", lastFileIndex=" + lastFileIndex +
+ ", currentFile=" + currentFile +
+ ", currentFileIndex=" + currentFileIndex +
+ ", behindCount=" + behindCount +
+ ", exist=" + exist +
+ '}';
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/GetMessageResult.java b/store/src/main/java/org/apache/rocketmq/store/GetMessageResult.java
index fafff6b..b027914 100644
--- a/store/src/main/java/org/apache/rocketmq/store/GetMessageResult.java
+++ b/store/src/main/java/org/apache/rocketmq/store/GetMessageResult.java
@@ -34,6 +34,8 @@ public class GetMessageResult {
private int bufferTotalSize = 0;
+ private int messageCount = 0;
+
private boolean suggestPullingFromSlave = false;
private int msgCount4Commercial = 0;
@@ -99,10 +101,20 @@ public class GetMessageResult {
}
public void addMessage(final SelectMappedBufferResult mapedBuffer, final long queueOffset) {
- addMessage(mapedBuffer);
+ this.messageMapedList.add(mapedBuffer);
+ this.messageBufferList.add(mapedBuffer.getByteBuffer());
+ this.bufferTotalSize += mapedBuffer.getSize();
+ this.msgCount4Commercial += (int) Math.ceil(
+ mapedBuffer.getSize() / BrokerStatsManager.SIZE_PER_COUNT);
this.messageQueueOffset.add(queueOffset);
}
+
+ public void addMessage(final SelectMappedBufferResult mapedBuffer, final long queueOffset, final int batchNum) {
+ addMessage(mapedBuffer, queueOffset);
+ messageCount += batchNum;
+ }
+
public void release() {
for (SelectMappedBufferResult select : this.messageMapedList) {
select.release();
@@ -113,12 +125,8 @@ public class GetMessageResult {
return bufferTotalSize;
}
- public void setBufferTotalSize(int bufferTotalSize) {
- this.bufferTotalSize = bufferTotalSize;
- }
-
public int getMessageCount() {
- return this.messageMapedList.size();
+ return messageCount;
}
public boolean isSuggestPullingFromSlave() {
@@ -144,8 +152,7 @@ public class GetMessageResult {
@Override
public String toString() {
return "GetMessageResult [status=" + status + ", nextBeginOffset=" + nextBeginOffset + ", minOffset="
- + minOffset + ", maxOffset=" + maxOffset + ", bufferTotalSize=" + bufferTotalSize
+ + minOffset + ", maxOffset=" + maxOffset + ", bufferTotalSize=" + bufferTotalSize + ", messageCount=" + messageCount
+ ", suggestPullingFromSlave=" + suggestPullingFromSlave + "]";
}
-
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java b/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
index 1aa9ef8..a7d5083 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
@@ -25,29 +25,34 @@ import java.util.Iterator;
import java.util.List;
import java.util.ListIterator;
import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Stream;
+
+import com.google.common.collect.Lists;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
+import org.apache.rocketmq.store.logfile.MappedFile;
-public class MappedFileQueue {
+public class MappedFileQueue implements Swappable {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
private static final InternalLogger LOG_ERROR = InternalLoggerFactory.getLogger(LoggerName.STORE_ERROR_LOGGER_NAME);
private static final int DELETE_FILES_BATCH_MAX = 10;
- private final String storePath;
+ protected final String storePath;
protected final int mappedFileSize;
protected final CopyOnWriteArrayList<MappedFile> mappedFiles = new CopyOnWriteArrayList<MappedFile>();
- private final AllocateMappedFileService allocateMappedFileService;
+ protected final AllocateMappedFileService allocateMappedFileService;
protected long flushedWhere = 0;
- private long committedWhere = 0;
+ protected long committedWhere = 0;
- private volatile long storeTimestamp = 0;
+ protected volatile long storeTimestamp = 0;
public MappedFileQueue(final String storePath, int mappedFileSize,
AllocateMappedFileService allocateMappedFileService) {
@@ -91,7 +96,7 @@ public class MappedFileQueue {
return (MappedFile) mfs[mfs.length - 1];
}
- private Object[] copyMappedFiles(final int reservedMappedFiles) {
+ protected Object[] copyMappedFiles(final int reservedMappedFiles) {
Object[] mfs;
if (this.mappedFiles.size() <= reservedMappedFiles) {
@@ -166,8 +171,8 @@ public class MappedFileQueue {
return true;
}
- try {
- MappedFile mappedFile = new MappedFile(file.getPath(), mappedFileSize);
+ try {
+ MappedFile mappedFile = new DefaultMappedFile(file.getPath(), mappedFileSize);
mappedFile.setWrotePosition(this.mappedFileSize);
mappedFile.setFlushedPosition(this.mappedFileSize);
@@ -229,13 +234,13 @@ public class MappedFileQueue {
if (this.allocateMappedFileService != null) {
mappedFile = this.allocateMappedFileService.putRequestAndReturnMappedFile(nextFilePath,
nextNextFilePath, this.mappedFileSize);
- } else {
- try {
- mappedFile = new MappedFile(nextFilePath, this.mappedFileSize);
- } catch (IOException e) {
- log.error("create mappedFile exception", e);
+ } else {
+ try {
+ mappedFile = new DefaultMappedFile(nextFilePath, this.mappedFileSize);
+ } catch (IOException e) {
+ log.error("create mappedFile exception", e);
+ }
}
- }
if (mappedFile != null) {
if (this.mappedFiles.isEmpty()) {
@@ -585,6 +590,70 @@ public class MappedFileQueue {
}
}
+ @Override
+ public void swapMap(int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs) {
+
+ if (mappedFiles.isEmpty()) {
+ return;
+ }
+
+ if (reserveNum < 3) {
+ reserveNum = 3;
+ }
+
+ Object[] mfs = this.copyMappedFiles(0);
+ if (null == mfs) {
+ return;
+ }
+
+ for (int i = mfs.length - reserveNum - 1; i >= 0; i--) {
+ MappedFile mappedFile = (MappedFile) mfs[i];
+ if (System.currentTimeMillis() - mappedFile.getRecentSwapMapTime() > forceSwapIntervalMs) {
+ mappedFile.swapMap();
+ continue;
+ }
+ if (System.currentTimeMillis() - mappedFile.getRecentSwapMapTime() > normalSwapIntervalMs
+ && mappedFile.getMappedByteBufferAccessCountSinceLastSwap() > 0) {
+ mappedFile.swapMap();
+ continue;
+ }
+ }
+ }
+
+ @Override
+ public void cleanSwappedMap(long forceCleanSwapIntervalMs) {
+
+ if (mappedFiles.isEmpty()) {
+ return;
+ }
+
+ int reserveNum = 3;
+ Object[] mfs = this.copyMappedFiles(0);
+ if (null == mfs) {
+ return;
+ }
+
+ for (int i = mfs.length - reserveNum - 1; i >= 0; i--) {
+ MappedFile mappedFile = (MappedFile) mfs[i];
+ if (System.currentTimeMillis() - mappedFile.getRecentSwapMapTime() > forceCleanSwapIntervalMs) {
+ mappedFile.cleanSwapedMap(false);
+ }
+ }
+ }
+
+ public Object[] snapshot() {
+ // return a safe copy
+ return this.mappedFiles.toArray();
+ }
+
+ public Stream<MappedFile> stream() {
+ return this.mappedFiles.stream();
+ }
+
+ public Stream<MappedFile> reversedStream() {
+ return Lists.reverse(this.mappedFiles).stream();
+ }
+
public long getFlushedWhere() {
return flushedWhere;
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java b/store/src/main/java/org/apache/rocketmq/store/MessageExtBatch.java
similarity index 76%
copy from common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
copy to store/src/main/java/org/apache/rocketmq/store/MessageExtBatch.java
index a2713cb..e62dfb4 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MessageExtBatch.java
@@ -15,19 +15,30 @@
* limitations under the License.
*/
-package org.apache.rocketmq.common.message;
+package org.apache.rocketmq.store;
import java.nio.ByteBuffer;
-public class MessageExtBatch extends MessageExt {
+public class MessageExtBatch extends MessageExtBrokerInner {
private static final long serialVersionUID = -2353110995348498537L;
-
+ /**
+ * Inner batch means the batch dose not need to be unwrapped
+ */
+ private boolean isInnerBatch = false;
public ByteBuffer wrap() {
assert getBody() != null;
return ByteBuffer.wrap(getBody(), 0, getBody().length);
}
+ public boolean isInnerBatch() {
+ return isInnerBatch;
+ }
+
+ public void setInnerBatch(boolean innerBatch) {
+ isInnerBatch = innerBatch;
+ }
+
private ByteBuffer encodedBuff;
public ByteBuffer getEncodedBuff() {
diff --git a/store/src/main/java/org/apache/rocketmq/store/MessageStore.java b/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
index 4917c25..51d8a24 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
@@ -20,11 +20,18 @@ import java.util.HashMap;
import java.util.LinkedList;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
+
+import org.apache.rocketmq.common.SystemClock;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.store.config.BrokerRole;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.ha.HAService;
+import org.apache.rocketmq.store.logfile.MappedFile;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.ConsumeQueueStore;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import org.apache.rocketmq.store.util.PerfCounter;
/**
* This class defines contracting interfaces to implement, allowing third-party vendor to use customized message store.
@@ -107,6 +114,22 @@ public interface MessageStore {
final long offset, final int maxMsgNums, final MessageFilter messageFilter);
/**
+ * Query at most <code>maxMsgNums</code> messages belonging to <code>topic</code> at <code>queueId</code> starting
+ * from given <code>offset</code>. Resulting messages will further be screened using provided message filter.
+ *
+ * @param group Consumer group that launches this query.
+ * @param topic Topic to query.
+ * @param queueId Queue ID to query.
+ * @param offset Logical offset to start from.
+ * @param maxMsgNums Maximum count of messages to query.
+ * @param maxTotalMsgSize Maxisum total msg size of the messages
+ * @param messageFilter Message filter used to screen desired messages.
+ * @return Matched messages.
+ */
+ GetMessageResult getMessage(final String group, final String topic, final int queueId,
+ final long offset, final int maxMsgNums, final int maxTotalMsgSize, final MessageFilter messageFilter);
+
+ /**
* Get maximum offset of the topic queue.
*
* @param topic Topic name.
@@ -163,6 +186,15 @@ public interface MessageStore {
MessageExt lookMessageByOffset(final long commitLogOffset);
/**
+ * Look up the message by given commit log offset and size.
+ *
+ * @param commitLogOffset physical offset.
+ * @param size message size
+ * @return Message whose physical offset is as specified.
+ */
+ MessageExt lookMessageByOffset(long commitLogOffset, int size);
+
+ /**
* Get one message from the specified commit log offset.
*
* @param commitLogOffset commit log offset.
@@ -393,7 +425,7 @@ public interface MessageStore {
* @param queueId Queue ID.
* @return Consume queue.
*/
- ConsumeQueue getConsumeQueue(String topic, int queueId);
+ ConsumeQueueInterface getConsumeQueue(String topic, int queueId);
ScheduleMessageService getScheduleMessageService();
@@ -409,4 +441,137 @@ public interface MessageStore {
* @param brokerRole
*/
void handleScheduleMessageService(BrokerRole brokerRole);
+
+ /**
+ * Will be triggered when a new message is appended to commit log.
+ * @param msg the msg that is appended to commit log
+ * @param result append message result
+ * @param commitLogFile commit log file
+ */
+ void onCommitLogAppend(MessageExtBrokerInner msg, AppendMessageResult result, MappedFile commitLogFile);
+
+ /**
+ * Will be triggered when a new dispatch request is sent to message store.
+ * @param dispatchRequest dispatch request
+ * @param doDispatch do dispatch if true
+ * @param commitLogFile commit log file
+ * @param isRecover is from recover process
+ * @param isFileEnd if the dispatch request represents 'file end'
+ */
+ void onCommitLogDispatch(DispatchRequest dispatchRequest, boolean doDispatch, MappedFile commitLogFile, boolean isRecover, boolean isFileEnd);
+
+ /**
+ * Get the message store config
+ * @return the message store config
+ */
+ MessageStoreConfig getMessageStoreConfig();
+
+ /**
+ * Get the statistics service
+ * @return the statistics service
+ */
+ StoreStatsService getStoreStatsService();
+
+ /**
+ * Get the store checkpoint component
+ * @return the checkpoint component
+ */
+ StoreCheckpoint getStoreCheckpoint();
+
+ /**
+ * Get the system clock
+ * @return the system clock
+ */
+ SystemClock getSystemClock();
+
+ /**
+ * Get the commit log
+ * @return the commit log
+ */
+ CommitLog getCommitLog();
+
+ /**
+ * Get running flags
+ * @return running flags
+ */
+ RunningFlags getRunningFlags();
+
+ /**
+ * Get the transient store pool
+ * @return the transient store pool
+ */
+ TransientStorePool getTransientStorePool();
+
+ /**
+ * Get the HA service
+ * @return the HA service
+ */
+ HAService getHaService();
+
+ /**
+ * Register clean file hook
+ * @param logicalQueueCleanHook logical queue clean hook
+ */
+ void registerCleanFileHook(CleanFilesHook logicalQueueCleanHook);
+
+ /**
+ * Get the allocate-mappedFile service
+ * @return the allocate-mappedFile service
+ */
+ AllocateMappedFileService getAllocateMappedFileService();
+
+ /**
+ * Truncate dirty logic files
+ * @param phyOffset physical offset
+ */
+ void truncateDirtyLogicFiles(long phyOffset);
+
+ /**
+ * Destroy logics files
+ */
+ void destroyLogics();
+
+ /**
+ * Unlock mappedFile
+ * @param unlockMappedFile the file that needs to be unlocked
+ */
+ void unlockMappedFile(MappedFile unlockMappedFile);
+
+ /**
+ * Get the perf counter component
+ * @return the perf counter component
+ */
+ PerfCounter.Ticks getPerfCounter();
+
+ /**
+ * Get the queue store
+ * @return the queue store
+ */
+ ConsumeQueueStore getQueueStore();
+
+ /**
+ * If 'sync disk flush' is configured in this message store
+ * @return yes if true, no if false
+ */
+ boolean isSyncDiskFlush();
+
+ /**
+ * If this message store is sync master role
+ * @return yes if true, no if false
+ */
+ boolean isSyncMaster();
+
+ /**
+ * assign an queue offset and increase it.
+ * @param topicQueueKey topic-queue key
+ * @param msg message
+ * @param batchNum batch num
+ */
+ void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum);
+
+ /**
+ * remove offset table
+ * @param topicQueueKey topic-queue key
+ */
+ void removeOffsetTable(String topicQueueKey);
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/MultiPathMappedFileQueue.java b/store/src/main/java/org/apache/rocketmq/store/MultiPathMappedFileQueue.java
index 669698f..c5b2316 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MultiPathMappedFileQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MultiPathMappedFileQueue.java
@@ -24,6 +24,7 @@ import java.util.function.Supplier;
import org.apache.commons.lang3.StringUtils;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.logfile.MappedFile;
import java.io.File;
import java.util.ArrayList;
diff --git a/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
new file mode 100644
index 0000000..c3dd89b
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
@@ -0,0 +1,31 @@
+package org.apache.rocketmq.store;
+
+public class PutMessageContext {
+ private String topicQueueTableKey;
+ private long[] phyPos;
+ private int batchSize;
+
+ public PutMessageContext(String topicQueueTableKey) {
+ this.topicQueueTableKey = topicQueueTableKey;
+ }
+
+ public String getTopicQueueTableKey() {
+ return topicQueueTableKey;
+ }
+
+ public long[] getPhyPos() {
+ return phyPos;
+ }
+
+ public void setPhyPos(long[] phyPos) {
+ this.phyPos = phyPos;
+ }
+
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+ }
\ No newline at end of file
diff --git a/store/src/main/java/org/apache/rocketmq/store/SelectMappedBufferResult.java b/store/src/main/java/org/apache/rocketmq/store/SelectMappedBufferResult.java
index 03061e6..76919f3 100644
--- a/store/src/main/java/org/apache/rocketmq/store/SelectMappedBufferResult.java
+++ b/store/src/main/java/org/apache/rocketmq/store/SelectMappedBufferResult.java
@@ -16,6 +16,8 @@
*/
package org.apache.rocketmq.store;
+import org.apache.rocketmq.store.logfile.MappedFile;
+
import java.nio.ByteBuffer;
public class SelectMappedBufferResult {
@@ -26,7 +28,7 @@ public class SelectMappedBufferResult {
private int size;
- private MappedFile mappedFile;
+ protected MappedFile mappedFile;
public SelectMappedBufferResult(long startOffset, ByteBuffer byteBuffer, int size, MappedFile mappedFile) {
this.startOffset = startOffset;
@@ -48,12 +50,19 @@ public class SelectMappedBufferResult {
this.byteBuffer.limit(this.size);
}
+ public MappedFile getMappedFile() {
+ return mappedFile;
+ }
+
public synchronized void release() {
if (this.mappedFile != null) {
this.mappedFile.release();
this.mappedFile = null;
}
}
+ public synchronized boolean hasReleased() {
+ return this.mappedFile == null;
+ }
public long getStartOffset() {
return startOffset;
diff --git a/store/src/main/java/org/apache/rocketmq/store/StoreCheckpoint.java b/store/src/main/java/org/apache/rocketmq/store/StoreCheckpoint.java
index 7e6c706..f209ed2 100644
--- a/store/src/main/java/org/apache/rocketmq/store/StoreCheckpoint.java
+++ b/store/src/main/java/org/apache/rocketmq/store/StoreCheckpoint.java
@@ -26,6 +26,7 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
public class StoreCheckpoint {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
@@ -38,12 +39,12 @@ public class StoreCheckpoint {
public StoreCheckpoint(final String scpPath) throws IOException {
File file = new File(scpPath);
- MappedFile.ensureDirOK(file.getParent());
+ DefaultMappedFile.ensureDirOK(file.getParent());
boolean fileExists = file.exists();
this.randomAccessFile = new RandomAccessFile(file, "rw");
this.fileChannel = this.randomAccessFile.getChannel();
- this.mappedByteBuffer = fileChannel.map(MapMode.READ_WRITE, 0, MappedFile.OS_PAGE_SIZE);
+ this.mappedByteBuffer = fileChannel.map(MapMode.READ_WRITE, 0, DefaultMappedFile.OS_PAGE_SIZE);
if (fileExists) {
log.info("store checkpoint file exists, " + scpPath);
@@ -66,7 +67,7 @@ public class StoreCheckpoint {
this.flush();
// unmap mappedByteBuffer
- MappedFile.clean(this.mappedByteBuffer);
+ DefaultMappedFile.clean(this.mappedByteBuffer);
try {
this.fileChannel.close();
diff --git a/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java b/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
index f63efd6..81b407e 100644
--- a/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
+++ b/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
@@ -16,10 +16,21 @@
*/
package org.apache.rocketmq.store;
+import com.google.common.base.Preconditions;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.logfile.MappedFile;
+
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
+import java.nio.ByteBuffer;
+
+import static java.lang.String.format;
public class StoreUtil {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
+
public static final long TOTAL_PHYSICAL_MEMORY_SIZE = getTotalPhysicalMemorySize();
@SuppressWarnings("restriction")
@@ -32,4 +43,41 @@ public class StoreUtil {
return physicalTotal;
}
+
+ public static boolean isStreamMode(MessageStore messageStore) {
+ return messageStore instanceof StreamMessageStore;
+ }
+
+ public static void fileAppend(MappedFile file, ByteBuffer data) {
+ boolean success = file.appendMessage(data);
+ if (!success) {
+ throw new RuntimeException(format("fileAppend failed for file: %s and data remaining: %d", file, data.remaining()));
+ }
+ }
+
+ public static FileQueueSnapshot getFileQueueSnapshot(MappedFileQueue mappedFileQueue) {
+ return getFileQueueSnapshot(mappedFileQueue, mappedFileQueue.getLastMappedFile().getFileFromOffset());
+ }
+
+ public static FileQueueSnapshot getFileQueueSnapshot(MappedFileQueue mappedFileQueue, final long currentFile) {
+ try {
+ Preconditions.checkNotNull(mappedFileQueue, "file queue shouldn't be null");
+ MappedFile firstFile = mappedFileQueue.getFirstMappedFile();
+ MappedFile lastFile = mappedFileQueue.getLastMappedFile();
+ int mappedFileSize = mappedFileQueue.getMappedFileSize();
+ if (firstFile == null || lastFile == null) {
+ return new FileQueueSnapshot(firstFile, -1, lastFile, -1, currentFile, -1, 0, false);
+ }
+
+ long firstFileIndex = 0;
+ long lastFileIndex = (lastFile.getFileFromOffset() - firstFile.getFileFromOffset()) / mappedFileSize;
+ long currentFileIndex = (currentFile - firstFile.getFileFromOffset()) / mappedFileSize;
+ long behind = (lastFile.getFileFromOffset() - currentFile) / mappedFileSize;
+ boolean exist = firstFile.getFileFromOffset() <= currentFile && currentFile <= lastFile.getFileFromOffset();
+ return new FileQueueSnapshot(firstFile, firstFileIndex, lastFile, lastFileIndex, currentFile, currentFileIndex, behind, exist);
+ } catch (Exception e) {
+ log.error("[BUG] get file queue snapshot failed. fileQueue: {}, currentFile: {}", mappedFileQueue, currentFile, e);
+ }
+ return new FileQueueSnapshot();
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java b/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java
new file mode 100644
index 0000000..d9277ff
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java
@@ -0,0 +1,2573 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.store;
+
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.MixAll;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.SystemClock;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.message.MessageAccessor;
+import org.apache.rocketmq.common.message.MessageConst;
+import org.apache.rocketmq.common.message.MessageDecoder;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.running.RunningStats;
+import org.apache.rocketmq.common.sysflag.MessageSysFlag;
+import org.apache.rocketmq.common.topic.TopicValidator;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.config.BrokerRole;
+import org.apache.rocketmq.store.config.FlushDiskType;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.config.StorePathConfigHelper;
+import org.apache.rocketmq.store.ha.HAService;
+import org.apache.rocketmq.store.index.IndexService;
+import org.apache.rocketmq.store.index.QueryOffsetResult;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
+import org.apache.rocketmq.store.logfile.MappedFile;
+import org.apache.rocketmq.store.queue.BatchConsumeQueue;
+import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
+import org.apache.rocketmq.store.queue.ConsumeQueueStore;
+import org.apache.rocketmq.store.queue.CqUnit;
+import org.apache.rocketmq.store.queue.ReferredIterator;
+import org.apache.rocketmq.store.schedule.ScheduleMessageService;
+import org.apache.rocketmq.store.stats.BrokerStatsManager;
+import org.apache.rocketmq.store.util.PerfCounter;
+import org.apache.rocketmq.store.util.QueueTypeUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileLock;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static java.lang.String.format;
+
+public class StreamMessageStore implements MessageStore {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
+
+ public final PerfCounter.Ticks perfs = new PerfCounter.Ticks(log);
+
+ private final MessageStoreConfig messageStoreConfig;
+ // CommitLog
+ private final CommitLog commitLog;
+
+ private final ConsumeQueueStore consumeQueueStore;
+
+ private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
+
+ protected HashMap<String/* topic-queueid */, Long/* offset */> topicQueueTable = new HashMap<String, Long>(1024);
+
+ protected HashMap<String/* topic-queueid */, Long/* offset */> batchTopicQueueTable = new HashMap<String, Long>(1024);
+
+ private final FlushConsumeQueueService flushConsumeQueueService;
+
+ private final CleanCommitLogService cleanCommitLogService;
+
+ private final CleanConsumeQueueService cleanConsumeQueueService;
+
+ private final CorrectLogicOffsetService correctLogicOffsetService;
+
+ private final IndexService indexService;
+
+ private final AllocateMappedFileService allocateMappedFileService;
+
+ private final ReputMessageService reputMessageService;
+
+ private final HAService haService;
+
+ private final ScheduleMessageService scheduleMessageService;
+
+ private final StoreStatsService storeStatsService;
+
+ private final TransientStorePool transientStorePool;
+
+ private final RunningFlags runningFlags = new RunningFlags();
+ private final SystemClock systemClock = new SystemClock();
+
+ private final ScheduledExecutorService scheduledExecutorService =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryImpl("StoreScheduledThread"));
+ private final BrokerStatsManager brokerStatsManager;
+ private final MessageArrivingListener messageArrivingListener;
+ private final BrokerConfig brokerConfig;
+
+ private volatile boolean shutdown = true;
+
+ private StoreCheckpoint storeCheckpoint;
+
+ private AtomicLong printTimes = new AtomicLong(0);
+
+ private final LinkedList<CommitLogDispatcher> dispatcherList;
+
+ private RandomAccessFile lockFile;
+
+ private FileLock lock;
+
+ boolean shutDownNormal = false;
+
+ //polish for reput
+ private ThreadPoolExecutor[] reputExecutors;
+
+ private BlockingQueue<Runnable>[] reputQueues;
+
+ private boolean isDispatchFromSenderThread;
+
+ private static final Future EMPTY_FUTURE = new Future() {
+ @Override
+ public boolean cancel(final boolean mayInterruptIfRunning) {
+ return false;
+ }
+
+ @Override
+ public boolean isCancelled() {
+ return false;
+ }
+
+ @Override
+ public boolean isDone() {
+ return true;
+ }
+
+ @Override
+ public Object get() {
+ return null;
+ }
+
+ @Override
+ public Object get(final long timeout, final TimeUnit unit) {
+ return null;
+ }
+ };
+
+ // Max pull msg size
+ private final static int MAX_PULL_MSG_SIZE = 128 * 1024 * 1024;
+
+ public StreamMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager,
+ final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException {
+ this.messageArrivingListener = messageArrivingListener;
+ this.brokerConfig = brokerConfig;
+ this.messageStoreConfig = messageStoreConfig;
+ this.brokerStatsManager = brokerStatsManager;
+ this.allocateMappedFileService = new AllocateMappedFileService(this);
+ if (messageStoreConfig.isEnableDLegerCommitLog()) {
+ throw new RuntimeException("dleger is not supported in this message store.");
+ }
+ this.isDispatchFromSenderThread = messageStoreConfig.isDispatchFromSenderThread();
+ this.commitLog = new CommitLog(this);
+ this.consumeQueueTable = new ConcurrentHashMap<>(32);
+ this.consumeQueueStore = new ConsumeQueueStore(this, this.messageStoreConfig, this.consumeQueueTable);
+
+ this.flushConsumeQueueService = new FlushConsumeQueueService();
+ this.cleanCommitLogService = new CleanCommitLogService();
+ this.cleanConsumeQueueService = new CleanConsumeQueueService();
+ this.correctLogicOffsetService = new CorrectLogicOffsetService();
+ this.storeStatsService = new StoreStatsService();
+ this.indexService = new IndexService(this);
+ if (!messageStoreConfig.isEnableDLegerCommitLog()) {
+ this.haService = new HAService(this);
+ } else {
+ this.haService = null;
+ }
+ if (isDispatchFromSenderThread) {
+ this.reputMessageService = new SyncReputMessageService();
+ } else {
+ this.reputMessageService = new ReputMessageService();
+ }
+
+ this.scheduleMessageService = new ScheduleMessageService(this);
+
+ this.transientStorePool = new TransientStorePool(messageStoreConfig);
+
+ if (messageStoreConfig.isTransientStorePoolEnable()) {
+ this.transientStorePool.init();
+ }
+
+ this.allocateMappedFileService.start();
+
+ this.indexService.start();
+
+ this.dispatcherList = new LinkedList<>();
+ this.dispatcherList.addLast(new CommitLogDispatcherBuildConsumeQueue());
+ this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex());
+
+ File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir()));
+ DefaultMappedFile.ensureDirOK(file.getParent());
+ lockFile = new RandomAccessFile(file, "rw");
+ initAsyncReputThreads(messageStoreConfig.getDispatchCqThreads(), messageStoreConfig.getDispatchCqCacheNum());
+ }
+
+ /**
+ * @throws IOException
+ */
+ @Override
+ public boolean load() {
+ boolean result = true;
+
+ try {
+ long start = System.currentTimeMillis();
+ boolean lastExitOK = !this.isTempFileExist();
+ log.info("last shutdown {}, root dir: {}", lastExitOK ? "normally" : "abnormally", messageStoreConfig.getStorePathRootDir());
+
+ if (null != scheduleMessageService) {
+ result = result && this.scheduleMessageService.load();
+ }
+
+ // load Commit Log
+ result = result && this.commitLog.load();
+
+ // load Batch Consume Queue
+ result = result && this.loadBatchConsumeQueue();
+
+ if (result) {
+ this.storeCheckpoint =
+ new StoreCheckpoint(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
+
+ this.indexService.load(lastExitOK);
+
+ this.recover(lastExitOK);
+
+ log.info("load over, and the max phy offset = {} cost = {}", this.getMaxPhyOffset(), System.currentTimeMillis() - start);
+ }
+ } catch (Exception e) {
+ log.error("load exception", e);
+ result = false;
+ }
+
+ if (!result) {
+ this.allocateMappedFileService.shutdown();
+ }
+
+ return result;
+ }
+
+ /**
+ * @throws Exception
+ */
+ @Override
+ public void start() throws Exception {
+
+ lock = lockFile.getChannel().tryLock(0, 1, false);
+ if (lock == null || lock.isShared() || !lock.isValid()) {
+ throw new RuntimeException("Lock failed,MQ already started");
+ }
+
+ lockFile.getChannel().write(ByteBuffer.wrap("lock".getBytes()));
+ lockFile.getChannel().force(true);
+
+ if (this.getMessageStoreConfig().isDuplicationEnable()) {
+ this.reputMessageService.setReputFromOffset(this.commitLog.getConfirmOffset());
+ } else {
+ this.reputMessageService.setReputFromOffset(this.commitLog.getMaxOffset());
+ }
+ this.reputMessageService.start();
+
+ if (!messageStoreConfig.isEnableDLegerCommitLog()) {
+ this.haService.start();
+ this.handleScheduleMessageService(messageStoreConfig.getBrokerRole());
+ }
+
+ this.flushConsumeQueueService.start();
+ this.commitLog.start();
+ this.storeStatsService.start();
+
+ this.createTempFile();
+ this.addScheduleTask();
+ this.perfs.start();
+ this.shutdown = false;
+ }
+
+ @Override
+ public void shutdown() {
+ if (!this.shutdown) {
+ this.shutdown = true;
+
+ this.scheduledExecutorService.shutdown();
+
+ try {
+
+ Thread.sleep(1000 * 3);
+ } catch (InterruptedException e) {
+ log.error("shutdown Exception, ", e);
+ }
+
+ if (this.scheduleMessageService != null) {
+ this.scheduleMessageService.shutdown();
+ }
+ if (this.haService != null) {
+ this.haService.shutdown();
+ }
+
+ this.storeStatsService.shutdown();
+ this.indexService.shutdown();
+ this.commitLog.shutdown();
+ this.reputMessageService.shutdown();
+ this.flushConsumeQueueService.shutdown();
+ this.allocateMappedFileService.shutdown();
+ this.storeCheckpoint.flush();
+ this.storeCheckpoint.shutdown();
+
+ this.perfs.shutdown();
+
+ if (this.runningFlags.isWriteable() && dispatchBehindBytes() == 0) {
+ this.deleteFile(StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
+ shutDownNormal = true;
+ } else {
+ log.warn("the store may be wrong, so shutdown abnormally, and keep abort file. writable: {}, dispatchBehindBytes: {}, abort file: {}",
+ this.runningFlags.isWriteable(), dispatchBehindBytes(), StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
+ }
+ }
+
+ this.transientStorePool.destroy();
+
+ if (lockFile != null && lock != null) {
+ try {
+ lock.release();
+ lockFile.close();
+ } catch (IOException e) {
+ }
+ }
+ }
+
+ @Override
+ public void destroy() {
+ this.destroyLogics();
+ this.commitLog.destroy();
+ this.indexService.destroy();
+ this.deleteFile(StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
+ this.deleteFile(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
+ }
+
+ @Override
+ public void destroyLogics() {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.destroy(logic);
+ }
+ }
+ }
+
+ @Override
+ public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
+ if (this.shutdown) {
+ log.warn("message store has shutdown, so putMessage is forbidden");
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ }
+
+ if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
+ long value = this.printTimes.getAndIncrement();
+ if ((value % 50000) == 0) {
+ log.warn("message store is slave mode, so putMessage is forbidden ");
+ }
+
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ }
+
+ if (msg.getProperties().containsKey(MessageConst.PROPERTY_INNER_NUM)
+ && !MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
+ log.warn("[BUG]The message had property {} but is not an inner batch", MessageConst.PROPERTY_INNER_NUM);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
+ CQType cqType = QueueTypeUtils.getCQType(this);
+
+ if (!CQType.BatchCQ.equals(cqType)) {
+ log.warn("[BUG]The message is an inner batch but cq type is not batch consume queue");
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+ }
+
+ if (!this.runningFlags.isWriteable()) {
+ long value = this.printTimes.getAndIncrement();
+ if ((value % 50000) == 0) {
+ log.warn("message store is not writeable, so putMessage is forbidden " + this.runningFlags.getFlagBits());
+ }
+
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ } else {
+ this.printTimes.set(0);
+ }
+
+ int topicLen = msg.getTopic().length();
+ if (topicLen > this.messageStoreConfig.getMaxTopicLength()) {
+ log.warn("putMessage message topic[{}] length too long {}", msg.getTopic(), topicLen);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (topicLen > Byte.MAX_VALUE) {
+ log.warn("putMessage message topic[{}] length too long {}, but it is not supported by broker",
+ msg.getTopic(), topicLen);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (msg.getPropertiesString() != null && msg.getPropertiesString().length() > Short.MAX_VALUE) {
+ log.warn("putMessage message properties length too long " + msg.getPropertiesString().length());
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.PROPERTIES_SIZE_EXCEEDED, null));
+ }
+
+ if (this.isOSPageCacheBusy()) {
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null));
+ }
+
+ long beginTime = this.getSystemClock().now();
+ perfs.startTick("PUT_MESSAGE_TIME_MS");
+ CompletableFuture<PutMessageResult> result = this.commitLog.asyncPutMessage(msg);
+ perfs.endTick("PUT_MESSAGE_TIME_MS");
+
+ long eclipseTime = this.getSystemClock().now() - beginTime;
+ if (eclipseTime > 500) {
+ log.warn("putMessage not in lock eclipse time(ms)={}, bodyLength={}", eclipseTime, msg.getBody().length);
+ }
+
+ return result;
+ }
+
+ @Override
+ public PutMessageResult putMessage(MessageExtBrokerInner msg) {
+ CompletableFuture<PutMessageResult> future = asyncPutMessage(msg);
+ try {
+ return future.get(3, TimeUnit.SECONDS);
+ } catch (Throwable t) {
+ log.error("Get async put result failed", t);
+ return new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null);
+ }
+ }
+
+ @Override
+ public PutMessageResult putMessages(MessageExtBatch messageExtBatch) {
+ CompletableFuture<PutMessageResult> future = asyncPutMessages(messageExtBatch);
+ try {
+ return future.get(3, TimeUnit.SECONDS);
+ } catch (Throwable t) {
+ log.error("Get async put result failed", t);
+ return new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null);
+ }
+ }
+
+ @Override
+ public CompletableFuture<PutMessageResult> asyncPutMessages(MessageExtBatch messageExtBatch) {
+ if (this.shutdown) {
+ log.warn("StreamMessageStore has shutdown, so putMessages is forbidden");
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ }
+
+ if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
+ long value = this.printTimes.getAndIncrement();
+ if ((value % 50000) == 0) {
+ log.warn("StreamMessageStore is in slave mode, so putMessages is forbidden ");
+ }
+
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ }
+
+ if (!this.runningFlags.isWriteable()) {
+ long value = this.printTimes.getAndIncrement();
+ if ((value % 50000) == 0) {
+ log.warn("StreamMessageStore is not writable, so putMessages is forbidden " + this.runningFlags.getFlagBits());
+ }
+
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
+ } else {
+ this.printTimes.set(0);
+ }
+
+ int topicLen = messageExtBatch.getTopic().length();
+ if (topicLen > this.messageStoreConfig.getMaxTopicLength()) {
+ log.warn("putMessage batch message topic[{}] length too long {}", messageExtBatch.getTopic(), topicLen);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (topicLen > Byte.MAX_VALUE) {
+ log.warn("putMessage batch message topic[{}] length too long {}, but it is not supported by broker",
+ messageExtBatch.getTopic(), topicLen);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (messageExtBatch.getBody().length > messageStoreConfig.getMaxMessageSize()) {
+ log.warn("PutMessages body length too long " + messageExtBatch.getBody().length);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (this.isOSPageCacheBusy()) {
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null));
+ }
+
+ long beginTime = this.getSystemClock().now();
+ CompletableFuture<PutMessageResult> result = this.commitLog.asyncPutMessages(messageExtBatch);
+
+ long eclipseTime = this.getSystemClock().now() - beginTime;
+ if (eclipseTime > 500) {
+ log.warn("not in lock eclipse time(ms)={}, bodyLength={}", eclipseTime, messageExtBatch.getBody().length);
+ }
+ this.storeStatsService.setPutMessageEntireTimeMax(eclipseTime);
+
+ return result;
+ }
+
+ @Override
+ public boolean isOSPageCacheBusy() {
+ long begin = this.getCommitLog().getBeginTimeInLock();
+ long diff = this.systemClock.now() - begin;
+
+ return diff < 10000000
+ && diff > this.messageStoreConfig.getOsPageCacheBusyTimeOutMills();
+ }
+
+ @Override
+ public long lockTimeMills() {
+ return this.commitLog.lockTimeMills();
+ }
+
+ @Override
+ public SystemClock getSystemClock() {
+ return systemClock;
+ }
+
+ @Override
+ public CommitLog getCommitLog() {
+ return commitLog;
+ }
+
+ public boolean isDispatchFromSenderThread() {
+ return isDispatchFromSenderThread;
+ }
+
+ @Override
+ public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
+ final int maxMsgNums,
+ final MessageFilter messageFilter) {
+ return getMessage(group, topic, queueId, offset, maxMsgNums, MAX_PULL_MSG_SIZE, messageFilter);
+ }
+
+ @Override
+ public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
+ final int maxMsgNums,
+ final int maxTotalMsgSize,
+ final MessageFilter messageFilter) {
+ if (this.shutdown) {
+ log.warn("message store has shutdown, so getMessage is forbidden");
+ return null;
+ }
+
+ if (!this.runningFlags.isReadable()) {
+ log.warn("message store is not readable, so getMessage is forbidden " + this.runningFlags.getFlagBits());
+ return null;
+ }
+
+ long beginTime = this.getSystemClock().now();
+
+ GetMessageStatus status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
+ long nextBeginOffset = offset;
+ long minOffset = 0;
+ long maxOffset = 0;
+
+ GetMessageResult getResult = new GetMessageResult();
+
+ final long maxOffsetPy = this.commitLog.getMaxOffset();
+
+ ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
+ if (consumeQueue != null) {
+ minOffset = consumeQueue.getMinOffsetInQueue();
+ maxOffset = consumeQueue.getMaxOffsetInQueue();
+
+ if (maxOffset == 0) {
+ status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
+ nextBeginOffset = nextOffsetCorrection(offset, 0);
+ } else if (offset < minOffset) {
+ status = GetMessageStatus.OFFSET_TOO_SMALL;
+ nextBeginOffset = nextOffsetCorrection(offset, minOffset);
+ } else if (offset == maxOffset) {
+ status = GetMessageStatus.OFFSET_OVERFLOW_ONE;
+ nextBeginOffset = nextOffsetCorrection(offset, offset);
+ } else if (offset > maxOffset) {
+ status = GetMessageStatus.OFFSET_OVERFLOW_BADLY;
+ if (0 == minOffset) {
+ nextBeginOffset = nextOffsetCorrection(offset, minOffset);
+ } else {
+ nextBeginOffset = nextOffsetCorrection(offset, maxOffset);
+ }
+ } else {
+ final int maxFilterMessageCount = Math.max(messageStoreConfig.getPullBatchMaxMessageCount(), maxMsgNums);
+ final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
+
+ long maxPullSize = Math.max(maxTotalMsgSize, 100);
+ if (maxPullSize > MAX_PULL_MSG_SIZE) {
+ log.warn("The max pull size is too large maxPullSize={} topic={} queueId={}", maxPullSize, topic, queueId);
+ maxPullSize = MAX_PULL_MSG_SIZE;
+ }
+ status = GetMessageStatus.NO_MATCHED_MESSAGE;
+ long maxPhyOffsetPulling = 0;
+ int cqFileNum = 0;
+
+ while (getResult.getBufferTotalSize() <= 0
+ && nextBeginOffset < maxOffset
+ && cqFileNum++ < this.messageStoreConfig.getTravelCqFileNumWhenGetMessage()) {
+ ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(nextBeginOffset);
+
+ if (bufferConsumeQueue == null) {
+ status = GetMessageStatus.OFFSET_FOUND_NULL;
+ nextBeginOffset = nextOffsetCorrection(nextBeginOffset, this.consumeQueueStore.rollNextFile(consumeQueue, nextBeginOffset));
+ log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
+ + maxOffset + ", but access logic queue failed. Correct nextBeginOffset to " + nextBeginOffset);
+ break;
+ }
+
+ try {
+ long nextPhyFileStartOffset = Long.MIN_VALUE;
+ while (bufferConsumeQueue.hasNext()
+ && nextBeginOffset < maxOffset) {
+ CqUnit cqUnit = bufferConsumeQueue.next();
+ long offsetPy = cqUnit.getPos();
+ int sizePy = cqUnit.getSize();
+
+ boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
+
+ if (cqUnit.getQueueOffset() - offset > maxFilterMessageCount) {
+ break;
+ }
+
+ if (this.isTheBatchFull(sizePy, cqUnit.getBatchNum(), maxMsgNums, maxPullSize, getResult.getBufferTotalSize(), getResult.getMessageCount(),
+ isInDisk)) {
+ break;
+ }
+
+ if (getResult.getBufferTotalSize() >= maxPullSize) {
+ break;
+ }
+
+ maxPhyOffsetPulling = offsetPy;
+
+ //Be careful, here should before the isTheBatchFull
+ nextBeginOffset = cqUnit.getQueueOffset() + cqUnit.getBatchNum();
+
+ if (nextPhyFileStartOffset != Long.MIN_VALUE) {
+ if (offsetPy < nextPhyFileStartOffset) {
+ continue;
+ }
+ }
+
+ if (messageFilter != null
+ && !messageFilter.isMatchedByConsumeQueue(cqUnit.getValidTagsCodeAsLong(), cqUnit.getCqExtUnit())) {
+ if (getResult.getBufferTotalSize() == 0) {
+ status = GetMessageStatus.NO_MATCHED_MESSAGE;
+ }
+
+ continue;
+ }
+
+ SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
+ if (null == selectResult) {
+ if (getResult.getBufferTotalSize() == 0) {
+ status = GetMessageStatus.MESSAGE_WAS_REMOVING;
+ }
+
+ nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy);
+ continue;
+ }
+
+ if (messageFilter != null
+ && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
+ if (getResult.getBufferTotalSize() == 0) {
+ status = GetMessageStatus.NO_MATCHED_MESSAGE;
+ }
+ // release...
+ selectResult.release();
+ continue;
+ }
+
+ this.storeStatsService.getGetMessageTransferedMsgCount().add(1);
+ getResult.addMessage(selectResult, cqUnit.getQueueOffset(), cqUnit.getBatchNum());
+ status = GetMessageStatus.FOUND;
+ nextPhyFileStartOffset = Long.MIN_VALUE;
+ }
+ } finally {
+ bufferConsumeQueue.release();
+ }
+ }
+
+ if (diskFallRecorded) {
+ long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
+ brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
+ }
+
+ long diff = maxOffsetPy - maxPhyOffsetPulling;
+ long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE
+ * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
+ getResult.setSuggestPullingFromSlave(diff > memory);
+ }
+ } else {
+ status = GetMessageStatus.NO_MATCHED_LOGIC_QUEUE;
+ nextBeginOffset = nextOffsetCorrection(offset, 0);
+ }
+
+ if (GetMessageStatus.FOUND == status) {
+ this.storeStatsService.getGetMessageTimesTotalFound().add(1);
+ } else {
+ this.storeStatsService.getGetMessageTimesTotalMiss().add(1);
+ }
+ long elapsedTime = this.getSystemClock().now() - beginTime;
+ this.storeStatsService.setGetMessageEntireTimeMax(elapsedTime);
+
+ getResult.setStatus(status);
+ getResult.setNextBeginOffset(nextBeginOffset);
+ getResult.setMaxOffset(maxOffset);
+ getResult.setMinOffset(minOffset);
+ return getResult;
+ }
+
+ @Override
+ public long getMaxOffsetInQueue(String topic, int queueId) {
+ ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
+ if (logic != null) {
+ return logic.getMaxOffsetInQueue();
+ }
+
+ return 0;
+ }
+
+ @Override
+ public long getMaxOffsetInQueue(String topic, int queueId, boolean committed) {
+ if (committed) {
+ ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
+ if (logic != null) {
+ return logic.getMaxOffsetInQueue();
+ }
+ } else {
+ Long offset = this.batchTopicQueueTable.get(topic + "-" + queueId);
+ if (offset != null) {
+ return offset;
+ }
+ }
+
+ return 0;
+ }
+
+ @Override
+ public long getMinOffsetInQueue(String topic, int queueId) {
+ ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
+ if (logic != null) {
+ return logic.getMinOffsetInQueue();
+ }
+
+ return -1;
+ }
+
+ @Override
+ public long getCommitLogOffsetInQueue(String topic, int queueId, long consumeQueueOffset) {
+ ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
+ if (consumeQueue != null) {
+
+ ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(consumeQueueOffset);
+ if (bufferConsumeQueue != null) {
+ try {
+ if (bufferConsumeQueue.hasNext()) {
+ long offsetPy = bufferConsumeQueue.next().getPos();
+ return offsetPy;
+ }
+ } finally {
+ bufferConsumeQueue.release();
+ }
+ }
+ }
+
+ return 0;
+ }
+
+ @Override
+ public long getOffsetInQueueByTime(String topic, int queueId, long timestamp) {
+ ConsumeQueueInterface logic = getConsumeQueue(topic, queueId);
+ if (logic != null) {
+ long resultOffset = logic.getOffsetInQueueByTime(timestamp);
+ // -1 means no msg found.
+ if (resultOffset == -1) {
+ return -1;
+ }
+ // Make sure the result offset should in valid range.
+ resultOffset = Math.max(resultOffset, logic.getMinOffsetInQueue());
+ resultOffset = Math.min(resultOffset, logic.getMaxOffsetInQueue());
+ return resultOffset;
+ }
+
+ // logic is null means there is no message in this queue, return -1.
+ return -1;
+ }
+
+ @Override
+ public MessageExt lookMessageByOffset(long commitLogOffset) {
+ SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, 4);
+ if (null != sbr) {
+ try {
+ // 1 TOTALSIZE
+ int size = sbr.getByteBuffer().getInt();
+ return lookMessageByOffset(commitLogOffset, size);
+ } finally {
+ sbr.release();
+ }
+ }
+
+ return null;
+ }
+
+ @Override
+ public SelectMappedBufferResult selectOneMessageByOffset(long commitLogOffset) {
+ SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, 4);
+ if (null != sbr) {
+ try {
+ // 1 TOTALSIZE
+ int size = sbr.getByteBuffer().getInt();
+ return this.commitLog.getMessage(commitLogOffset, size);
+ } finally {
+ sbr.release();
+ }
+ }
+
+ return null;
+ }
+
+ @Override
+ public SelectMappedBufferResult selectOneMessageByOffset(long commitLogOffset, int msgSize) {
+ return this.commitLog.getMessage(commitLogOffset, msgSize);
+ }
+
+ @Override
+ public String getRunningDataInfo() {
+ return this.storeStatsService.toString();
+ }
+
+ private String getStorePathPhysic() {
+ String storePathPhysic = StreamMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
+ return storePathPhysic;
+ }
+
+ @Override
+ public HashMap<String, String> getRuntimeInfo() {
+ HashMap<String, String> result = this.storeStatsService.getRuntimeInfo();
+
+ {
+ String storePathPhysic = this.getMessageStoreConfig().getStorePathCommitLog();
+ double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
+ result.put(RunningStats.commitLogDiskRatio.name(), String.valueOf(physicRatio));
+
+ }
+
+ {
+
+ String storePathLogics = StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir());
+ double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
+ result.put(RunningStats.consumeQueueDiskRatio.name(), String.valueOf(logicsRatio));
+ }
+
+ {
+ if (this.scheduleMessageService != null) {
+ this.scheduleMessageService.buildRunningStats(result);
+ }
+ }
+
+ result.put(RunningStats.commitLogMinOffset.name(), String.valueOf(this.getMinPhyOffset()));
+ result.put(RunningStats.commitLogMaxOffset.name(), String.valueOf(this.getMaxPhyOffset()));
+
+ return result;
+ }
+
+ @Override
+ public long getMaxPhyOffset() {
+ return this.commitLog.getMaxOffset();
+ }
+
+ @Override
+ public long getMinPhyOffset() {
+ return this.commitLog.getMinOffset();
+ }
+
+ @Override
+ public long getEarliestMessageTime(String topic, int queueId) {
+ ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
+ if (logicQueue != null) {
+ return getStoreTime(logicQueue.getEarliestUnit());
+ }
+
+ return -1;
+ }
+
+ private long getStoreTime(CqUnit result) {
+ if (result != null) {
+ try {
+ final long phyOffset = result.getPos();
+ final int size = result.getSize();
+ long storeTime = this.getCommitLog().pickupStoreTimestamp(phyOffset, size);
+ return storeTime;
+ } catch (Exception e) {
+ }
+ }
+ return -1;
+ }
+
+ @Override
+ public long getEarliestMessageTime() {
+ final long minPhyOffset = this.getMinPhyOffset();
+ final int size = this.messageStoreConfig.getMaxMessageSize() * 2;
+ return this.getCommitLog().pickupStoreTimestamp(minPhyOffset, size);
+ }
+
+ @Override
+ public long getMessageStoreTimeStamp(String topic, int queueId, long consumeQueueOffset) {
+ ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
+ if (logicQueue != null) {
+ return getStoreTime(logicQueue.get(consumeQueueOffset));
+ }
+
+ return -1;
+ }
+
+ @Override
+ public long getMessageTotalInQueue(String topic, int queueId) {
+ ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
+ if (logicQueue != null) {
+ return logicQueue.getMessageTotalInQueue();
+ }
+
+ return -1;
+ }
+
+ @Override
+ public SelectMappedBufferResult getCommitLogData(final long offset) {
+ if (this.shutdown) {
+ log.warn("message store has shutdown, so getPhyQueueData is forbidden");
+ return null;
+ }
+
+ return this.commitLog.getData(offset);
+ }
+
+ @Override
+ public boolean appendToCommitLog(long startOffset, byte[] data, int dataStart, int dataLength) {
+ if (this.shutdown) {
+ log.warn("message store has shutdown, so appendToPhyQueue is forbidden");
+ return false;
+ }
+
+ boolean result = this.commitLog.appendData(startOffset, data, dataStart, dataLength);
+ if (result) {
+ this.reputMessageService.wakeup();
+ } else {
+ log.error("appendToPhyQueue failed " + startOffset + " " + data.length);
+ }
+
+ return result;
+ }
+
+ @Override
+ public void executeDeleteFilesManually() {
+ this.cleanCommitLogService.excuteDeleteFilesManualy();
+ }
+
+ @Override
+ public QueryMessageResult queryMessage(String topic, String key, int maxNum, long begin, long end) {
+ QueryMessageResult queryMessageResult = new QueryMessageResult();
+
+ long lastQueryMsgTime = end;
+
+ for (int i = 0; i < 3; i++) {
+ QueryOffsetResult queryOffsetResult = this.indexService.queryOffset(topic, key, maxNum, begin, lastQueryMsgTime);
+ if (queryOffsetResult.getPhyOffsets().isEmpty()) {
+ break;
+ }
+
+ Collections.sort(queryOffsetResult.getPhyOffsets());
+
+ queryMessageResult.setIndexLastUpdatePhyoffset(queryOffsetResult.getIndexLastUpdatePhyoffset());
+ queryMessageResult.setIndexLastUpdateTimestamp(queryOffsetResult.getIndexLastUpdateTimestamp());
+
+ for (int m = 0; m < queryOffsetResult.getPhyOffsets().size(); m++) {
+ long offset = queryOffsetResult.getPhyOffsets().get(m);
+
+ try {
+
+ boolean match = true;
+ MessageExt msg = this.lookMessageByOffset(offset);
+ if (0 == m) {
+ lastQueryMsgTime = msg.getStoreTimestamp();
+ }
+
+// String[] keyArray = msg.getKeys().split(MessageConst.KEY_SEPARATOR);
+// if (topic.equals(msg.getTopic())) {
+// for (String k : keyArray) {
+// if (k.equals(key)) {
+// match = true;
+// break;
+// }
+// }
+// }
+
+ if (match) {
+ SelectMappedBufferResult result = this.commitLog.getData(offset, false);
+ if (result != null) {
+ int size = result.getByteBuffer().getInt(0);
+ result.getByteBuffer().limit(size);
+ result.setSize(size);
+ queryMessageResult.addMessage(result);
+ }
+ } else {
+ log.warn("queryMessage hash duplicate, {} {}", topic, key);
+ }
+ } catch (Exception e) {
+ log.error("queryMessage exception", e);
+ }
+ }
+
+ if (queryMessageResult.getBufferTotalSize() > 0) {
+ break;
+ }
+
+ if (lastQueryMsgTime < begin) {
+ break;
+ }
+ }
+
+ return queryMessageResult;
+ }
+
+ @Override
+ public void updateHaMasterAddress(String newAddr) {
+ this.haService.updateMasterAddress(newAddr);
+ }
+
+ @Override
+ public long slaveFallBehindMuch() {
+ return this.commitLog.getMaxOffset() - this.haService.getPush2SlaveMaxOffset().get();
+ }
+
+ @Override
+ public long now() {
+ return this.systemClock.now();
+ }
+
+ @Override
+ public int cleanUnusedTopic(Set<String> topics) {
+ Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = consumeQueueTable.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ String topic = next.getKey();
+
+ if (!topics.contains(topic) && !topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
+ for (ConsumeQueueInterface cq : queueTable.values()) {
+ this.consumeQueueStore.destroy(cq);
+ log.info("cleanUnusedTopic: {} {} ConsumeQueue cleaned",
+ cq.getTopic(),
+ cq.getQueueId()
+ );
+
+ this.commitLog.removeQueueFromTopicQueueTable(cq.getTopic(), cq.getQueueId());
+ }
+ it.remove();
+ log.info("cleanUnusedTopic: {},topic consumeQueue destroyed", topic);
+ }
+ }
+ return 0;
+ }
+
+ @Override
+ public void cleanExpiredConsumerQueue() {
+ long minCommitLogOffset = this.commitLog.getMinOffset();
+
+ Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ String topic = next.getKey();
+ if (!topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
+ Iterator<Map.Entry<Integer, ConsumeQueueInterface>> itQT = queueTable.entrySet().iterator();
+ while (itQT.hasNext()) {
+ Map.Entry<Integer, ConsumeQueueInterface> nextQT = itQT.next();
+ long maxCLOffsetInConsumeQueue = nextQT.getValue().getMaxPhysicOffset();
+
+ if (maxCLOffsetInConsumeQueue == -1) {
+ log.warn("maybe ConsumeQueue was created just now. topic={} queueId={} maxPhysicOffset={} minLogicOffset={}.",
+ nextQT.getValue().getTopic(),
+ nextQT.getValue().getQueueId(),
+ nextQT.getValue().getMaxPhysicOffset(),
+ nextQT.getValue().getMinLogicOffset());
+ } else if (maxCLOffsetInConsumeQueue < minCommitLogOffset) {
+ log.info(
+ "cleanExpiredConsumerQueue: {} {} consumer queue destroyed, minCommitLogOffset: {} maxCLOffsetInConsumeQueue: {}",
+ topic,
+ nextQT.getKey(),
+ minCommitLogOffset,
+ maxCLOffsetInConsumeQueue);
+
+ this.commitLog.removeQueueFromTopicQueueTable(nextQT.getValue().getTopic(),
+ nextQT.getValue().getQueueId());
+
+ this.consumeQueueStore.destroy(nextQT.getValue());
+ itQT.remove();
+ }
+ }
+
+ if (queueTable.isEmpty()) {
+ log.info("cleanExpiredConsumerQueue: {},topic destroyed", topic);
+ it.remove();
+ }
+ }
+ }
+ }
+
+ public double getDiskSpaceWarningLevelRatio() {
+ return cleanCommitLogService.getDiskSpaceWarningLevelRatio();
+ }
+
+ @Override
+ public boolean checkInDiskByConsumeOffset(final String topic, final int queueId, long consumeOffset) {
+
+ final long maxOffsetPy = this.commitLog.getMaxOffset();
+
+ ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
+ if (consumeQueue != null) {
+ CqUnit cqUnit = consumeQueue.get(consumeOffset);
+
+ if (cqUnit != null) {
+ long offsetPy = cqUnit.getPos();
+ return checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
+ } else {
+ return false;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public long dispatchBehindBytes() {
+ return this.reputMessageService.behind();
+ }
+
+ @Override
+ public long flush() {
+ return this.commitLog.flush();
+ }
+
+ @Override
+ public boolean resetWriteOffset(long phyOffset) {
+ return this.commitLog.resetOffset(phyOffset);
+ }
+
+ @Override
+ public long getConfirmOffset() {
+ return this.commitLog.getConfirmOffset();
+ }
+
+ @Override
+ public void setConfirmOffset(long phyOffset) {
+ this.commitLog.setConfirmOffset(phyOffset);
+ }
+
+ @Override
+ public MessageExt lookMessageByOffset(long commitLogOffset, int size) {
+ SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, size);
+ if (null != sbr) {
+ try {
+ return MessageDecoder.decode(sbr.getByteBuffer(), true, false);
+ } finally {
+ sbr.release();
+ }
+ }
+
+ return null;
+ }
+
+ private long nextOffsetCorrection(long oldOffset, long newOffset) {
+ long nextOffset = oldOffset;
+ if (this.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE || this.getMessageStoreConfig().isOffsetCheckInSlave()) {
+ nextOffset = newOffset;
+ }
+ return nextOffset;
+ }
+
+ private boolean checkInDiskByCommitOffset(long offsetPy, long maxOffsetPy) {
+ long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
+ return (maxOffsetPy - offsetPy) > memory;
+ }
+
+ private boolean isTheBatchFull(int sizePy, int unitBatchNum, int maxMsgNums, long maxMsgSize, int bufferTotal,
+ int messageTotal, boolean isInDisk) {
+
+ //At least has one message(batch)
+ if (0 == bufferTotal || 0 == messageTotal) {
+ return false;
+ }
+
+ if (messageTotal + unitBatchNum > maxMsgNums) {
+ return true;
+ }
+
+ if (bufferTotal + sizePy > maxMsgSize) {
+ return true;
+ }
+
+ if (isInDisk) {
+ if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInDisk()) {
+ return true;
+ }
+ if (messageTotal + unitBatchNum > this.messageStoreConfig.getMaxTransferCountOnMessageInDisk()) {
+ return true;
+ }
+ } else {
+ if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInMemory()) {
+ return true;
+ }
+
+ if (messageTotal + unitBatchNum > this.messageStoreConfig.getMaxTransferCountOnMessageInMemory()) {
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private void deleteFile(final String fileName) {
+ File file = new File(fileName);
+ boolean result = file.delete();
+ log.info(fileName + (result ? " delete OK" : " delete Failed"));
+ }
+
+ /**
+ * @throws IOException
+ */
+ private void createTempFile() throws IOException {
+ String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
+ File file = new File(fileName);
+ DefaultMappedFile.ensureDirOK(file.getParent());
+ boolean result = file.createNewFile();
+ log.info(fileName + (result ? " create OK" : " already exists"));
+ }
+
+ private void addScheduleTask() {
+
+ this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ @Override
+ public void run() {
+ StreamMessageStore.this.cleanFilesPeriodically();
+ }
+ }, 1000 * 60, this.messageStoreConfig.getCleanResourceInterval(), TimeUnit.MILLISECONDS);
+
+ this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ @Override
+ public void run() {
+ StreamMessageStore.this.checkSelf();
+ }
+ }, 1, 10, TimeUnit.MINUTES);
+
+ this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ if (!getMessageStoreConfig().isMappedFileSwapEnable()) {
+ log.warn("Swap is not enabled.");
+ return ;
+ }
+ StreamMessageStore.this.commitLog.swapMap(getMessageStoreConfig().getCommitLogSwapMapReserveFileNum(),
+ getMessageStoreConfig().getCommitLogForceSwapMapInterval(), getMessageStoreConfig().getCommitLogSwapMapInterval());
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : StreamMessageStore.this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ StreamMessageStore.this.consumeQueueStore.swapMap(logic, getMessageStoreConfig().getLogicQueueSwapMapReserveFileNum(),
+ getMessageStoreConfig().getLogicQueueForceSwapMapInterval(), getMessageStoreConfig().getLogicQueueSwapMapInterval());
+ }
+ }
+ } catch (Exception e) {
+ log.error("swap map exception", e);
+ }
+ }
+ }, 1, 5, TimeUnit.MINUTES);
+
+ this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ StreamMessageStore.this.commitLog.cleanSwappedMap(getMessageStoreConfig().getCleanSwapedMapInterval());
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : StreamMessageStore.this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ StreamMessageStore.this.consumeQueueStore.cleanSwappedMap(logic, getMessageStoreConfig().getCleanSwapedMapInterval());
+ }
+ }
+ } catch (Exception e) {
+ log.error("clean swap map exception", e);
+ }
+ }
+ }, 1, 5, TimeUnit.MINUTES);
+
+ this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ @Override
+ public void run() {
+ if (StreamMessageStore.this.getMessageStoreConfig().isDebugLockEnable()) {
+ try {
+ if (StreamMessageStore.this.commitLog.getBeginTimeInLock() != 0) {
+ long lockTime = System.currentTimeMillis() - StreamMessageStore.this.commitLog.getBeginTimeInLock();
+ if (lockTime > 1000 && lockTime < 10000000) {
+
+ String stack = UtilAll.jstack();
+ final String fileName = System.getProperty("user.home") + File.separator + "debug/lock/stack-"
+ + StreamMessageStore.this.commitLog.getBeginTimeInLock() + "-" + lockTime;
+ MixAll.string2FileNotSafe(stack, fileName);
+ }
+ }
+ } catch (Exception e) {
+ }
+ }
+ }
+ }, 1, 1, TimeUnit.SECONDS);
+
+ // this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
+ // @Override
+ // public void run() {
+ // StreamMessageStore.this.cleanExpiredConsumerQueue();
+ // }
+ // }, 1, 1, TimeUnit.HOURS);
+ }
+
+ private void cleanFilesPeriodically() {
+ this.cleanCommitLogService.run();
+ this.cleanConsumeQueueService.run();
+ this.correctLogicOffsetService.run();
+ }
+
+ private void checkSelf() {
+ this.commitLog.checkSelf();
+
+ Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ Iterator<Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
+ while (itNext.hasNext()) {
+ Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
+ this.consumeQueueStore.checkSelf(cq.getValue());
+ }
+ }
+ }
+
+ private boolean isTempFileExist() {
+ String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
+ File file = new File(fileName);
+ return file.exists();
+ }
+
+ protected boolean loadBatchConsumeQueue() {
+ checkOtherConsumeQueue();
+
+ File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ File[] fileTopicList = dirLogic.listFiles();
+ if (fileTopicList != null) {
+
+ for (File fileTopic : fileTopicList) {
+ String topic = fileTopic.getName();
+
+ File[] fileQueueIdList = fileTopic.listFiles();
+ if (fileQueueIdList != null) {
+ for (File fileQueueId : fileQueueIdList) {
+ int queueId;
+ try {
+ queueId = Integer.parseInt(fileQueueId.getName());
+ } catch (NumberFormatException e) {
+ continue;
+ }
+ ConsumeQueueInterface logic = new BatchConsumeQueue(
+ topic,
+ queueId,
+ StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ this.getMessageStoreConfig().getMapperFileSizeBatchConsumeQueue(),
+ this);
+ this.putConsumeQueue(topic, queueId, logic);
+ if (!this.consumeQueueStore.load(logic)) {
+ return false;
+ }
+ }
+ }
+ }
+ }
+
+ log.info("load logics queue all over, OK");
+
+ return true;
+ }
+
+ private void checkOtherConsumeQueue() {
+ File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ if (dirLogic.exists()) {
+ throw new RuntimeException(format("Consume queue directory: [%s] exist. Can not load batch consume queue while consume queue exists.",
+ StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir())));
+ }
+ }
+
+ private void recover(final boolean lastExitOK) {
+ long recoverCqStart = System.currentTimeMillis();
+ long maxPhyOffsetOfConsumeQueue = this.recoverConsumeQueue();
+ long recoverCqEnd = System.currentTimeMillis();
+
+ if (lastExitOK) {
+ this.commitLog.recoverNormally(maxPhyOffsetOfConsumeQueue);
+ } else {
+ this.commitLog.recoverAbnormally(maxPhyOffsetOfConsumeQueue);
+ }
+ long recoverClogEnd = System.currentTimeMillis();
+ this.recoverTopicQueueTable();
+ long recoverOffsetEnd = System.currentTimeMillis();
+
+ log.info("Recover end total:{} recoverCq:{} recoverClog:{} recoverOffset:{}",
+ recoverOffsetEnd - recoverCqStart, recoverCqEnd - recoverCqStart, recoverClogEnd - recoverCqEnd, recoverOffsetEnd - recoverClogEnd);
+ }
+
+ @Override
+ public MessageStoreConfig getMessageStoreConfig() {
+ return messageStoreConfig;
+ }
+
+ @Override
+ public TransientStorePool getTransientStorePool() {
+ return transientStorePool;
+ }
+
+ private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueueInterface consumeQueue) {
+ ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface> map = this.consumeQueueTable.get(topic);
+ if (null == map) {
+ map = new ConcurrentHashMap<Integer/* queueId */, ConsumeQueueInterface>();
+ map.put(queueId, consumeQueue);
+ this.consumeQueueTable.put(topic, map);
+ } else {
+ map.put(queueId, consumeQueue);
+ }
+ }
+
+ private long recoverConsumeQueue() {
+ long maxPhysicOffset = -1;
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.recover(logic);
+ if (logic.getMaxPhysicOffset() > maxPhysicOffset) {
+ maxPhysicOffset = logic.getMaxPhysicOffset();
+ }
+ }
+ }
+
+ return maxPhysicOffset;
+ }
+
+ public void recoverTopicQueueTable() {
+ HashMap<String/* topic-queueid */, Long/* offset */> table = new HashMap<String, Long>(1024);
+ long minPhyOffset = this.commitLog.getMinOffset();
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ String key = logic.getTopic() + "-" + logic.getQueueId();
+ table.put(key, logic.getMaxOffsetInQueue());
+ this.consumeQueueStore.correctMinOffset(logic, minPhyOffset);
+ }
+ }
+
+ this.batchTopicQueueTable = table;
+ }
+
+ @Override
+ public AllocateMappedFileService getAllocateMappedFileService() {
+ return allocateMappedFileService;
+ }
+
+ @Override
+ public void truncateDirtyLogicFiles(long phyOffset) {
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
+
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.consumeQueueStore.truncateDirtyLogicFiles(logic, phyOffset);
+ }
+ }
+ }
+
+ @Override
+ public StoreStatsService getStoreStatsService() {
+ return storeStatsService;
+ }
+
+ public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> getConsumeQueueTable() {
+ return consumeQueueTable;
+ }
+
+ @Override
+ public StoreCheckpoint getStoreCheckpoint() {
+ return storeCheckpoint;
+ }
+
+ @Override
+ public HAService getHaService() {
+ return haService;
+ }
+
+ @Override
+ public void registerCleanFileHook(CleanFilesHook logicalQueueCleanHook) {
+
+ }
+
+ @Override
+ public ScheduleMessageService getScheduleMessageService() {
+ return scheduleMessageService;
+ }
+
+ @Override
+ public RunningFlags getRunningFlags() {
+ return runningFlags;
+ }
+
+ public void initAsyncReputThreads(int tsNum, int cacheNum) {
+ if (tsNum <= 0) {
+ tsNum = 1;
+ }
+ if (cacheNum < 512) {
+ cacheNum = 512;
+ }
+ reputExecutors = new ThreadPoolExecutor[tsNum];
+ reputQueues = new BlockingQueue[tsNum];
+
+ for (int i = 0; i < tsNum; i++) {
+ final int tmp = i;
+ reputQueues[i] = new LinkedBlockingDeque<>(cacheNum);
+ //Each executor can only have one thread, otherwise the cq index will get wrong
+ reputExecutors[i] = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
+ reputQueues[i],
+ new ThreadFactory() {
+ @Override
+ public Thread newThread(Runnable r) {
+ return new Thread(r, "MQDispatchThread-" + tmp);
+ }
+ });
+ }
+ for (ThreadPoolExecutor executorService: reputExecutors) {
+ if (executorService.getMaximumPoolSize() != 1 ||
+ executorService.getCorePoolSize() != 1) {
+ throw new RuntimeException("The MQDispatchThreadPoll can only have one thread");
+ }
+ }
+
+ }
+
+ public Future doDispatch(final DispatchRequest request) {
+ return doDispatch(request, false);
+ }
+
+ public Future doDispatch(final DispatchRequest request, boolean async) {
+ Runnable task = new Runnable() {
+ @Override
+ public void run() {
+ for (CommitLogDispatcher dispatcher : StreamMessageStore.this.dispatcherList) {
+ dispatcher.dispatch(request);
+ }
+ }
+ };
+ if (!async) {
+ task.run();
+ return EMPTY_FUTURE;
+ }
+ int hash = Math.abs((request.getTopic() + request.getQueueId()).hashCode());
+ int slot = hash % reputExecutors.length;
+ try {
+ return reputExecutors[slot].submit(task);
+ } catch (RejectedExecutionException ignored) {
+ int tryNum = 0;
+ while (tryNum++ < Integer.MAX_VALUE) {
+ try {
+ Thread.sleep(1);
+ } catch (Throwable ignored2) {
+
+ }
+ try {
+ return reputExecutors[slot].submit(task);
+ } catch (RejectedExecutionException e) {
+ log.warn("DispatchReject topic:{} queue:{} pyOffset:{} tryNum:{}", request.getTopic(), request.getQueueId(), request.getCommitLogOffset(), tryNum, e);
+ }
+ }
+ }
+ return EMPTY_FUTURE;
+ }
+
+ public void syncProcessDispatchRequest(DispatchRequest request, boolean isRecover) throws InterruptedException {
+ if (!isDispatchFromSenderThread) {
+ log.error("addDispatchRequestQueue operation not supported while isCreateDispatchRequestAsync is true");
+ } else {
+ if (isRecover) {
+ ((SyncReputMessageService) this.reputMessageService).processDispatchRequestForRecover(request);
+ } else {
+ ((SyncReputMessageService) this.reputMessageService).processDispatchRequest(request);
+ }
+ }
+ }
+
+ public boolean dispatched(long physicalOffset) {
+ return reputMessageService.dispatched(physicalOffset);
+ }
+
+ public void putMessagePositionInfo(DispatchRequest dispatchRequest) {
+ ConsumeQueueInterface cq = this.findOrCreateConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
+ this.consumeQueueStore.putMessagePositionInfoWrapper(cq, dispatchRequest);
+ }
+
+ private ConsumeQueueInterface findOrCreateConsumeQueue(String topic, int queueId) {
+ return this.consumeQueueStore.findOrCreateConsumeQueue(topic, queueId);
+ }
+
+ @Override
+ public BrokerStatsManager getBrokerStatsManager() {
+ return brokerStatsManager;
+ }
+
+ @Override
+ public void handleScheduleMessageService(final BrokerRole brokerRole) {
+ if (this.scheduleMessageService != null) {
+ if (brokerRole == BrokerRole.SLAVE) {
+ this.scheduleMessageService.shutdown();
+ } else {
+ this.scheduleMessageService.start();
+ }
+ }
+
+ }
+
+ public int remainTransientStoreBufferNumbs() {
+ return this.transientStorePool.availableBufferNums();
+ }
+
+ @Override
+ public boolean isTransientStorePoolDeficient() {
+ return remainTransientStoreBufferNumbs() == 0;
+ }
+
+ @Override
+ public LinkedList<CommitLogDispatcher> getDispatcherList() {
+ return this.dispatcherList;
+ }
+
+ @Override
+ public ConsumeQueueInterface getConsumeQueue(String topic, int queueId) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
+ if (map == null) {
+ return null;
+ }
+ return map.get(queueId);
+ }
+
+ @Override
+ public void unlockMappedFile(final MappedFile mappedFile) {
+ this.scheduledExecutorService.schedule(new Runnable() {
+ @Override
+ public void run() {
+ mappedFile.munlock();
+ }
+ }, 6, TimeUnit.SECONDS);
+ }
+
+ @Override
+ public PerfCounter.Ticks getPerfCounter() {
+ return perfs;
+ }
+
+ @Override
+ public ConsumeQueueStore getQueueStore() {
+ return consumeQueueStore;
+ }
+
+ @Override
+ public boolean isSyncDiskFlush() {
+ return FlushDiskType.SYNC_FLUSH == this.getMessageStoreConfig().getFlushDiskType();
+ }
+
+ @Override
+ public boolean isSyncMaster() {
+ return BrokerRole.SYNC_MASTER == this.getMessageStoreConfig().getBrokerRole();
+ }
+
+ @Override
+ public void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
+ final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
+
+ if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
+ assignOffsetForCq(topicQueueKey, msg);
+ assignOffsetForBcq(topicQueueKey, msg, batchNum);
+ }
+ }
+
+ private void assignOffsetForCq(String topicQueueKey, MessageExtBrokerInner msg) {
+ // not supported yet
+ }
+
+ private void assignOffsetForBcq(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
+ Long batchTopicOffset = this.batchTopicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ CQType cqType = QueueTypeUtils.getCQType(this);
+ if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG) || CQType.BatchCQ.equals(cqType)) {
+ MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_BASE, String.valueOf(batchTopicOffset));
+ msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
+ }
+ msg.setQueueOffset(batchTopicOffset);
+ this.batchTopicQueueTable.put(topicQueueKey, batchTopicOffset + batchNum);
+ }
+
+ @Override
+ public void removeOffsetTable(String topicQueueKey) {
+ this.topicQueueTable.remove(topicQueueKey);
+ this.batchTopicQueueTable.remove(topicQueueKey);
+ }
+
+ @Override
+ public void onCommitLogAppend(MessageExtBrokerInner msg, AppendMessageResult result, MappedFile commitLogFile) {
+ DispatchRequest dispatchRequest;
+ switch (result.getStatus()) {
+ case PUT_OK:
+ dispatchRequest = constructDispatchRequest(msg, result);
+ onCommitLogDispatch(dispatchRequest, this.isDispatchFromSenderThread(), commitLogFile, false, false);
+ break;
+ case END_OF_FILE:
+ dispatchRequest = new DispatchRequest(0, true);
+ onCommitLogDispatch(dispatchRequest, this.isDispatchFromSenderThread(), commitLogFile, false, true);
+ break;
+ default:
+ throw new RuntimeException("");
+ }
+ }
+
+ @Override
+ public void onCommitLogDispatch(DispatchRequest dispatchRequest, boolean doDispatch, MappedFile commitLogFile, boolean isRecover, boolean isFileEnd) {
+ if (isFileEnd) {
+ if (doDispatch) {
+ long nextReputFromOffset = this.getCommitLog().rollNextFile(commitLogFile.getFileFromOffset());
+ dispatchRequest.setNextReputFromOffset(nextReputFromOffset);
+ syncDispatch(dispatchRequest, isRecover);
+ }
+ } else {
+ if (doDispatch) {
+ dispatchRequest.setNextReputFromOffset(dispatchRequest.getCommitLogOffset() + dispatchRequest.getMsgSize());
+ syncDispatch(dispatchRequest, isRecover);
+ }
+ }
+ }
+
+ private DispatchRequest constructDispatchRequest(MessageExtBrokerInner msg, AppendMessageResult appendResult) {
+ long tagsCode = 0;
+ String keys = "";
+ String uniqKey = null;
+ int sysFlag = msg.getSysFlag();
+ String topic = msg.getTopic();
+ long storeTimestamp = msg.getStoreTimestamp();
+ int queueId = msg.getQueueId();
+ long preparedTransactionOffset = msg.getPreparedTransactionOffset();
+ Map<String, String> propertiesMap = msg.getProperties();
+
+ if (msg.getProperties() != null && msg.getProperties().size() > 0) {
+
+ keys = propertiesMap.get(MessageConst.PROPERTY_KEYS);
+
+ uniqKey = propertiesMap.get(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX);
+
+ String tags = propertiesMap.get(MessageConst.PROPERTY_TAGS);
+ if (tags != null && tags.length() > 0) {
+ tagsCode = MessageExtBrokerInner.tagsString2tagsCode(MessageExt.parseTopicFilterType(sysFlag), tags);
+ }
+
+ // Timing message processing
+ {
+ String t = propertiesMap.get(MessageConst.PROPERTY_DELAY_TIME_LEVEL);
+ if (TopicValidator.RMQ_SYS_SCHEDULE_TOPIC.equals(topic) && t != null) {
+ int delayLevel = Integer.parseInt(t);
+
+ if (delayLevel > this.getScheduleMessageService().getMaxDelayLevel()) {
+ delayLevel = this.getScheduleMessageService().getMaxDelayLevel();
+ }
+
+ if (delayLevel > 0) {
+ tagsCode = this.getScheduleMessageService().computeDeliverTimestamp(delayLevel,
+ storeTimestamp);
+ }
+ }
+ }
+ }
+
+ DispatchRequest dispatchRequest = new DispatchRequest(
+ topic,
+ queueId,
+ appendResult.getWroteOffset(),
+ appendResult.getWroteBytes(),
+ tagsCode,
+ storeTimestamp,
+ appendResult.getLogicsOffset(),
+ keys,
+ uniqKey,
+ sysFlag,
+ preparedTransactionOffset,
+ propertiesMap
+ );
+
+ if (null != propertiesMap && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_NUM) && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_BASE)) {
+ dispatchRequest.setMsgBaseOffset(Long.parseLong(propertiesMap.get(MessageConst.PROPERTY_INNER_BASE)));
+ dispatchRequest.setBatchSize(Short.parseShort(propertiesMap.get(MessageConst.PROPERTY_INNER_NUM)));
+ }
+ return dispatchRequest;
+ }
+
+ private void syncDispatch(DispatchRequest dispatchRequest, boolean isRecover) {
+ try {
+ this.syncProcessDispatchRequest(dispatchRequest, isRecover);
+ } catch (InterruptedException e) {
+ log.error("OnCommitlogAppend sync dispatch failed, addDispatchRequestQueue interrupted. DispatchRequest:{}", dispatchRequest);
+ }
+ }
+
+ class CommitLogDispatcherBuildConsumeQueue implements CommitLogDispatcher {
+
+ @Override
+ public void dispatch(DispatchRequest dispatchRequest) {
+ final int tranType = MessageSysFlag.getTransactionValue(dispatchRequest.getSysFlag());
+ switch (tranType) {
+ case MessageSysFlag.TRANSACTION_NOT_TYPE:
+ case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
+ StreamMessageStore.this.putMessagePositionInfo(dispatchRequest);
+ if (BrokerRole.SLAVE != StreamMessageStore.this.getMessageStoreConfig().getBrokerRole()
+ && StreamMessageStore.this.brokerConfig.isLongPollingEnable()) {
+ StreamMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
+ dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
+ dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
+ dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
+ }
+ if (StreamMessageStore.this.getMessageStoreConfig().getBrokerRole() == BrokerRole.SLAVE) {
+ StreamMessageStore.this.storeStatsService
+ .getSinglePutMessageTopicTimesTotal(dispatchRequest.getTopic()).add(1);
+ StreamMessageStore.this.storeStatsService
+ .getSinglePutMessageTopicSizeTotal(dispatchRequest.getTopic())
+ .add(dispatchRequest.getMsgSize());
+ }
+ break;
+ case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
+ case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
+ break;
+ }
+ }
+ }
+
+ class CommitLogDispatcherBuildIndex implements CommitLogDispatcher {
+
+ @Override
+ public void dispatch(DispatchRequest request) {
+ if (StreamMessageStore.this.messageStoreConfig.isMessageIndexEnable()) {
+ StreamMessageStore.this.indexService.buildIndex(request);
+ }
+ }
+ }
+
+ class CleanCommitLogService {
+
+ private final static int MAX_MANUAL_DELETE_FILE_TIMES = 20;
+ private final String diskSpaceWarningLevelRatio =
+ System.getProperty("rocketmq.broker.diskSpaceWarningLevelRatio", "");
+
+ private final String diskSpaceCleanForciblyRatio =
+ System.getProperty("rocketmq.broker.diskSpaceCleanForciblyRatio", "");
+ private long lastRedeleteTimestamp = 0;
+
+ private volatile int manualDeleteFileSeveralTimes = 0;
+
+ private volatile boolean cleanImmediately = false;
+
+ double getDiskSpaceWarningLevelRatio() {
+ double finalDiskSpaceWarningLevelRatio;
+ if ("".equals(diskSpaceWarningLevelRatio)) {
+ finalDiskSpaceWarningLevelRatio = StreamMessageStore.this.getMessageStoreConfig().getDiskSpaceWarningLevelRatio() / 100.0;
+ } else {
+ finalDiskSpaceWarningLevelRatio = Double.parseDouble(diskSpaceWarningLevelRatio);
+ }
+
+ if (finalDiskSpaceWarningLevelRatio > 0.90) {
+ finalDiskSpaceWarningLevelRatio = 0.90;
+ }
+ if (finalDiskSpaceWarningLevelRatio < 0.35) {
+ finalDiskSpaceWarningLevelRatio = 0.35;
+ }
+
+ return finalDiskSpaceWarningLevelRatio;
+ }
+
+ double getDiskSpaceCleanForciblyRatio() {
+ double finalDiskSpaceCleanForciblyRatio;
+ if ("".equals(diskSpaceCleanForciblyRatio)) {
+ finalDiskSpaceCleanForciblyRatio = StreamMessageStore.this.getMessageStoreConfig().getDiskSpaceCleanForciblyRatio() / 100.0;
+ } else {
+ finalDiskSpaceCleanForciblyRatio = Double.parseDouble(diskSpaceCleanForciblyRatio);
+ }
+
+ if (finalDiskSpaceCleanForciblyRatio > 0.85) {
+ finalDiskSpaceCleanForciblyRatio = 0.85;
+ }
+ if (finalDiskSpaceCleanForciblyRatio < 0.30) {
+ finalDiskSpaceCleanForciblyRatio = 0.30;
+ }
+
+ return finalDiskSpaceCleanForciblyRatio;
+ }
+
+ public void excuteDeleteFilesManualy() {
+ this.manualDeleteFileSeveralTimes = MAX_MANUAL_DELETE_FILE_TIMES;
+ StreamMessageStore.log.info("executeDeleteFilesManually was invoked");
+ }
+
+ public long run() {
+ int deleteCount = 0;
+ try {
+ deleteCount = this.deleteExpiredFiles();
+
+ this.redeleteHangedFile();
+ } catch (Throwable e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ return deleteCount;
+ }
+
+ private int deleteExpiredFiles() {
+ int deleteCount = 0;
+ long fileReservedTime = StreamMessageStore.this.getMessageStoreConfig().getFileReservedTime();
+ int deletePhysicFilesInterval = StreamMessageStore.this.getMessageStoreConfig().getDeleteCommitLogFilesInterval();
+ int destroyMapedFileIntervalForcibly = StreamMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly();
+ int maxBatchDeleteFilesNum = StreamMessageStore.this.getMessageStoreConfig().getMaxBatchDeleteFilesNum();
+ if (maxBatchDeleteFilesNum < 10) {
+ maxBatchDeleteFilesNum = 10;
+ }
+
+ boolean timeup = this.isTimeToDelete();
+ boolean spacefull = this.isSpaceToDelete();
+ boolean manualDelete = this.manualDeleteFileSeveralTimes > 0;
+
+ boolean needDelete = timeup || spacefull || manualDelete;
+
+ if (needDelete) {
+
+ if (manualDelete)
+ this.manualDeleteFileSeveralTimes--;
+
+ boolean cleanAtOnce = StreamMessageStore.this.getMessageStoreConfig().isCleanFileForciblyEnable() && this.cleanImmediately;
+
+ String storePathPhysic = StreamMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
+ double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
+ long totalSpace = UtilAll.getTotalSpace(storePathPhysic);
+ double realRatio = (StreamMessageStore.this.commitLog.getMaxOffset() - StreamMessageStore.this.commitLog.getMinOffset()) / (totalSpace + 0.001);
+
+ cleanAtOnce = cleanAtOnce && (realRatio > 0.3);
+
+ log.info("begin to delete before {} hours file. timeup:{} spacefull:{} manualDeleteFileSeveralTimes:{} cleanAtOnce:{} maxBatchDeleteFilesNum:{} physicRatio:{}",
+ fileReservedTime,
+ timeup,
+ spacefull,
+ manualDeleteFileSeveralTimes,
+ cleanAtOnce,
+ maxBatchDeleteFilesNum,
+ physicRatio);
+
+ fileReservedTime *= 60 * 60 * 1000;
+
+ deleteCount = StreamMessageStore.this.commitLog.deleteExpiredFile(fileReservedTime, deletePhysicFilesInterval,
+ destroyMapedFileIntervalForcibly, cleanAtOnce);
+ if (deleteCount > 0) {
+ } else if (spacefull) {
+ log.warn("disk space will be full soon, but delete file failed. timeup:{} manualDelete:{} cleanAtOnce:{} physicRatio:{}",
+ timeup,
+ manualDelete,
+ cleanAtOnce,
+ physicRatio);
+ }
+ }
+ return deleteCount;
+ }
+
+ private void redeleteHangedFile() {
+ int interval = StreamMessageStore.this.getMessageStoreConfig().getRedeleteHangedFileInterval();
+ long currentTimestamp = System.currentTimeMillis();
+ if ((currentTimestamp - this.lastRedeleteTimestamp) > interval) {
+ this.lastRedeleteTimestamp = currentTimestamp;
+ int destroyMapedFileIntervalForcibly =
+ StreamMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly();
+ if (StreamMessageStore.this.commitLog.retryDeleteFirstFile(destroyMapedFileIntervalForcibly)) {
+ }
+ }
+ }
+
+ public String getServiceName() {
+ return CleanCommitLogService.class.getSimpleName();
+ }
+
+ private boolean isTimeToDelete() {
+ String when = StreamMessageStore.this.getMessageStoreConfig().getDeleteWhen();
+ if (UtilAll.isItTimeToDo(when)) {
+ StreamMessageStore.log.info("it's time to reclaim disk space, " + when);
+ return true;
+ }
+
+ return false;
+ }
+
+ private boolean isSpaceToDelete() {
+ double ratio = StreamMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 100.0;
+
+ cleanImmediately = false;
+
+ {
+ double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(getStorePathPhysic());
+ if (physicRatio > getDiskSpaceWarningLevelRatio()) {
+ boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskFull();
+ if (diskok) {
+ StreamMessageStore.log.error("physic disk maybe full soon " + physicRatio + ", so mark disk full");
+ }
+
+ cleanImmediately = true;
+ } else if (physicRatio > getDiskSpaceCleanForciblyRatio()) {
+ cleanImmediately = true;
+ } else {
+ boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskOK();
+ if (!diskok) {
+ StreamMessageStore.log.info("physic disk space OK " + physicRatio + ", so mark disk ok");
+ }
+ }
+
+ if (physicRatio < 0 || physicRatio > ratio) {
+ StreamMessageStore.log.info("physic disk maybe full soon, so reclaim space, {}, cleanImmediately {}", physicRatio, cleanImmediately);
+ return true;
+ }
+ }
+
+ {
+ String storePathLogics = StorePathConfigHelper
+ .getStorePathConsumeQueue(StreamMessageStore.this.getMessageStoreConfig().getStorePathRootDir());
+ double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
+ if (logicsRatio > getDiskSpaceWarningLevelRatio()) {
+ boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskFull();
+ if (diskok) {
+ StreamMessageStore.log.error("logics disk maybe full soon " + logicsRatio + ", so mark disk full");
+ }
+
+ cleanImmediately = true;
+ } else if (logicsRatio > getDiskSpaceCleanForciblyRatio()) {
+ cleanImmediately = true;
+ } else {
+ boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskOK();
+ if (!diskok) {
+ StreamMessageStore.log.info("logics disk space OK " + logicsRatio + ", so mark disk ok");
+ }
+ }
+
+ if (logicsRatio < 0 || logicsRatio > ratio) {
+ StreamMessageStore.log.info("logics disk maybe full soon, so reclaim space, " + logicsRatio);
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ public int getManualDeleteFileSeveralTimes() {
+ return manualDeleteFileSeveralTimes;
+ }
+
+ public void setManualDeleteFileSeveralTimes(int manualDeleteFileSeveralTimes) {
+ this.manualDeleteFileSeveralTimes = manualDeleteFileSeveralTimes;
+ }
+ }
+
+ class CleanConsumeQueueService {
+ private long lastPhysicalMinOffset = 0;
+
+ public void run() {
+ try {
+ this.deleteExpiredFiles();
+ } catch (Throwable e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ private void deleteExpiredFiles() {
+ int deleteLogicsFilesInterval = StreamMessageStore.this.getMessageStoreConfig().getDeleteConsumeQueueFilesInterval();
+
+ long minOffset = StreamMessageStore.this.commitLog.getMinOffset();
+ if (minOffset > this.lastPhysicalMinOffset) {
+ this.lastPhysicalMinOffset = minOffset;
+
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
+
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ int deleteCount = StreamMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minOffset);
+
+ if (deleteCount > 0 && deleteLogicsFilesInterval > 0) {
+ try {
+ Thread.sleep(deleteLogicsFilesInterval);
+ } catch (InterruptedException ignored) {
+ }
+ }
+ }
+ }
+
+ StreamMessageStore.this.indexService.deleteExpiredFile(minOffset);
+ }
+ }
+
+ public String getServiceName() {
+ return CleanConsumeQueueService.class.getSimpleName();
+ }
+ }
+
+ class CorrectLogicOffsetService {
+
+ private long lastForceCorrectTime = -1L;
+
+ public void run() {
+ try {
+ this.correctLogicMinOffset();
+ } catch (Throwable e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ private boolean needCorrect(ConsumeQueueInterface logic, long minPhyOffset, long lastForeCorrectTimeCurRun) {
+ if (logic == null) {
+ return false;
+ }
+ // If first exist and not available, it means first file may destroy failed, delete it.
+ if (StreamMessageStore.this.consumeQueueStore.isFirstFileExist(logic) && !StreamMessageStore.this.consumeQueueStore.isFirstFileAvailable(logic)) {
+ log.error("CorrectLogicOffsetService.needCorrect. first file not available, trigger correct." +
+ " topic:{}, queue:{}, maxPhyOffset in queue:{}, minPhyOffset " +
+ "in commit log:{}, minOffset in queue:{}, maxOffset in queue:{}, cqType:{}"
+ , logic.getTopic(), logic.getQueueId(), logic.getMaxPhysicOffset()
+ , minPhyOffset, logic.getMinOffsetInQueue(), logic.getMaxOffsetInQueue(), logic.getCQType());
+ return true;
+ }
+
+ // logic.getMaxPhysicOffset() or minPhyOffset = -1
+ // means there is no message in current queue, so no need to correct.
+ if (logic.getMaxPhysicOffset() == -1 || minPhyOffset == -1) {
+ return false;
+ }
+
+ if (logic.getMaxPhysicOffset() < minPhyOffset) {
+ if (logic.getMinOffsetInQueue() < logic.getMaxOffsetInQueue()) {
+ log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is less than min phy offset: {}, " +
+ "but min offset: {} is less than max offset: {}. topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ } else if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
+ return false;
+ } else {
+ log.error("CorrectLogicOffsetService.needCorrect. It should not happen, logic max phy offset: {} is less than min phy offset: {}," +
+ " but min offset: {} is larger than max offset: {}. topic:{}, queue:{}, cqType:{}"
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return false;
+ }
+ }
+ //the logic.getMaxPhysicOffset() >= minPhyOffset
+ int forceCorrectInterval = StreamMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetForceInterval();
+ if ((System.currentTimeMillis() - lastForeCorrectTimeCurRun) > forceCorrectInterval) {
+ lastForceCorrectTime = System.currentTimeMillis();
+ CqUnit cqUnit = logic.getEarliestUnit();
+ if (cqUnit == null) {
+ if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
+ return false;
+ } else {
+ log.error("CorrectLogicOffsetService.needCorrect. cqUnit is null, logic max phy offset: {} is greater than min phy offset: {}, " +
+ "but min offset: {} is not equal to max offset: {}. topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ }
+ }
+
+ if (cqUnit.getPos() < minPhyOffset) {
+ log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is greater than min phy offset: {}, " +
+ "but minPhyPos in cq is: {}. min offset in queue: {}, max offset in queue: {}, topic:{}, queue:{}, cqType:{}."
+ , logic.getMaxPhysicOffset(), minPhyOffset, cqUnit.getPos(), logic.getMinOffsetInQueue()
+ , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
+ return true;
+ }
+
+ if (cqUnit.getPos() >= minPhyOffset) {
+
+ // Normal case, do not need correct.
+ return false;
+ }
+ }
+
+ return false;
+ }
+
+ private void correctLogicMinOffset() {
+
+ long lastForeCorrectTimeCurRun = lastForceCorrectTime;
+ long minPhyOffset = getMinPhyOffset();
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ if (needCorrect(logic, minPhyOffset, lastForeCorrectTimeCurRun)) {
+ doCorrect(logic, minPhyOffset);
+ }
+ }
+ }
+ }
+
+ private void doCorrect(ConsumeQueueInterface logic, long minPhyOffset) {
+ StreamMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minPhyOffset);
+ int sleepIntervalWhenCorrectMinOffset = StreamMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetSleepInterval();
+ if (sleepIntervalWhenCorrectMinOffset > 0) {
+ try {
+ Thread.sleep(sleepIntervalWhenCorrectMinOffset);
+ } catch (InterruptedException ignored) {
+ }
+ }
+ }
+
+ public String getServiceName() {
+ return CorrectLogicOffsetService.class.getSimpleName();
+ }
+ }
+
+ class FlushConsumeQueueService extends ServiceThread {
+ private static final int RETRY_TIMES_OVER = 3;
+ private long lastFlushTimestamp = 0;
+
+ private void doFlush(int retryTimes) {
+ int flushConsumeQueueLeastPages = StreamMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueLeastPages();
+
+ if (retryTimes == RETRY_TIMES_OVER) {
+ flushConsumeQueueLeastPages = 0;
+ }
+
+ long logicsMsgTimestamp = 0;
+
+ int flushConsumeQueueThoroughInterval = StreamMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueThoroughInterval();
+ long currentTimeMillis = System.currentTimeMillis();
+ if (currentTimeMillis >= (this.lastFlushTimestamp + flushConsumeQueueThoroughInterval)) {
+ this.lastFlushTimestamp = currentTimeMillis;
+ flushConsumeQueueLeastPages = 0;
+ logicsMsgTimestamp = StreamMessageStore.this.getStoreCheckpoint().getLogicsMsgTimestamp();
+ }
+
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
+
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
+ for (ConsumeQueueInterface cq : maps.values()) {
+ boolean result = false;
+ for (int i = 0; i < retryTimes && !result; i++) {
+ result = StreamMessageStore.this.consumeQueueStore.flush(cq, flushConsumeQueueLeastPages);
+ }
+ }
+ }
+
+ if (0 == flushConsumeQueueLeastPages) {
+ if (logicsMsgTimestamp > 0) {
+ StreamMessageStore.this.getStoreCheckpoint().setLogicsMsgTimestamp(logicsMsgTimestamp);
+ }
+ StreamMessageStore.this.getStoreCheckpoint().flush();
+ }
+ }
+
+ @Override
+ public void run() {
+ StreamMessageStore.log.info(this.getServiceName() + " service started");
+
+ while (!this.isStopped()) {
+ try {
+ int interval = StreamMessageStore.this.getMessageStoreConfig().getFlushIntervalConsumeQueue();
+ this.waitForRunning(interval);
+ this.doFlush(1);
+ } catch (Exception e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ this.doFlush(RETRY_TIMES_OVER);
+
+ StreamMessageStore.log.info(this.getServiceName() + " service end");
+ }
+
+ @Override
+ public String getServiceName() {
+ return FlushConsumeQueueService.class.getSimpleName();
+ }
+
+ @Override
+ public long getJointime() {
+ return 1000 * 60;
+ }
+ }
+
+ private class SyncReputMessageServiceFutureItem {
+ private Future future;
+ private long nextReputFromOffset;
+
+ public SyncReputMessageServiceFutureItem(Future future, long nextReputFromOffset) {
+ this.future = future;
+ this.nextReputFromOffset = nextReputFromOffset;
+ }
+
+ public Future getFuture() {
+ return future;
+ }
+
+ public void setFuture(Future future) {
+ this.future = future;
+ }
+
+ public long getNextReputFromOffset() {
+ return nextReputFromOffset;
+ }
+
+ public void setNextReputFromOffset(long nextReputFromOffset) {
+ this.nextReputFromOffset = nextReputFromOffset;
+ }
+ }
+
+ class SyncReputMessageService extends ReputMessageService {
+
+ private BlockingQueue<SyncReputMessageServiceFutureItem> reputResultQueue;
+
+ SyncReputMessageService() {
+ reputResultQueue = new LinkedBlockingDeque<>(messageStoreConfig.getDispatchCqThreads() * messageStoreConfig.getDispatchCqCacheNum());
+ }
+
+ void processDispatchRequestForRecover(DispatchRequest dispatchRequest) {
+ int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
+ if (dispatchRequest.isSuccess() && size > 0) {
+ StreamMessageStore.this.doDispatch(dispatchRequest);
+ this.reputFromOffset = Math.max(dispatchRequest.getNextReputFromOffset(), this.reputFromOffset);
+ }
+ }
+
+ void processDispatchRequest(DispatchRequest dispatchRequest) throws InterruptedException {
+ int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
+ if (dispatchRequest.isSuccess() && size > 0) {
+ Future future = StreamMessageStore.this.doDispatch(dispatchRequest, messageStoreConfig.isEnableAsyncReput());
+ reputResultQueue.put(new SyncReputMessageServiceFutureItem(future, dispatchRequest.getNextReputFromOffset()));
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ for (int i = 0; i < 30 && this.reputResultQueue.size() > 0; i++) {
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ignored) {
+ }
+ }
+
+ if (this.reputResultQueue.size() > 0) {
+ log.warn("shutdown SyncReputMessageService, but reputResultQueue not all processed, CLMaxOffset: {} reputFromOffset: {}",
+ StreamMessageStore.this.commitLog.getMaxOffset(), this.reputFromOffset);
+ }
+
+ super.shutdown();
+ }
+ @Override
+ public void run() {
+ StreamMessageStore.log.info(this.getServiceName() + " service started");
+ while (!this.isStopped()) {
+ try {
+ SyncReputMessageServiceFutureItem item = reputResultQueue.poll(1, TimeUnit.SECONDS);
+ if (null == item) {
+ continue;
+ }
+ item.getFuture().get();
+ this.reputFromOffset = Math.max(item.getNextReputFromOffset(), this.reputFromOffset);
+
+ } catch (Exception e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ waitForRunning(100);
+ }
+ }
+
+ StreamMessageStore.log.info(this.getServiceName() + " service end");
+ }
+
+ @Override
+ public String getServiceName() {
+ return SyncReputMessageService.class.getSimpleName();
+ }
+ }
+
+ class ReputMessageService extends ServiceThread {
+
+ protected volatile long reputFromOffset = 0;
+
+ public long getReputFromOffset() {
+ return reputFromOffset;
+ }
+
+ public void setReputFromOffset(long reputFromOffset) {
+ this.reputFromOffset = reputFromOffset;
+ }
+
+ public boolean dispatched(long physicalOffset) {
+ return this.reputFromOffset > physicalOffset;
+ }
+
+ @Override
+ public void shutdown() {
+ for (int i = 0; i < 30 && this.isCommitLogAvailable(); i++) {
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ignored) {
+ }
+ }
+
+ if (this.isCommitLogAvailable()) {
+ log.warn("shutdown ReputMessageService, but commitlog have not finish to be dispatched, CL: {} reputFromOffset: {}",
+ StreamMessageStore.this.commitLog.getMaxOffset(), this.reputFromOffset);
+ }
+
+ super.shutdown();
+ }
+
+ public long behind() {
+ return StreamMessageStore.this.commitLog.getMaxOffset() - this.reputFromOffset;
+ }
+
+ protected boolean isCommitLogAvailable() {
+ return this.reputFromOffset < StreamMessageStore.this.commitLog.getMaxOffset();
+ }
+
+ private void doReput() throws Exception {
+ if (this.reputFromOffset < StreamMessageStore.this.commitLog.getMinOffset()) {
+ log.warn("The reputFromOffset={} is smaller than minPyOffset={}, this usually indicate that the dispatch behind too much and the commitlog has expired.",
+ this.reputFromOffset, StreamMessageStore.this.commitLog.getMinOffset());
+ this.reputFromOffset = StreamMessageStore.this.commitLog.getMinOffset();
+ }
+ for (boolean doNext = true; this.isCommitLogAvailable() && doNext; ) {
+
+ if (StreamMessageStore.this.getMessageStoreConfig().isDuplicationEnable()
+ && this.reputFromOffset >= StreamMessageStore.this.getConfirmOffset()) {
+ break;
+ }
+
+ SelectMappedBufferResult result = StreamMessageStore.this.commitLog.getData(reputFromOffset);
+ boolean reputAsync = messageStoreConfig.isEnableAsyncReput();
+ if (result != null) {
+ long cacheReputFromOffset = this.reputFromOffset;
+ List<Future> futures = new LinkedList<>();
+ try {
+ this.reputFromOffset = result.getStartOffset();
+ cacheReputFromOffset = this.reputFromOffset;
+
+ for (int readSize = 0; readSize < result.getSize() && doNext; ) {
+ DispatchRequest dispatchRequest =
+ StreamMessageStore.this.commitLog.checkMessageAndReturnSize(result.getByteBuffer(), false, false);
+ int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
+
+ if (dispatchRequest.isSuccess()) {
+ if (size > 0) {
+ futures.add(StreamMessageStore.this.doDispatch(dispatchRequest, reputAsync));
+
+ if (BrokerRole.SLAVE != StreamMessageStore.this.getMessageStoreConfig().getBrokerRole()
+ && StreamMessageStore.this.brokerConfig.isLongPollingEnable()) {
+ StreamMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
+ dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
+ dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
+ dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
+ }
+
+ cacheReputFromOffset += size;
+ readSize += size;
+ if (StreamMessageStore.this.getMessageStoreConfig().getBrokerRole() == BrokerRole.SLAVE) {
+ StreamMessageStore.this.storeStatsService
+ .getSinglePutMessageTopicTimesTotal(dispatchRequest.getTopic()).add(1);
+ StreamMessageStore.this.storeStatsService
+ .getSinglePutMessageTopicSizeTotal(dispatchRequest.getTopic())
+ .add(dispatchRequest.getMsgSize());
+ }
+ } else if (size == 0) {
+ cacheReputFromOffset = StreamMessageStore.this.commitLog.rollNextFile(cacheReputFromOffset);
+ readSize = result.getSize();
+ }
+ } else if (!dispatchRequest.isSuccess()) {
+
+ if (size > 0) {
+ log.error("[BUG]read total count not equals msg total size. reputFromOffset={}", cacheReputFromOffset);
+ cacheReputFromOffset += size;
+ } else {
+ doNext = false;
+ if (StreamMessageStore.this.brokerConfig.getBrokerId() == MixAll.MASTER_ID) {
+ log.error("[BUG]the master dispatch message to consume queue error, COMMITLOG OFFSET: {}",
+ cacheReputFromOffset);
+ cacheReputFromOffset += result.getSize() - readSize;
+ }
+ }
+ }
+ }
+ for (Future future: futures) {
+ future.get();
+ }
+ futures.clear();
+ this.reputFromOffset = cacheReputFromOffset;
+ } finally {
+ result.release();
+ }
+ } else {
+ doNext = false;
+ }
+ }
+ }
+
+ @Override
+ public void run() {
+ StreamMessageStore.log.info(this.getServiceName() + " service started");
+
+ while (!this.isStopped()) {
+ try {
+ Thread.sleep(1);
+ this.doReput();
+ } catch (Exception e) {
+ StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
+ }
+ }
+
+ StreamMessageStore.log.info(this.getServiceName() + " service end");
+ }
+
+ @Override
+ public String getServiceName() {
+ return ReputMessageService.class.getSimpleName();
+ }
+
+ }
+}
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java b/store/src/main/java/org/apache/rocketmq/store/Swappable.java
similarity index 55%
copy from common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
copy to store/src/main/java/org/apache/rocketmq/store/Swappable.java
index a2713cb..cb8dee5 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
+++ b/store/src/main/java/org/apache/rocketmq/store/Swappable.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
@@ -6,7 +6,7 @@
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
@@ -14,27 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.rocketmq.store;
-package org.apache.rocketmq.common.message;
-
-import java.nio.ByteBuffer;
-
-public class MessageExtBatch extends MessageExt {
-
- private static final long serialVersionUID = -2353110995348498537L;
-
- public ByteBuffer wrap() {
- assert getBody() != null;
- return ByteBuffer.wrap(getBody(), 0, getBody().length);
- }
-
- private ByteBuffer encodedBuff;
-
- public ByteBuffer getEncodedBuff() {
- return encodedBuff;
- }
-
- public void setEncodedBuff(ByteBuffer encodedBuff) {
- this.encodedBuff = encodedBuff;
- }
+/**
+ * Clean up page-table on super large disk
+ */
+public interface Swappable {
+ void swapMap(int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs);
+ void cleanSwappedMap(long forceCleanSwapIntervalMs);
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
new file mode 100644
index 0000000..0338cf7
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
@@ -0,0 +1,29 @@
+package org.apache.rocketmq.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class TopicQueueLock {
+ private final int size;
+ private final List<Lock> lockList;
+
+ public TopicQueueLock() {
+ this.size = 32;
+ this.lockList = new ArrayList<>(32);
+ for (int i = 0; i < this.size; i++) {
+ this.lockList.add(new ReentrantLock());
+ }
+ }
+
+ public void lock(String topicQueueKey) {
+ Lock lock = this.lockList.get((topicQueueKey.hashCode() & 0x7fffffff) % this.size);
+ lock.lock();
+ }
+
+ public void unlock(String topicQueueKey) {
+ Lock lock = this.lockList.get((topicQueueKey.hashCode() & 0x7fffffff) % this.size);
+ lock.unlock();
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java b/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
index e1439a0..7a41c27 100644
--- a/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
+++ b/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
@@ -16,10 +16,12 @@
*/
package org.apache.rocketmq.store.config;
-import java.io.File;
-
import org.apache.rocketmq.common.annotation.ImportantField;
import org.apache.rocketmq.store.ConsumeQueue;
+import org.apache.rocketmq.store.queue.BatchConsumeQueue;
+import org.apache.rocketmq.store.queue.CQType;
+
+import java.io.File;
public class MessageStoreConfig {
@@ -44,6 +46,7 @@ public class MessageStoreConfig {
private boolean enableConsumeQueueExt = false;
// ConsumeQueue extend file size, 48M
private int mappedFileSizeConsumeQueueExt = 48 * 1024 * 1024;
+ private int mapperFileSizeBatchConsumeQueue = 300000 * BatchConsumeQueue.CQ_STORE_UNIT_SIZE;
// Bit count of filter bit map.
// this will be set by pipe of calculate filter bit map.
private int bitMapLengthConsumeQueueExt = 64;
@@ -58,6 +61,12 @@ public class MessageStoreConfig {
@ImportantField
private int commitIntervalCommitLog = 200;
+ private int maxRecoveryCommitlogFiles = 30;
+
+ private int diskSpaceWarningLevelRatio = 90;
+
+ private int diskSpaceCleanForciblyRatio = 85;
+
/**
* introduced since 4.0.x. Determine whether to use mutex reentrantLock when putting message.<br/>
*/
@@ -159,6 +168,51 @@ public class MessageStoreConfig {
private boolean enableScheduleMessageStats = true;
+ private int maxBatchDeleteFilesNum = 50;
+ //Polish dispatch
+ private int dispatchCqThreads = 10;
+ private int dispatchCqCacheNum = 1024 * 4;
+ private boolean enableAsyncReput = true;
+ //For recheck the reput
+ private boolean recheckReputOffsetFromCq = false;
+
+ // Maximum length of topic
+ private int maxTopicLength = 1000;
+ private int travelCqFileNumWhenGetMessage = 1;
+ // Sleep interval between to corrections
+ private int correctLogicMinOffsetSleepInterval = 1;
+ // Force correct min offset interval
+ private int correctLogicMinOffsetForceInterval = 5 * 60 * 1000;
+ // swap
+ private boolean mappedFileSwapEnable = true;
+ private long commitLogForceSwapMapInterval = 12L * 60 * 60 * 1000;
+ private long commitLogSwapMapInterval = 1L * 60 * 60 * 1000;
+ private int commitLogSwapMapReserveFileNum = 100;
+ private long logicQueueForceSwapMapInterval = 12L * 60 * 60 * 1000;
+ private long logicQueueSwapMapInterval = 1L * 60 * 60 * 1000;
+ private long cleanSwapedMapInterval = 5L * 60 * 1000;
+ private int logicQueueSwapMapReserveFileNum = 20;
+
+ private boolean searchBcqByCacheEnable = true;
+
+ @ImportantField
+ private boolean dispatchFromSenderThread = false;
+
+ @ImportantField
+ private boolean wakeCommitWhenPutMessage = true;
+ @ImportantField
+ private boolean wakeFlushWhenPutMessage = false;
+
+ @ImportantField
+ private boolean enableCleanExpiredOffset = false;
+
+ @ImportantField
+ private String defaultCQType = CQType.SimpleCQ.toString();
+
+ private int maxAsyncPutMessageRequests = 5000;
+
+ private int pullBatchMaxMessageCount = 160;
+
public boolean isDebugLockEnable() {
return debugLockEnable;
}
@@ -281,6 +335,38 @@ public class MessageStoreConfig {
this.maxMessageSize = maxMessageSize;
}
+ public int getMaxTopicLength() {
+ return maxTopicLength;
+ }
+
+ public void setMaxTopicLength(int maxTopicLength) {
+ this.maxTopicLength = maxTopicLength;
+ }
+
+ public int getTravelCqFileNumWhenGetMessage() {
+ return travelCqFileNumWhenGetMessage;
+ }
+
+ public void setTravelCqFileNumWhenGetMessage(int travelCqFileNumWhenGetMessage) {
+ this.travelCqFileNumWhenGetMessage = travelCqFileNumWhenGetMessage;
+ }
+
+ public int getCorrectLogicMinOffsetSleepInterval() {
+ return correctLogicMinOffsetSleepInterval;
+ }
+
+ public void setCorrectLogicMinOffsetSleepInterval(int correctLogicMinOffsetSleepInterval) {
+ this.correctLogicMinOffsetSleepInterval = correctLogicMinOffsetSleepInterval;
+ }
+
+ public int getCorrectLogicMinOffsetForceInterval() {
+ return correctLogicMinOffsetForceInterval;
+ }
+
+ public void setCorrectLogicMinOffsetForceInterval(int correctLogicMinOffsetForceInterval) {
+ this.correctLogicMinOffsetForceInterval = correctLogicMinOffsetForceInterval;
+ }
+
public boolean isCheckCRCOnRecover() {
return checkCRCOnRecover;
}
@@ -626,8 +712,7 @@ public class MessageStoreConfig {
* @return <tt>true</tt> or <tt>false</tt>
*/
public boolean isTransientStorePoolEnable() {
- return transientStorePoolEnable && FlushDiskType.ASYNC_FLUSH == getFlushDiskType()
- && BrokerRole.SLAVE != getBrokerRole();
+ return transientStorePoolEnable && BrokerRole.SLAVE != getBrokerRole();
}
public void setTransientStorePoolEnable(final boolean transientStorePoolEnable) {
@@ -682,6 +767,45 @@ public class MessageStoreConfig {
this.commitCommitLogThoroughInterval = commitCommitLogThoroughInterval;
}
+ public boolean isWakeCommitWhenPutMessage() {
+ return wakeCommitWhenPutMessage;
+ }
+
+ public void setWakeCommitWhenPutMessage(boolean wakeCommitWhenPutMessage) {
+ this.wakeCommitWhenPutMessage = wakeCommitWhenPutMessage;
+ }
+
+ public boolean isWakeFlushWhenPutMessage() {
+ return wakeFlushWhenPutMessage;
+ }
+
+ public void setWakeFlushWhenPutMessage(boolean wakeFlushWhenPutMessage) {
+ this.wakeFlushWhenPutMessage = wakeFlushWhenPutMessage;
+ }
+
+ public int getMapperFileSizeBatchConsumeQueue() {
+ return mapperFileSizeBatchConsumeQueue;
+ }
+
+ public void setMapperFileSizeBatchConsumeQueue(int mapperFileSizeBatchConsumeQueue) {
+ this.mapperFileSizeBatchConsumeQueue = mapperFileSizeBatchConsumeQueue;
+ }
+
+ public boolean isEnableCleanExpiredOffset() {
+ return enableCleanExpiredOffset;
+ }
+
+ public void setEnableCleanExpiredOffset(boolean enableCleanExpiredOffset) {
+ this.enableCleanExpiredOffset = enableCleanExpiredOffset;
+ }
+
+ public String getDefaultCQType() {
+ return defaultCQType;
+ }
+
+ public void setDefaultCQType(String defaultCQType) {
+ this.defaultCQType = defaultCQType;
+ }
public String getReadOnlyCommitLogStorePaths() {
return readOnlyCommitLogStorePaths;
}
@@ -744,4 +868,164 @@ public class MessageStoreConfig {
public void setEnableScheduleMessageStats(boolean enableScheduleMessageStats) {
this.enableScheduleMessageStats = enableScheduleMessageStats;
}
+
+ public int getMaxAsyncPutMessageRequests() {
+ return maxAsyncPutMessageRequests;
+ }
+
+ public void setMaxAsyncPutMessageRequests(int maxAsyncPutMessageRequests) {
+ this.maxAsyncPutMessageRequests = maxAsyncPutMessageRequests;
+ }
+
+ public int getMaxRecoveryCommitlogFiles() {
+ return maxRecoveryCommitlogFiles;
+ }
+
+ public void setMaxRecoveryCommitlogFiles(final int maxRecoveryCommitlogFiles) {
+ this.maxRecoveryCommitlogFiles = maxRecoveryCommitlogFiles;
+ }
+
+ public boolean isDispatchFromSenderThread() {
+ return dispatchFromSenderThread;
+ }
+
+ public void setDispatchFromSenderThread(boolean dispatchFromSenderThread) {
+ this.dispatchFromSenderThread = dispatchFromSenderThread;
+ }
+
+ public int getDispatchCqThreads() {
+ return dispatchCqThreads;
+ }
+
+ public void setDispatchCqThreads(final int dispatchCqThreads) {
+ this.dispatchCqThreads = dispatchCqThreads;
+ }
+
+ public int getDispatchCqCacheNum() {
+ return dispatchCqCacheNum;
+ }
+
+ public void setDispatchCqCacheNum(final int dispatchCqCacheNum) {
+ this.dispatchCqCacheNum = dispatchCqCacheNum;
+ }
+
+ public boolean isEnableAsyncReput() {
+ return enableAsyncReput;
+ }
+
+ public void setEnableAsyncReput(final boolean enableAsyncReput) {
+ this.enableAsyncReput = enableAsyncReput;
+ }
+
+ public boolean isRecheckReputOffsetFromCq() {
+ return recheckReputOffsetFromCq;
+ }
+
+ public void setRecheckReputOffsetFromCq(final boolean recheckReputOffsetFromCq) {
+ this.recheckReputOffsetFromCq = recheckReputOffsetFromCq;
+ }
+
+ public long getCommitLogForceSwapMapInterval() {
+ return commitLogForceSwapMapInterval;
+ }
+
+ public void setCommitLogForceSwapMapInterval(long commitLogForceSwapMapInterval) {
+ this.commitLogForceSwapMapInterval = commitLogForceSwapMapInterval;
+ }
+
+ public int getCommitLogSwapMapReserveFileNum() {
+ return commitLogSwapMapReserveFileNum;
+ }
+
+ public void setCommitLogSwapMapReserveFileNum(int commitLogSwapMapReserveFileNum) {
+ this.commitLogSwapMapReserveFileNum = commitLogSwapMapReserveFileNum;
+ }
+
+ public long getLogicQueueForceSwapMapInterval() {
+ return logicQueueForceSwapMapInterval;
+ }
+
+ public void setLogicQueueForceSwapMapInterval(long logicQueueForceSwapMapInterval) {
+ this.logicQueueForceSwapMapInterval = logicQueueForceSwapMapInterval;
+ }
+
+ public int getLogicQueueSwapMapReserveFileNum() {
+ return logicQueueSwapMapReserveFileNum;
+ }
+
+ public void setLogicQueueSwapMapReserveFileNum(int logicQueueSwapMapReserveFileNum) {
+ this.logicQueueSwapMapReserveFileNum = logicQueueSwapMapReserveFileNum;
+ }
+
+ public long getCleanSwapedMapInterval() {
+ return cleanSwapedMapInterval;
+ }
+
+ public void setCleanSwapedMapInterval(long cleanSwapedMapInterval) {
+ this.cleanSwapedMapInterval = cleanSwapedMapInterval;
+ }
+
+ public long getCommitLogSwapMapInterval() {
+ return commitLogSwapMapInterval;
+ }
+
+ public void setCommitLogSwapMapInterval(long commitLogSwapMapInterval) {
+ this.commitLogSwapMapInterval = commitLogSwapMapInterval;
+ }
+
+ public long getLogicQueueSwapMapInterval() {
+ return logicQueueSwapMapInterval;
+ }
+
+ public void setLogicQueueSwapMapInterval(long logicQueueSwapMapInterval) {
+ this.logicQueueSwapMapInterval = logicQueueSwapMapInterval;
+ }
+
+ public int getMaxBatchDeleteFilesNum() {
+ return maxBatchDeleteFilesNum;
+ }
+
+ public void setMaxBatchDeleteFilesNum(int maxBatchDeleteFilesNum) {
+ this.maxBatchDeleteFilesNum = maxBatchDeleteFilesNum;
+ }
+
+ public boolean isSearchBcqByCacheEnable() {
+ return searchBcqByCacheEnable;
+ }
+
+ public void setSearchBcqByCacheEnable(boolean searchBcqByCacheEnable) {
+ this.searchBcqByCacheEnable = searchBcqByCacheEnable;
+ }
+
+ public int getDiskSpaceWarningLevelRatio() {
+ return diskSpaceWarningLevelRatio;
+ }
+
+ public void setDiskSpaceWarningLevelRatio(int diskSpaceWarningLevelRatio) {
+ this.diskSpaceWarningLevelRatio = diskSpaceWarningLevelRatio;
+ }
+
+ public int getDiskSpaceCleanForciblyRatio() {
+ return diskSpaceCleanForciblyRatio;
+ }
+
+ public void setDiskSpaceCleanForciblyRatio(int diskSpaceCleanForciblyRatio) {
+ this.diskSpaceCleanForciblyRatio = diskSpaceCleanForciblyRatio;
+ }
+
+ public boolean isMappedFileSwapEnable() {
+ return mappedFileSwapEnable;
+ }
+
+ public void setMappedFileSwapEnable(boolean mappedFileSwapEnable) {
+ this.mappedFileSwapEnable = mappedFileSwapEnable;
+ }
+
+ public int getPullBatchMaxMessageCount() {
+ return pullBatchMaxMessageCount;
+ }
+
+ public void setPullBatchMaxMessageCount(int pullBatchMaxMessageCount) {
+ this.pullBatchMaxMessageCount = pullBatchMaxMessageCount;
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/config/StorePathConfigHelper.java b/store/src/main/java/org/apache/rocketmq/store/config/StorePathConfigHelper.java
index ccd76c4..2f34e7d 100644
--- a/store/src/main/java/org/apache/rocketmq/store/config/StorePathConfigHelper.java
+++ b/store/src/main/java/org/apache/rocketmq/store/config/StorePathConfigHelper.java
@@ -27,6 +27,9 @@ public class StorePathConfigHelper {
public static String getStorePathConsumeQueueExt(final String rootDir) {
return rootDir + File.separator + "consumequeue_ext";
}
+ public static String getStorePathBatchConsumeQueue(final String rootDir) {
+ return rootDir + File.separator + "batchconsumequeue";
+ }
public static String getStorePathIndex(final String rootDir) {
return rootDir + File.separator + "index";
diff --git a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
index 4939175..50078c9 100644
--- a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
+++ b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
@@ -41,7 +41,6 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
-import org.apache.rocketmq.common.message.MessageExtBatch;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.store.AppendMessageResult;
@@ -49,12 +48,13 @@ import org.apache.rocketmq.store.AppendMessageStatus;
import org.apache.rocketmq.store.CommitLog;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.DispatchRequest;
-import org.apache.rocketmq.store.MappedFile;
+import org.apache.rocketmq.store.MessageExtBatch;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.StoreStatsService;
+import org.apache.rocketmq.store.logfile.MappedFile;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
/**
@@ -428,55 +428,51 @@ public class DLedgerCommitLog extends CommitLog {
AppendFuture<AppendEntryResponse> dledgerFuture;
EncodeResult encodeResult;
- encodeResult = this.messageSerializer.serialize(msg);
- if (encodeResult.status != AppendMessageStatus.PUT_OK) {
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, new AppendMessageResult(encodeResult.status)));
- }
- putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
- long elapsedTimeInLock;
- long queueOffset;
+ String topicQueueKey = msg.getTopic() + "-" + msg.getQueueId();
+ topicQueueLock.lock(topicQueueKey);
try {
- beginTimeInDledgerLock = this.defaultMessageStore.getSystemClock().now();
- queueOffset = getQueueOffsetByKey(encodeResult.queueOffsetKey, tranType);
- encodeResult.setQueueOffsetKey(queueOffset, false);
- AppendEntryRequest request = new AppendEntryRequest();
- request.setGroup(dLedgerConfig.getGroup());
- request.setRemoteId(dLedgerServer.getMemberState().getSelfId());
- request.setBody(encodeResult.getData());
- dledgerFuture = (AppendFuture<AppendEntryResponse>) dLedgerServer.handleAppend(request);
- if (dledgerFuture.getPos() == -1) {
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
- }
- long wroteOffset = dledgerFuture.getPos() + DLedgerEntry.BODY_OFFSET;
+ defaultMessageStore.assignOffset(topicQueueKey, msg, getBatchNum(msg));
- int msgIdLength = (msg.getSysFlag() & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 + 8 : 16 + 4 + 8;
- ByteBuffer buffer = ByteBuffer.allocate(msgIdLength);
+ encodeResult = this.messageSerializer.serialize(msg);
+ if (encodeResult.status != AppendMessageStatus.PUT_OK) {
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, new AppendMessageResult(encodeResult.status)));
+ }
+ putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
+ long elapsedTimeInLock;
+ long queueOffset;
+ try {
+ beginTimeInDledgerLock = this.defaultMessageStore.getSystemClock().now();
+ queueOffset = getQueueOffsetByKey(msg, tranType);
+ encodeResult.setQueueOffsetKey(queueOffset, false);
+ AppendEntryRequest request = new AppendEntryRequest();
+ request.setGroup(dLedgerConfig.getGroup());
+ request.setRemoteId(dLedgerServer.getMemberState().getSelfId());
+ request.setBody(encodeResult.getData());
+ dledgerFuture = (AppendFuture<AppendEntryResponse>) dLedgerServer.handleAppend(request);
+ if (dledgerFuture.getPos() == -1) {
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
+ }
+ long wroteOffset = dledgerFuture.getPos() + DLedgerEntry.BODY_OFFSET;
+
+ int msgIdLength = (msg.getSysFlag() & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 + 8 : 16 + 4 + 8;
+ ByteBuffer buffer = ByteBuffer.allocate(msgIdLength);
+
+ String msgId = MessageDecoder.createMessageId(buffer, msg.getStoreHostBytes(), wroteOffset);
+ elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginTimeInDledgerLock;
+ appendResult = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, encodeResult.getData().length, msgId, System.currentTimeMillis(), queueOffset, elapsedTimeInLock);
+ } catch (Exception e) {
+ log.error("Put message error", e);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
+ } finally {
+ beginTimeInDledgerLock = 0;
+ putMessageLock.unlock();
+ }
- String msgId = MessageDecoder.createMessageId(buffer, msg.getStoreHostBytes(), wroteOffset);
- elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginTimeInDledgerLock;
- appendResult = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, encodeResult.getData().length, msgId, System.currentTimeMillis(), queueOffset, elapsedTimeInLock);
- switch (tranType) {
- case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
- case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
- break;
- case MessageSysFlag.TRANSACTION_NOT_TYPE:
- case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
- // The next update ConsumeQueue information
- DLedgerCommitLog.this.topicQueueTable.put(encodeResult.queueOffsetKey, queueOffset + 1);
- break;
- default:
- break;
+ if (elapsedTimeInLock > 500) {
+ log.warn("[NOTIFYME]putMessage in lock cost time(ms)={}, bodyLength={} AppendMessageResult={}", elapsedTimeInLock, msg.getBody().length, appendResult);
}
- } catch (Exception e) {
- log.error("Put message error", e);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
} finally {
- beginTimeInDledgerLock = 0;
- putMessageLock.unlock();
- }
-
- if (elapsedTimeInLock > 500) {
- log.warn("[NOTIFYME]putMessage in lock cost time(ms)={}, bodyLength={} AppendMessageResult={}", elapsedTimeInLock, msg.getBody().length, appendResult);
+ topicQueueLock.unlock(topicQueueKey);
}
return dledgerFuture.thenApply(appendEntryResponse -> {
@@ -546,64 +542,71 @@ public class DLedgerCommitLog extends CommitLog {
.status)));
}
- putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
- msgIdBuilder.setLength(0);
- long elapsedTimeInLock;
- long queueOffset;
- int msgNum = 0;
+ int batchNum = encodeResult.batchData.size();
+ topicQueueLock.lock(encodeResult.queueOffsetKey);
try {
- beginTimeInDledgerLock = this.defaultMessageStore.getSystemClock().now();
- queueOffset = getQueueOffsetByKey(encodeResult.queueOffsetKey, tranType);
- encodeResult.setQueueOffsetKey(queueOffset, true);
- BatchAppendEntryRequest request = new BatchAppendEntryRequest();
- request.setGroup(dLedgerConfig.getGroup());
- request.setRemoteId(dLedgerServer.getMemberState().getSelfId());
- request.setBatchMsgs(encodeResult.batchData);
- AppendFuture<AppendEntryResponse> appendFuture = (AppendFuture<AppendEntryResponse>) dLedgerServer.handleAppend(request);
- if (appendFuture.getPos() == -1) {
- log.warn("HandleAppend return false due to error code {}", appendFuture.get().getCode());
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
- }
- dledgerFuture = (BatchAppendFuture<AppendEntryResponse>) appendFuture;
-
- long wroteOffset = 0;
-
- int msgIdLength = (messageExtBatch.getSysFlag() & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 + 8 : 16 + 4 + 8;
- ByteBuffer buffer = ByteBuffer.allocate(msgIdLength);
-
- boolean isFirstOffset = true;
- long firstWroteOffset = 0;
- for (long pos : dledgerFuture.getPositions()) {
- wroteOffset = pos + DLedgerEntry.BODY_OFFSET;
- if (isFirstOffset) {
- firstWroteOffset = wroteOffset;
- isFirstOffset = false;
+ defaultMessageStore.assignOffset(encodeResult.queueOffsetKey, messageExtBatch, (short) batchNum);
+
+ putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
+ msgIdBuilder.setLength(0);
+ long elapsedTimeInLock;
+ long queueOffset;
+ int msgNum = 0;
+ try {
+ beginTimeInDledgerLock = this.defaultMessageStore.getSystemClock().now();
+ queueOffset = getQueueOffsetByKey(messageExtBatch, tranType);
+ encodeResult.setQueueOffsetKey(queueOffset, true);
+ BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setGroup(dLedgerConfig.getGroup());
+ request.setRemoteId(dLedgerServer.getMemberState().getSelfId());
+ request.setBatchMsgs(encodeResult.batchData);
+ AppendFuture<AppendEntryResponse> appendFuture = (AppendFuture<AppendEntryResponse>) dLedgerServer.handleAppend(request);
+ if (appendFuture.getPos() == -1) {
+ log.warn("HandleAppend return false due to error code {}", appendFuture.get().getCode());
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
}
- String msgId = MessageDecoder.createMessageId(buffer, messageExtBatch.getStoreHostBytes(), wroteOffset);
- if (msgIdBuilder.length() > 0) {
- msgIdBuilder.append(',').append(msgId);
- } else {
- msgIdBuilder.append(msgId);
+ dledgerFuture = (BatchAppendFuture<AppendEntryResponse>) appendFuture;
+
+ long wroteOffset = 0;
+
+ int msgIdLength = (messageExtBatch.getSysFlag() & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 + 8 : 16 + 4 + 8;
+ ByteBuffer buffer = ByteBuffer.allocate(msgIdLength);
+
+ boolean isFirstOffset = true;
+ long firstWroteOffset = 0;
+ for (long pos : dledgerFuture.getPositions()) {
+ wroteOffset = pos + DLedgerEntry.BODY_OFFSET;
+ if (isFirstOffset) {
+ firstWroteOffset = wroteOffset;
+ isFirstOffset = false;
+ }
+ String msgId = MessageDecoder.createMessageId(buffer, messageExtBatch.getStoreHostBytes(), wroteOffset);
+ if (msgIdBuilder.length() > 0) {
+ msgIdBuilder.append(',').append(msgId);
+ } else {
+ msgIdBuilder.append(msgId);
+ }
+ msgNum++;
}
- msgNum++;
+
+ elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginTimeInDledgerLock;
+ appendResult = new AppendMessageResult(AppendMessageStatus.PUT_OK, firstWroteOffset, encodeResult.totalMsgLen,
+ msgIdBuilder.toString(), System.currentTimeMillis(), queueOffset, elapsedTimeInLock);
+ appendResult.setMsgNum(msgNum);
+ } catch (Exception e) {
+ log.error("Put message error", e);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
+ } finally {
+ beginTimeInDledgerLock = 0;
+ putMessageLock.unlock();
}
- elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginTimeInDledgerLock;
- appendResult = new AppendMessageResult(AppendMessageStatus.PUT_OK, firstWroteOffset, encodeResult.totalMsgLen,
- msgIdBuilder.toString(), System.currentTimeMillis(), queueOffset, elapsedTimeInLock);
- appendResult.setMsgNum(msgNum);
- DLedgerCommitLog.this.topicQueueTable.put(encodeResult.queueOffsetKey, queueOffset + msgNum);
- } catch (Exception e) {
- log.error("Put message error", e);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR)));
+ if (elapsedTimeInLock > 500) {
+ log.warn("[NOTIFYME]putMessage in lock cost time(ms)={}, bodyLength={} AppendMessageResult={}",
+ elapsedTimeInLock, messageExtBatch.getBody().length, appendResult);
+ }
} finally {
- beginTimeInDledgerLock = 0;
- putMessageLock.unlock();
- }
-
- if (elapsedTimeInLock > 500) {
- log.warn("[NOTIFYME]putMessage in lock cost time(ms)={}, bodyLength={} AppendMessageResult={}",
- elapsedTimeInLock, messageExtBatch.getBody().length, appendResult);
+ topicQueueLock.unlock(encodeResult.queueOffsetKey);
}
return dledgerFuture.thenApply(appendEntryResponse -> {
@@ -657,16 +660,6 @@ public class DLedgerCommitLog extends CommitLog {
}
@Override
- public HashMap<String, Long> getTopicQueueTable() {
- return topicQueueTable;
- }
-
- @Override
- public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
- this.topicQueueTable = topicQueueTable;
- }
-
- @Override
public void destroy() {
super.destroy();
dLedgerFileList.destroy();
@@ -698,12 +691,8 @@ public class DLedgerCommitLog extends CommitLog {
return diff;
}
- private long getQueueOffsetByKey(String key, int tranType) {
- Long queueOffset = DLedgerCommitLog.this.topicQueueTable.get(key);
- if (null == queueOffset) {
- queueOffset = 0L;
- DLedgerCommitLog.this.topicQueueTable.put(key, queueOffset);
- }
+ private long getQueueOffsetByKey(MessageExtBrokerInner msg, int tranType) {
+ Long queueOffset = msg.getQueueOffset();
// Transaction messages that require special handling
switch (tranType) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/ha/HAService.java b/store/src/main/java/org/apache/rocketmq/store/ha/HAService.java
index 845935b..deeb69f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/ha/HAService.java
+++ b/store/src/main/java/org/apache/rocketmq/store/ha/HAService.java
@@ -37,7 +37,7 @@ import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.common.RemotingUtil;
import org.apache.rocketmq.store.CommitLog;
-import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageSpinLock;
import org.apache.rocketmq.store.PutMessageStatus;
@@ -50,7 +50,7 @@ public class HAService {
private final AcceptSocketService acceptSocketService;
- private final DefaultMessageStore defaultMessageStore;
+ private final MessageStore defaultMessageStore;
private final WaitNotifyObject waitNotifyObject = new WaitNotifyObject();
private final AtomicLong push2SlaveMaxOffset = new AtomicLong(0);
@@ -59,7 +59,7 @@ public class HAService {
private final HAClient haClient;
- public HAService(final DefaultMessageStore defaultMessageStore) throws IOException {
+ public HAService(final MessageStore defaultMessageStore) throws IOException {
this.defaultMessageStore = defaultMessageStore;
this.acceptSocketService =
new AcceptSocketService(defaultMessageStore.getMessageStoreConfig().getHaListenPort());
@@ -142,7 +142,7 @@ public class HAService {
}
}
- public DefaultMessageStore getDefaultMessageStore() {
+ public MessageStore getDefaultMessageStore() {
return defaultMessageStore;
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/index/IndexFile.java b/store/src/main/java/org/apache/rocketmq/store/index/IndexFile.java
index e513edf..e920c84 100644
--- a/store/src/main/java/org/apache/rocketmq/store/index/IndexFile.java
+++ b/store/src/main/java/org/apache/rocketmq/store/index/IndexFile.java
@@ -25,7 +25,8 @@ import java.util.List;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
-import org.apache.rocketmq.store.MappedFile;
+import org.apache.rocketmq.store.logfile.DefaultMappedFile;
+import org.apache.rocketmq.store.logfile.MappedFile;
public class IndexFile {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
@@ -43,7 +44,7 @@ public class IndexFile {
final long endPhyOffset, final long endTimestamp) throws IOException {
int fileTotalSize =
IndexHeader.INDEX_HEADER_SIZE + (hashSlotNum * hashSlotSize) + (indexNum * indexSize);
- this.mappedFile = new MappedFile(fileName, fileTotalSize);
+ this.mappedFile = new DefaultMappedFile(fileName, fileTotalSize);
this.fileChannel = this.mappedFile.getFileChannel();
this.mappedByteBuffer = this.mappedFile.getMappedByteBuffer();
this.hashSlotNum = hashSlotNum;
diff --git a/store/src/main/java/org/apache/rocketmq/store/index/IndexService.java b/store/src/main/java/org/apache/rocketmq/store/index/IndexService.java
index bf17ecf..f2c5616 100644
--- a/store/src/main/java/org/apache/rocketmq/store/index/IndexService.java
+++ b/store/src/main/java/org/apache/rocketmq/store/index/IndexService.java
@@ -29,8 +29,8 @@ import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
-import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
public class IndexService {
@@ -39,14 +39,14 @@ public class IndexService {
* Maximum times to attempt index file creation.
*/
private static final int MAX_TRY_IDX_CREATE = 3;
- private final DefaultMessageStore defaultMessageStore;
+ private final MessageStore defaultMessageStore;
private final int hashSlotNum;
private final int indexNum;
private final String storePath;
private final ArrayList<IndexFile> indexFileList = new ArrayList<IndexFile>();
private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
- public IndexService(final DefaultMessageStore store) {
+ public IndexService(final MessageStore store) {
this.defaultMessageStore = store;
this.hashSlotNum = store.getMessageStoreConfig().getMaxHashSlotNum();
this.indexNum = store.getMessageStoreConfig().getMaxIndexNum();
@@ -282,7 +282,7 @@ public class IndexService {
}
if (null == indexFile) {
- this.defaultMessageStore.getAccessRights().makeIndexFileError();
+ this.defaultMessageStore.getRunningFlags().makeIndexFileError();
log.error("Mark index file cannot build flag");
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java b/store/src/main/java/org/apache/rocketmq/store/logfile/AbstractMappedFile.java
similarity index 55%
copy from common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
copy to store/src/main/java/org/apache/rocketmq/store/logfile/AbstractMappedFile.java
index a2713cb..28d443c 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
+++ b/store/src/main/java/org/apache/rocketmq/store/logfile/AbstractMappedFile.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
@@ -6,7 +6,7 @@
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
@@ -14,27 +14,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.rocketmq.store.logfile;
-package org.apache.rocketmq.common.message;
+import org.apache.rocketmq.store.ReferenceResource;
-import java.nio.ByteBuffer;
-
-public class MessageExtBatch extends MessageExt {
-
- private static final long serialVersionUID = -2353110995348498537L;
-
- public ByteBuffer wrap() {
- assert getBody() != null;
- return ByteBuffer.wrap(getBody(), 0, getBody().length);
- }
-
- private ByteBuffer encodedBuff;
-
- public ByteBuffer getEncodedBuff() {
- return encodedBuff;
- }
-
- public void setEncodedBuff(ByteBuffer encodedBuff) {
- this.encodedBuff = encodedBuff;
- }
+public abstract class AbstractMappedFile extends ReferenceResource implements MappedFile {
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/MappedFile.java b/store/src/main/java/org/apache/rocketmq/store/logfile/DefaultMappedFile.java
similarity index 75%
rename from store/src/main/java/org/apache/rocketmq/store/MappedFile.java
rename to store/src/main/java/org/apache/rocketmq/store/logfile/DefaultMappedFile.java
index 297271d..1ee8a88 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MappedFile.java
+++ b/store/src/main/java/org/apache/rocketmq/store/logfile/DefaultMappedFile.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store;
+package org.apache.rocketmq.store.logfile;
import com.sun.jna.NativeLong;
import com.sun.jna.Pointer;
@@ -36,22 +36,28 @@ import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.message.MessageExtBatch;
-import org.apache.rocketmq.store.CommitLog.PutMessageContext;
+import org.apache.rocketmq.store.AppendMessageCallback;
+import org.apache.rocketmq.store.AppendMessageResult;
+import org.apache.rocketmq.store.AppendMessageStatus;
+import org.apache.rocketmq.store.MessageExtBatch;
+import org.apache.rocketmq.store.MessageExtBrokerInner;
+import org.apache.rocketmq.store.PutMessageContext;
+import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.TransientStorePool;
import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.util.LibC;
import sun.nio.ch.DirectBuffer;
-public class MappedFile extends ReferenceResource {
+public class DefaultMappedFile extends AbstractMappedFile {
public static final int OS_PAGE_SIZE = 1024 * 4;
protected static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
- private static final AtomicLong TOTAL_MAPPED_VIRTUAL_MEMORY = new AtomicLong(0);
+ protected static final AtomicLong TOTAL_MAPPED_VIRTUAL_MEMORY = new AtomicLong(0);
- private static final AtomicInteger TOTAL_MAPPED_FILES = new AtomicInteger(0);
+ protected static final AtomicInteger TOTAL_MAPPED_FILES = new AtomicInteger(0);
protected final AtomicInteger wrotePosition = new AtomicInteger(0);
protected final AtomicInteger committedPosition = new AtomicInteger(0);
- private final AtomicInteger flushedPosition = new AtomicInteger(0);
+ protected final AtomicInteger flushedPosition = new AtomicInteger(0);
protected int fileSize;
protected FileChannel fileChannel;
/**
@@ -59,21 +65,26 @@ public class MappedFile extends ReferenceResource {
*/
protected ByteBuffer writeBuffer = null;
protected TransientStorePool transientStorePool = null;
- private String fileName;
- private long fileFromOffset;
- private File file;
- private MappedByteBuffer mappedByteBuffer;
- private volatile long storeTimestamp = 0;
- private boolean firstCreateInQueue = false;
+ protected String fileName;
+ protected long fileFromOffset;
+ protected File file;
+ protected MappedByteBuffer mappedByteBuffer;
+ protected volatile long storeTimestamp = 0;
+ protected boolean firstCreateInQueue = false;
+ private long lastFlushTime = -1L;
- public MappedFile() {
+ protected MappedByteBuffer mappedByteBufferWaitToClean = null;
+ protected long swapMapTime = 0L;
+ protected long mappedByteBufferAccessCountSinceLastSwap = 0L;
+
+ public DefaultMappedFile() {
}
- public MappedFile(final String fileName, final int fileSize) throws IOException {
+ public DefaultMappedFile(final String fileName, final int fileSize) throws IOException {
init(fileName, fileSize);
}
- public MappedFile(final String fileName, final int fileSize,
+ public DefaultMappedFile(final String fileName, final int fileSize,
final TransientStorePool transientStorePool) throws IOException {
init(fileName, fileSize, transientStorePool);
}
@@ -96,6 +107,7 @@ public class MappedFile extends ReferenceResource {
private static Object invoke(final Object target, final String methodName, final Class<?>... args) {
return AccessController.doPrivileged(new PrivilegedAction<Object>() {
+ @Override
public Object run() {
try {
Method method = method(target, methodName, args);
@@ -142,8 +154,9 @@ public class MappedFile extends ReferenceResource {
return TOTAL_MAPPED_VIRTUAL_MEMORY.get();
}
+ @Override
public void init(final String fileName, final int fileSize,
- final TransientStorePool transientStorePool) throws IOException {
+ final TransientStorePool transientStorePool) throws IOException {
init(fileName, fileSize);
this.writeBuffer = transientStorePool.borrowBuffer();
this.transientStorePool = transientStorePool;
@@ -177,23 +190,28 @@ public class MappedFile extends ReferenceResource {
}
}
+ @Override
public long getLastModifiedTimestamp() {
return this.file.lastModified();
}
+ @Override
public int getFileSize() {
return fileSize;
}
+ @Override
public FileChannel getFileChannel() {
return fileChannel;
}
+ @Override
public AppendMessageResult appendMessage(final MessageExtBrokerInner msg, final AppendMessageCallback cb,
PutMessageContext putMessageContext) {
return appendMessagesInner(msg, cb, putMessageContext);
}
+ @Override
public AppendMessageResult appendMessages(final MessageExtBatch messageExtBatch, final AppendMessageCallback cb,
PutMessageContext putMessageContext) {
return appendMessagesInner(messageExtBatch, cb, putMessageContext);
@@ -207,15 +225,17 @@ public class MappedFile extends ReferenceResource {
int currentPos = this.wrotePosition.get();
if (currentPos < this.fileSize) {
- ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice();
+ ByteBuffer byteBuffer = appendMessageBuffer().slice();
byteBuffer.position(currentPos);
AppendMessageResult result;
- if (messageExt instanceof MessageExtBrokerInner) {
- result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos,
- (MessageExtBrokerInner) messageExt, putMessageContext);
- } else if (messageExt instanceof MessageExtBatch) {
+ if (messageExt instanceof MessageExtBatch && !((MessageExtBatch) messageExt).isInnerBatch()) {
+ // traditional batch message
result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos,
(MessageExtBatch) messageExt, putMessageContext);
+ } else if (messageExt instanceof MessageExtBrokerInner) {
+ // traditional single message or newly introduced inner-batch message
+ result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos,
+ (MessageExtBrokerInner) messageExt, putMessageContext);
} else {
return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
}
@@ -227,24 +247,38 @@ public class MappedFile extends ReferenceResource {
return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
}
+ protected ByteBuffer appendMessageBuffer() {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
+ return writeBuffer != null ? writeBuffer : this.mappedByteBuffer;
+ }
+
+ @Override
public long getFileFromOffset() {
return this.fileFromOffset;
}
+ @Override
public boolean appendMessage(final byte[] data) {
+ return appendMessage(data, 0, data.length);
+ }
+
+ @Override
+ public boolean appendMessage(ByteBuffer data) {
int currentPos = this.wrotePosition.get();
+ int remaining = data.remaining();
- if ((currentPos + data.length) <= this.fileSize) {
+ if ((currentPos + remaining) <= this.fileSize) {
try {
this.fileChannel.position(currentPos);
- this.fileChannel.write(ByteBuffer.wrap(data));
+ while (data.hasRemaining()) {
+ this.fileChannel.write(data);
+ }
} catch (Throwable e) {
log.error("Error occurred when append message to mappedFile.", e);
}
- this.wrotePosition.addAndGet(data.length);
+ this.wrotePosition.addAndGet(remaining);
return true;
}
-
return false;
}
@@ -254,13 +288,17 @@ public class MappedFile extends ReferenceResource {
* @param offset The offset of the subarray to be used.
* @param length The length of the subarray to be used.
*/
+ @Override
public boolean appendMessage(final byte[] data, final int offset, final int length) {
int currentPos = this.wrotePosition.get();
if ((currentPos + length) <= this.fileSize) {
try {
this.fileChannel.position(currentPos);
- this.fileChannel.write(ByteBuffer.wrap(data, offset, length));
+ ByteBuffer buffer = ByteBuffer.wrap(data, offset, length);
+ while (buffer.hasRemaining()) {
+ this.fileChannel.write(buffer);
+ }
} catch (Throwable e) {
log.error("Error occurred when append message to mappedFile.", e);
}
@@ -274,18 +312,22 @@ public class MappedFile extends ReferenceResource {
/**
* @return The current flushed position
*/
+ @Override
public int flush(final int flushLeastPages) {
if (this.isAbleToFlush(flushLeastPages)) {
if (this.hold()) {
int value = getReadPosition();
try {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
+
//We only append data to fileChannel or mappedByteBuffer, never both.
if (writeBuffer != null || this.fileChannel.position() != 0) {
this.fileChannel.force(false);
} else {
this.mappedByteBuffer.force();
}
+ this.lastFlushTime = System.currentTimeMillis();
} catch (Throwable e) {
log.error("Error occurred when force data to disk.", e);
}
@@ -300,6 +342,7 @@ public class MappedFile extends ReferenceResource {
return this.getFlushedPosition();
}
+ @Override
public int commit(final int commitLeastPages) {
if (writeBuffer == null) {
//no need to commit data to file channel, so just regard wrotePosition as committedPosition.
@@ -371,22 +414,28 @@ public class MappedFile extends ReferenceResource {
return write > flush;
}
+ @Override
public int getFlushedPosition() {
return flushedPosition.get();
}
+ @Override
public void setFlushedPosition(int pos) {
this.flushedPosition.set(pos);
}
+ @Override
public boolean isFull() {
return this.fileSize == this.wrotePosition.get();
}
+ @Override
public SelectMappedBufferResult selectMappedBuffer(int pos, int size) {
int readPosition = getReadPosition();
if ((pos + size) <= readPosition) {
if (this.hold()) {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
+
ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
byteBuffer.position(pos);
ByteBuffer byteBufferNew = byteBuffer.slice();
@@ -404,10 +453,12 @@ public class MappedFile extends ReferenceResource {
return null;
}
+ @Override
public SelectMappedBufferResult selectMappedBuffer(int pos) {
int readPosition = getReadPosition();
if (pos < readPosition && pos >= 0) {
if (this.hold()) {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
byteBuffer.position(pos);
int size = readPosition - pos;
@@ -435,26 +486,31 @@ public class MappedFile extends ReferenceResource {
}
clean(this.mappedByteBuffer);
+ clean(this.mappedByteBufferWaitToClean);
+ this.mappedByteBufferWaitToClean = null;
TOTAL_MAPPED_VIRTUAL_MEMORY.addAndGet(this.fileSize * (-1));
TOTAL_MAPPED_FILES.decrementAndGet();
log.info("unmap file[REF:" + currentRef + "] " + this.fileName + " OK");
return true;
}
+ @Override
public boolean destroy(final long intervalForcibly) {
this.shutdown(intervalForcibly);
if (this.isCleanupOver()) {
try {
+ long lastModified = getLastModifiedTimestamp();
this.fileChannel.close();
log.info("close file channel " + this.fileName + " OK");
long beginTime = System.currentTimeMillis();
boolean result = this.file.delete();
log.info("delete file[REF:" + this.getRefCount() + "] " + this.fileName
- + (result ? " OK, " : " Failed, ") + "W:" + this.getWrotePosition() + " M:"
- + this.getFlushedPosition() + ", "
- + UtilAll.computeElapsedTimeMilliseconds(beginTime));
+ + (result ? " OK, " : " Failed, ") + "W:" + this.getWrotePosition() + " M:"
+ + this.getFlushedPosition() + ", "
+ + UtilAll.computeElapsedTimeMilliseconds(beginTime)
+ + "," + (System.currentTimeMillis() - lastModified));
} catch (Exception e) {
log.warn("close file channel " + this.fileName + " Failed. ", e);
}
@@ -462,16 +518,18 @@ public class MappedFile extends ReferenceResource {
return true;
} else {
log.warn("destroy mapped file[REF:" + this.getRefCount() + "] " + this.fileName
- + " Failed. cleanupOver: " + this.cleanupOver);
+ + " Failed. cleanupOver: " + this.cleanupOver);
}
return false;
}
+ @Override
public int getWrotePosition() {
return wrotePosition.get();
}
+ @Override
public void setWrotePosition(int pos) {
this.wrotePosition.set(pos);
}
@@ -479,20 +537,25 @@ public class MappedFile extends ReferenceResource {
/**
* @return The max position which have valid data
*/
+ @Override
public int getReadPosition() {
return this.writeBuffer == null ? this.wrotePosition.get() : this.committedPosition.get();
}
+ @Override
public void setCommittedPosition(int pos) {
this.committedPosition.set(pos);
}
+ @Override
public void warmMappedFile(FlushDiskType type, int pages) {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
+
long beginTime = System.currentTimeMillis();
ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
int flush = 0;
long time = System.currentTimeMillis();
- for (int i = 0, j = 0; i < this.fileSize; i += MappedFile.OS_PAGE_SIZE, j++) {
+ for (int i = 0, j = 0; i < this.fileSize; i += DefaultMappedFile.OS_PAGE_SIZE, j++) {
byteBuffer.put(i, (byte) 0);
// force flush when flush disk type is sync
if (type == FlushDiskType.SYNC_FLUSH) {
@@ -526,30 +589,99 @@ public class MappedFile extends ReferenceResource {
this.mlock();
}
+ @Override
+ public boolean swapMap() {
+ if (getRefCount() == 1 && this.mappedByteBufferWaitToClean == null) {
+
+ if (!hold()) {
+ log.warn("in swapMap, hold failed, fileName: " + this.fileName);
+ return false;
+ }
+ try {
+ this.mappedByteBufferWaitToClean = this.mappedByteBuffer;
+ this.mappedByteBuffer = this.fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, fileSize);
+ this.mappedByteBufferAccessCountSinceLastSwap = 0L;
+ this.swapMapTime = System.currentTimeMillis();
+ log.info("swap file " + this.fileName + " success.");
+ return true;
+ } catch (Exception e) {
+ log.error("swapMap file " + this.fileName + " Failed. ", e);
+ } finally {
+ this.release();
+ }
+ } else {
+ log.info("Will not swap file: " + this.fileName + ", ref=" + getRefCount());
+ }
+ return false;
+ }
+
+ @Override
+ public void cleanSwapedMap(boolean force) {
+ try {
+ if (this.mappedByteBufferWaitToClean == null) {
+ return;
+ }
+ long minGapTime = 120 * 1000L;
+ long gapTime = System.currentTimeMillis() - this.swapMapTime;
+ if (!force && gapTime < minGapTime) {
+ Thread.sleep(minGapTime - gapTime);
+ }
+ clean(this.mappedByteBufferWaitToClean);
+ mappedByteBufferWaitToClean = null;
+ log.info("cleanSwapedMap file " + this.fileName + " success.");
+ } catch (Exception e) {
+ log.error("cleanSwapedMap file " + this.fileName + " Failed. ", e);
+ }
+ }
+
+ @Override
+ public long getRecentSwapMapTime() {
+ return 0;
+ }
+
+ @Override
+ public long getMappedByteBufferAccessCountSinceLastSwap() {
+ return this.mappedByteBufferAccessCountSinceLastSwap;
+ }
+
+ @Override
+ public long getLastFlushTime() {
+ return this.lastFlushTime;
+ }
+
+ @Override
public String getFileName() {
return fileName;
}
+ @Override
public MappedByteBuffer getMappedByteBuffer() {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
return mappedByteBuffer;
}
+ @Override
public ByteBuffer sliceByteBuffer() {
+ this.mappedByteBufferAccessCountSinceLastSwap++;
return this.mappedByteBuffer.slice();
}
+ @Override
public long getStoreTimestamp() {
return storeTimestamp;
}
+ @Override
public boolean isFirstCreateInQueue() {
return firstCreateInQueue;
}
+ @Override
public void setFirstCreateInQueue(boolean firstCreateInQueue) {
this.firstCreateInQueue = firstCreateInQueue;
}
+ @Override
public void mlock() {
final long beginTime = System.currentTimeMillis();
final long address = ((DirectBuffer) (this.mappedByteBuffer)).address();
@@ -565,6 +697,7 @@ public class MappedFile extends ReferenceResource {
}
}
+ @Override
public void munlock() {
final long beginTime = System.currentTimeMillis();
final long address = ((DirectBuffer) (this.mappedByteBuffer)).address();
@@ -573,8 +706,8 @@ public class MappedFile extends ReferenceResource {
log.info("munlock {} {} {} ret = {} time consuming = {}", address, this.fileName, this.fileSize, ret, System.currentTimeMillis() - beginTime);
}
- //testable
- File getFile() {
+ @Override
+ public File getFile() {
return this.file;
}
@@ -582,4 +715,5 @@ public class MappedFile extends ReferenceResource {
public String toString() {
return this.fileName;
}
+
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java b/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java
new file mode 100644
index 0000000..0b6a9dd
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.store.logfile;
+
+import org.apache.rocketmq.store.AppendMessageCallback;
+import org.apache.rocketmq.store.AppendMessageResult;
+import org.apache.rocketmq.store.CommitLog;
+import org.apache.rocketmq.store.MessageExtBatch;
+import org.apache.rocketmq.store.MessageExtBrokerInner;
+import org.apache.rocketmq.store.PutMessageContext;
+import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.TransientStorePool;
+import org.apache.rocketmq.store.config.FlushDiskType;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.MappedByteBuffer;
+import java.nio.channels.FileChannel;
+
+public interface MappedFile {
+ /**
+ * Returns the file name of the {@code MappedFile}.
+ *
+ * @return the file name
+ */
+ String getFileName();
+
+ /**
+ * Returns the file size of the {@code MappedFile}.
+ *
+ * @return the file size
+ */
+ int getFileSize();
+
+ /**
+ * Returns the {@code FileChannel} behind the {@code MappedFile}.
+ *
+ * @return the file channel
+ */
+ FileChannel getFileChannel();
+
+ /**
+ * Returns true if this {@code MappedFile} is full and no new messages can be added.
+ *
+ * @return true if the file is full
+ */
+ boolean isFull();
+
+ /**
+ * Returns true if this {@code MappedFile} is available.
+ * <p>
+ * The mapped file will be not available if it's shutdown or destroyed.
+ *
+ * @return true if the file is available
+ */
+ boolean isAvailable();
+
+ /**
+ * Appends a message object to the current {@code MappedFile} with a specific call back.
+ *
+ * @param message a message to append
+ * @param messageCallback the specific call back to execute the real append action
+ * @param putMessageContext
+ * @return the append result
+ */
+ AppendMessageResult appendMessage(MessageExtBrokerInner message, AppendMessageCallback messageCallback, PutMessageContext putMessageContext);
+
+ /**
+ * Appends a batch message object to the current {@code MappedFile} with a specific call back.
+ *
+ * @param message a message to append
+ * @param messageCallback the specific call back to execute the real append action
+ * @param putMessageContext
+ * @return the append result
+ */
+ AppendMessageResult appendMessages(MessageExtBatch message, AppendMessageCallback messageCallback, PutMessageContext putMessageContext);
+
+ /**
+ * Appends a raw message data represents by a byte array to the current {@code MappedFile}.
+ *
+ * @param data the byte array to append
+ * @return true if success; false otherwise.
+ */
+ boolean appendMessage(byte[] data);
+
+ /**
+ * Appends a raw message data represents by a byte array to the current {@code MappedFile}.
+ *
+ * @param data the byte buffer to append
+ * @return true if success; false otherwise.
+ */
+ boolean appendMessage(ByteBuffer data);
+
+ /**
+ * Appends a raw message data represents by a byte array to the current {@code MappedFile},
+ * starting at the given offset in the array.
+ *
+ * @param data the byte array to append
+ * @param offset the offset within the array of the first byte to be read
+ * @param length the number of bytes to be read from the given array
+ * @return true if success; false otherwise.
+ */
+ boolean appendMessage(byte[] data, int offset, int length);
+
+ /**
+ * Returns the global offset of the current {code MappedFile}, it's a long value of the file name.
+ *
+ * @return the offset of this file
+ */
+ long getFileFromOffset();
+
+ /**
+ * Flushes the data in cache to disk immediately.
+ *
+ * @param flushLeastPages the least pages to flush
+ * @return the flushed position after the method call
+ */
+ int flush(int flushLeastPages);
+
+ /**
+ * Flushes the data in the secondary cache to page cache or disk immediately.
+ *
+ * @param commitLeastPages the least pages to commit
+ * @return the committed position after the method call
+ */
+ int commit(int commitLeastPages);
+
+ /**
+ * Selects a slice of the mapped byte buffer's sub-region behind the mapped file,
+ * starting at the given position.
+ *
+ * @param pos the given position
+ * @param size the size of the returned sub-region
+ * @return a {@code SelectMappedBufferResult} instance contains the selected slice
+ */
+ SelectMappedBufferResult selectMappedBuffer(int pos, int size);
+
+ /**
+ * Selects a slice of the mapped byte buffer's sub-region behind the mapped file,
+ * starting at the given position.
+ *
+ * @param pos the given position
+ * @return a {@code SelectMappedBufferResult} instance contains the selected slice
+ */
+ SelectMappedBufferResult selectMappedBuffer(int pos);
+
+ /**
+ * Returns the mapped byte buffer behind the mapped file.
+ *
+ * @return the mapped byte buffer
+ */
+ MappedByteBuffer getMappedByteBuffer();
+
+ /**
+ * Returns a slice of the mapped byte buffer behind the mapped file.
+ *
+ * @return the slice of the mapped byte buffer
+ */
+ ByteBuffer sliceByteBuffer();
+
+ /**
+ * Returns the store timestamp of the last message.
+ *
+ * @return the store timestamp
+ */
+ long getStoreTimestamp();
+
+ /**
+ * Returns the last modified timestamp of the file.
+ *
+ * @return the last modified timestamp
+ */
+ long getLastModifiedTimestamp();
+
+ /**
+ * Destroys the file and delete it from the file system.
+ *
+ * @param intervalForcibly If {@code true} then this method will destroy the file forcibly and ignore the reference
+ * @return true if success; false otherwise.
+ */
+ boolean destroy(long intervalForcibly);
+
+ /**
+ * Shutdowns the file and mark it unavailable.
+ *
+ * @param intervalForcibly If {@code true} then this method will shutdown the file forcibly and ignore the reference
+ */
+ void shutdown(long intervalForcibly);
+
+ /**
+ * Decreases the reference count by {@code 1} and clean up the mapped file if the reference count reaches at
+ * {@code 0}.
+ */
+ void release();
+
+ /**
+ * Increases the reference count by {@code 1}.
+ *
+ * @return true if success; false otherwise.
+ */
+ boolean hold();
+
+ /**
+ * Returns true if the current file is first mapped file of some consume queue.
+ *
+ * @return true or false
+ */
+ boolean isFirstCreateInQueue();
+
+ /**
+ * Sets the flag whether the current file is first mapped file of some consume queue.
+ *
+ * @param firstCreateInQueue true or false
+ */
+ void setFirstCreateInQueue(boolean firstCreateInQueue);
+
+ /**
+ * Returns the flushed position of this mapped file.
+ *
+ * @return the flushed posotion
+ */
+ int getFlushedPosition();
+
+ /**
+ * Sets the flushed position of this mapped file.
+ *
+ * @param flushedPosition the specific flushed position
+ */
+ void setFlushedPosition(int flushedPosition);
+
+ /**
+ * Returns the wrote position of this mapped file.
+ *
+ * @return the wrote position
+ */
+ int getWrotePosition();
+
+ /**
+ * Sets the wrote position of this mapped file.
+ *
+ * @param wrotePosition the specific wrote position
+ */
+ void setWrotePosition(int wrotePosition);
+
+ /**
+ * Returns the current max readable position of this mapped file.
+ *
+ * @return the max readable position
+ */
+ int getReadPosition();
+
+ /**
+ * Sets the committed position of this mapped file.
+ *
+ * @param committedPosition the specific committed position
+ */
+ void setCommittedPosition(int committedPosition);
+
+ /**
+ * Lock the mapped bytebuffer
+ */
+ void mlock();
+
+ /**
+ * Unlock the mapped bytebuffer
+ */
+ void munlock();
+
+ /**
+ * Warm up the mapped bytebuffer
+ * @param type
+ * @param pages
+ */
+ void warmMappedFile(FlushDiskType type, int pages);
+
+ /**
+ * Swap map
+ */
+ boolean swapMap();
+
+ /**
+ * Clean pageTable
+ */
+ void cleanSwapedMap(boolean force);
+
+ /**
+ * Get recent swap map time
+ */
+ long getRecentSwapMapTime();
+
+ /**
+ * Get recent MappedByteBuffer access count since last swap
+ */
+ long getMappedByteBufferAccessCountSinceLastSwap();
+
+ /**
+ * Get the underlying file
+ * @return
+ */
+ File getFile();
+
+ /**
+ * Get the last flush time
+ * @return
+ */
+ long getLastFlushTime();
+
+ /**
+ * Init mapped file
+ * @param fileName file name
+ * @param fileSize file size
+ * @param transientStorePool transient store pool
+ * @throws IOException
+ */
+ void init(String fileName, int fileSize, TransientStorePool transientStorePool) throws IOException;
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
new file mode 100644
index 0000000..de0e636
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
@@ -0,0 +1,952 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store.queue;
+
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.MappedFileQueue;
+import org.apache.rocketmq.store.MessageStore;
+import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.apache.rocketmq.store.config.BrokerRole;
+import org.apache.rocketmq.store.logfile.MappedFile;
+
+import java.io.File;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+public class BatchConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
+
+ //position 8, size 4, tagscode 8, storetime 8, msgBaseOffset 8, batchSize 2, compactedOffset 4, reserved 4
+ public static final int CQ_STORE_UNIT_SIZE = 46;
+ public static final int MSG_STORE_TIME_OFFSET_INDEX = 20;
+ public static final int MSG_BASE_OFFSET_INDEX = 28;
+ public static final int MSG_BATCH_SIZE_INDEX = 36;
+ public static final int MSG_COMPACT_OFFSET_INDEX = 38;
+ private static final int MSG_COMPACT_OFFSET_LENGTH = 4;
+ public static final int INVALID_POS = -1;
+ final MappedFileQueue mappedFileQueue;
+ private MessageStore defaultMessageStore;
+ private final String topic;
+ private final int queueId;
+ private final ByteBuffer byteBufferItem;
+
+ private final String storePath;
+ private final int mappedFileSize;
+ private volatile long maxMsgPhyOffsetInCommitLog = -1;
+
+ private volatile long minLogicOffset = 0;
+
+ private volatile long maxOffsetInQueue = 0;
+ private volatile long minOffsetInQueue = -1;
+ private final int commitLogSize;
+
+ private ConcurrentSkipListMap<Long, MappedFile> offsetCache = new ConcurrentSkipListMap<>();
+ private ConcurrentSkipListMap<Long, MappedFile> timeCache = new ConcurrentSkipListMap<>();
+
+ public BatchConsumeQueue(
+ final String topic,
+ final int queueId,
+ final String storePath,
+ final int mappedFileSize,
+ final MessageStore defaultMessageStore) {
+ this.storePath = storePath;
+ this.mappedFileSize = mappedFileSize;
+ this.defaultMessageStore = defaultMessageStore;
+ this.commitLogSize = defaultMessageStore.getCommitLog().getCommitLogSize();
+
+ this.topic = topic;
+ this.queueId = queueId;
+
+ String queueDir = this.storePath
+ + File.separator + topic
+ + File.separator + queueId;
+
+ this.mappedFileQueue = new MappedFileQueue(queueDir, mappedFileSize, null);
+
+ this.byteBufferItem = ByteBuffer.allocate(CQ_STORE_UNIT_SIZE);
+ }
+
+ @Override
+ public boolean load() {
+ boolean result = this.mappedFileQueue.load();
+ log.info("Load batch consume queue {}-{} {} {}", topic, queueId, result ? "OK" : "Failed", mappedFileQueue.getMappedFiles().size());
+ return result;
+ }
+
+ private void refreshCache() {
+ if (!this.defaultMessageStore.getMessageStoreConfig().isSearchBcqByCacheEnable()) {
+ return ;
+ }
+ ConcurrentSkipListMap<Long, MappedFile> newOffsetCache = new ConcurrentSkipListMap<>();
+ ConcurrentSkipListMap<Long, MappedFile> newTimeCache = new ConcurrentSkipListMap<>();
+
+ List<MappedFile> mappedFiles = mappedFileQueue.getMappedFiles();
+ // iterate all BCQ files
+ for (int i = 0; i < mappedFiles.size(); i++) {
+ MappedFile bcq = mappedFiles.get(i);
+ if (isNewFile(bcq)) {
+ continue;
+ }
+
+ BatchOffsetIndex min = getMinMsgOffset(bcq, false, true);
+ newOffsetCache.put(min.getMsgOffset(), min.getMappedFile());
+ newTimeCache.put(min.getStoreTimestamp(), min.getMappedFile());
+ }
+
+ this.offsetCache = newOffsetCache;
+ this.timeCache = newTimeCache;
+
+ log.info("refreshCache for BCQ [Topic: {}, QueueId: {}]." +
+ "offsetCacheSize: {}, minCachedMsgOffset: {}, maxCachedMsgOffset: {}, " +
+ "timeCacheSize: {}, minCachedTime: {}, maxCachedTime: {}", this.topic, this.queueId,
+ this.offsetCache.size(), this.offsetCache.firstEntry(), this.offsetCache.lastEntry(),
+ this.timeCache.size(), this.timeCache.firstEntry(), this.timeCache.lastEntry());
+ }
+
+ private void destroyCache() {
+ this.offsetCache.clear();
+ this.timeCache.clear();
+
+ log.info("BCQ [Topic: {}, QueueId: {}]. Cache destroyed", this.topic, this.queueId);
+ }
+
+ private void cacheBcq(MappedFile bcq) {
+ try {
+ BatchOffsetIndex min = getMinMsgOffset(bcq, false, true);
+ this.offsetCache.put(min.getMsgOffset(), min.getMappedFile());
+ this.timeCache.put(min.getStoreTimestamp(), min.getMappedFile());
+ } catch (Exception e) {
+ log.error("Failed caching offset and time on BCQ [Topic: {}, QueueId: {}, File: {}]", this.topic, this.queueId, bcq);
+ }
+ }
+
+ private boolean isNewFile(MappedFile mappedFile) {
+ return mappedFile.getReadPosition() < CQ_STORE_UNIT_SIZE;
+ }
+
+ private MappedFile searchOffsetFromCache(long msgOffset) {
+ Map.Entry<Long, MappedFile> floorEntry = this.offsetCache.floorEntry(msgOffset);
+ if (floorEntry == null) {
+ // the offset is too small.
+ return null;
+ } else {
+ return floorEntry.getValue();
+ }
+ }
+
+ private MappedFile searchTimeFromCache(long time) {
+ Map.Entry<Long, MappedFile> floorEntry = this.timeCache.floorEntry(time);
+ if (floorEntry == null) {
+ // the timestamp is too small. so we decide to result first BCQ file.
+ return this.mappedFileQueue.getFirstMappedFile();
+ } else {
+ return floorEntry.getValue();
+ }
+ }
+
+ @Override
+ public void recover() {
+ final List<MappedFile> mappedFiles = this.mappedFileQueue.getMappedFiles();
+ if (!mappedFiles.isEmpty()) {
+ int index = mappedFiles.size() - 3;
+ if (index < 0)
+ index = 0;
+
+ int mappedFileSizeLogics = this.mappedFileSize;
+ MappedFile mappedFile = mappedFiles.get(index);
+ ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
+ long processOffset = mappedFile.getFileFromOffset();
+ long mappedFileOffset = 0;
+ while (true) {
+ for (int i = 0; i < mappedFileSizeLogics; i += CQ_STORE_UNIT_SIZE) {
+ byteBuffer.position(i);
+ long offset = byteBuffer.getLong();
+ int size = byteBuffer.getInt();
+ byteBuffer.getLong();//tagscode
+ byteBuffer.getLong();//timestamp
+ long msgBaseOffset = byteBuffer.getLong();
+ short batchSize = byteBuffer.getShort();
+ if (offset >= 0 && size > 0 && msgBaseOffset >= 0 && batchSize > 0) {
+ mappedFileOffset = i + CQ_STORE_UNIT_SIZE;
+ this.maxMsgPhyOffsetInCommitLog = offset;
+ } else {
+ log.info("Recover current batch consume queue file over, file:{} offset:{} size:{} msgBaseOffset:{} batchSize:{} mappedFileOffset:{}",
+ mappedFile.getFileName(), offset, size, msgBaseOffset, batchSize, mappedFileOffset);
+ break;
+ }
+ }
+
+ if (mappedFileOffset == mappedFileSizeLogics) {
+ index++;
+ if (index >= mappedFiles.size()) {
+ log.info("Recover last batch consume queue file over, last mapped file:{} ", mappedFile.getFileName());
+ break;
+ } else {
+ mappedFile = mappedFiles.get(index);
+ byteBuffer = mappedFile.sliceByteBuffer();
+ processOffset = mappedFile.getFileFromOffset();
+ mappedFileOffset = 0;
+ log.info("Recover next batch consume queue file: " + mappedFile.getFileName());
+ }
+ } else {
+ log.info("Recover current batch consume queue file over:{} processOffset:{}", mappedFile.getFileName(), processOffset + mappedFileOffset);
+ break;
+ }
+ }
+ processOffset += mappedFileOffset;
+ this.mappedFileQueue.setFlushedWhere(processOffset);
+ this.mappedFileQueue.setCommittedWhere(processOffset);
+ this.mappedFileQueue.truncateDirtyFiles(processOffset);
+ reviseMaxAndMinOffsetInQueue();
+ }
+ }
+
+ void reviseMinOffsetInQueue() {
+ MappedFile firstMappedFile = this.mappedFileQueue.getFirstMappedFile();
+ if (null == firstMappedFile) {
+ maxOffsetInQueue = 0;
+ minOffsetInQueue = -1;
+ minLogicOffset = -1;
+ log.info("reviseMinOffsetInQueue found firstMappedFile null, topic:{} queue:{}", topic, queueId);
+ return;
+ }
+ minLogicOffset = firstMappedFile.getFileFromOffset();
+ BatchOffsetIndex min = getMinMsgOffset(firstMappedFile, false, false);
+ minOffsetInQueue = null == min ? -1 : min.getMsgOffset();
+ }
+
+ void reviseMaxOffsetInQueue() {
+ MappedFile lastMappedFile = this.mappedFileQueue.getLastMappedFile();
+ BatchOffsetIndex max = getMaxMsgOffset(lastMappedFile, true, false);
+ if (null == max && this.mappedFileQueue.getMappedFiles().size() >= 2) {
+ MappedFile lastTwoMappedFile = this.mappedFileQueue.getMappedFiles().get(this.mappedFileQueue.getMappedFiles().size() - 2);
+ max = getMaxMsgOffset(lastTwoMappedFile, true, false);
+ }
+ maxOffsetInQueue = (null == max) ? 0 : max.getMsgOffset() + max.getBatchSize();
+ }
+
+ void reviseMaxAndMinOffsetInQueue() {
+ reviseMinOffsetInQueue();
+ reviseMaxOffsetInQueue();
+ }
+
+ @Override
+ public long getMaxPhysicOffset() {
+ return maxMsgPhyOffsetInCommitLog;
+ }
+
+ @Override
+ public long getMinLogicOffset() {
+ return minLogicOffset;
+ }
+
+ @Override
+ public ReferredIterator<CqUnit> iterateFrom(long startOffset) {
+ SelectMappedBufferResult sbr = getBatchMsgIndexBuffer(startOffset);
+ if (sbr == null) {
+ return null;
+ }
+ return new BatchConsumeQueueIterator(sbr);
+ }
+
+ @Override
+ public CqUnit get(long offset) {
+ ReferredIterator<CqUnit> it = iterateFrom(offset);
+ if (it == null) {
+ return null;
+ }
+ return it.nextAndRelease();
+ }
+
+ @Override
+ public CqUnit getEarliestUnit() {
+ return get(minOffsetInQueue);
+ }
+
+ @Override
+ public CqUnit getLatestUnit() {
+ return get(maxOffsetInQueue - 1);
+ }
+
+ @Override
+ public long getLastOffset() {
+ CqUnit latestUnit = getLatestUnit();
+ return latestUnit.getPos() + latestUnit.getSize();
+ }
+
+ @Override
+ public boolean isFirstFileAvailable() {
+ MappedFile mappedFile = this.mappedFileQueue.getFirstMappedFile();
+ if (mappedFile != null) {
+ return mappedFile.isAvailable();
+ }
+ return false;
+ }
+
+ @Override
+ public boolean isFirstFileExist() {
+ MappedFile mappedFile = this.mappedFileQueue.getFirstMappedFile();
+ return mappedFile != null;
+ }
+
+ @Override
+ public void truncateDirtyLogicFiles(long phyOffset) {
+
+ long oldMinOffset = minOffsetInQueue;
+ long oldMaxOffset = maxOffsetInQueue;
+
+ int logicFileSize = this.mappedFileSize;
+
+ this.maxMsgPhyOffsetInCommitLog = phyOffset - 1;
+ boolean stop = false;
+ while (!stop) {
+ MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
+ if (mappedFile != null) {
+ ByteBuffer byteBuffer = mappedFile.sliceByteBuffer();
+
+ mappedFile.setWrotePosition(0);
+ mappedFile.setCommittedPosition(0);
+ mappedFile.setFlushedPosition(0);
+
+ for (int i = 0; i < logicFileSize; i += CQ_STORE_UNIT_SIZE) {
+ byteBuffer.position(i);
+ long offset = byteBuffer.getLong();
+ int size = byteBuffer.getInt();
+ byteBuffer.getLong();//tagscode
+ byteBuffer.getLong();//timestamp
+ long msgBaseOffset = byteBuffer.getLong();
+ short batchSize = byteBuffer.getShort();
+
+ if (0 == i) {
+ if (offset >= phyOffset) {
+ this.mappedFileQueue.deleteLastMappedFile();
+ break;
+ } else {
+ int pos = i + CQ_STORE_UNIT_SIZE;
+ mappedFile.setWrotePosition(pos);
+ mappedFile.setCommittedPosition(pos);
+ mappedFile.setFlushedPosition(pos);
+ this.maxMsgPhyOffsetInCommitLog = offset;
+ }
+ } else {
+ if (offset >= 0 && size > 0 && msgBaseOffset >= 0 && batchSize > 0) {
+ if (offset >= phyOffset) {
+ stop = true;
+ break;
+ }
+
+ int pos = i + CQ_STORE_UNIT_SIZE;
+ mappedFile.setWrotePosition(pos);
+ mappedFile.setCommittedPosition(pos);
+ mappedFile.setFlushedPosition(pos);
+ this.maxMsgPhyOffsetInCommitLog = offset;
+ if (pos == logicFileSize) {
+ stop = true;
+ break;
+ }
+ } else {
+ stop = true;
+ break;
+ }
+ }
+ }
+ } else {
+ break;
+ }
+ }
+ reviseMaxAndMinOffsetInQueue();
+ log.info("Truncate batch logic file topic={} queue={} oldMinOffset={} oldMaxOffset={} minOffset={} maxOffset={} maxPhyOffsetHere={} maxPhyOffsetThere={}",
+ topic, queueId, oldMinOffset, oldMaxOffset, minOffsetInQueue, maxOffsetInQueue, maxMsgPhyOffsetInCommitLog, phyOffset);
+ }
+
+ @Override
+ public boolean flush(final int flushLeastPages) {
+ boolean result = this.mappedFileQueue.flush(flushLeastPages);
+ return result;
+ }
+
+ @Override
+ public int deleteExpiredFile(long offset) {
+ int cnt = this.mappedFileQueue.deleteExpiredFileByOffset(offset, CQ_STORE_UNIT_SIZE);
+ this.correctMinOffset(offset);
+ return cnt;
+ }
+
+ @Override
+ public void correctMinOffset(long phyMinOffset) {
+ reviseMinOffsetInQueue();
+ refreshCache();
+ long oldMinOffset = minOffsetInQueue;
+ MappedFile mappedFile = this.mappedFileQueue.getFirstMappedFile();
+ if (mappedFile != null) {
+ SelectMappedBufferResult result = mappedFile.selectMappedBuffer(0);
+ if (result != null) {
+ try {
+ int startPos = result.getByteBuffer().position();
+ for (int i = 0; i < result.getSize(); i += BatchConsumeQueue.CQ_STORE_UNIT_SIZE) {
+ result.getByteBuffer().position(startPos + i);
+ long offsetPy = result.getByteBuffer().getLong();
+ result.getByteBuffer().getInt(); //size
+ result.getByteBuffer().getLong();//tagscode
+ result.getByteBuffer().getLong();//timestamp
+ long msgBaseOffset = result.getByteBuffer().getLong();
+ short batchSize = result.getByteBuffer().getShort();
+
+ if (offsetPy < phyMinOffset) {
+ this.minOffsetInQueue = msgBaseOffset + batchSize;
+ } else {
+ break;
+ }
+ }
+ } catch (Exception e) {
+ log.error("Exception thrown when correctMinOffset", e);
+ } finally {
+ result.release();
+ }
+ } else {
+ /**
+ * It will go to here under two conditions:
+ 1. the files number is 1, and it has no data
+ 2. the pull process hold the cq reference, and release it just the moment
+ */
+ log.warn("Correct min offset found null cq file topic:{} queue:{} files:{} minOffset:{} maxOffset:{}",
+ topic, queueId, this.mappedFileQueue.getMappedFiles().size(), minOffsetInQueue, maxOffsetInQueue);
+ }
+ }
+ if (oldMinOffset != this.minOffsetInQueue) {
+ log.info("BatchCQ Compute new minOffset:{} oldMinOffset{} topic:{} queue:{}", minOffsetInQueue, oldMinOffset, topic, queueId);
+ }
+ }
+
+ @Override
+ public void putMessagePositionInfoWrapper(DispatchRequest request) {
+ final int maxRetries = 30;
+ boolean canWrite = this.defaultMessageStore.getRunningFlags().isCQWriteable();
+ if (request.getMsgBaseOffset() < 0 || request.getBatchSize() < 0) {
+ log.warn("[NOTIFYME]unexpected dispacth request in batch consume queue topic:{} queue:{} offset:{}", topic, queueId, request.getCommitLogOffset());
+ return;
+ }
+ for (int i = 0; i < maxRetries && canWrite; i++) {
+ boolean result = this.putBatchMessagePositionInfo(request.getCommitLogOffset(),
+ request.getMsgSize(), request.getTagsCode(),
+ request.getStoreTimestamp(), request.getMsgBaseOffset(), request.getBatchSize());
+ if (result) {
+ if (BrokerRole.SLAVE == this.defaultMessageStore.getMessageStoreConfig().getBrokerRole()) {
+ this.defaultMessageStore.getStoreCheckpoint().setPhysicMsgTimestamp(request.getStoreTimestamp());
+ }
+ this.defaultMessageStore.getStoreCheckpoint().setLogicsMsgTimestamp(request.getStoreTimestamp());
+ return;
+ } else {
+ // XXX: warn and notify me
+ log.warn("[NOTIFYME]put commit log position info to batch consume queue " + topic + ":" + queueId + " " + request.getCommitLogOffset()
+ + " failed, retry " + i + " times");
+
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ log.warn("", e);
+ }
+ }
+ }
+ // XXX: warn and notify me
+ log.error("[NOTIFYME]batch consume queue can not write, {} {}", this.topic, this.queueId);
+ this.defaultMessageStore.getRunningFlags().makeLogicsQueueError();
+ }
+
+ boolean putBatchMessagePositionInfo(final long offset, final int size, final long tagsCode, final long storeTime,
+ final long msgBaseOffset, final short batchSize) {
+
+ if (offset <= this.maxMsgPhyOffsetInCommitLog) {
+ if (System.currentTimeMillis() % 1000 == 0) {
+ log.warn("Build batch consume queue repeatedly, maxMsgPhyOffsetInCommitLog:{} offset:{} Topic: {} QID: {}",
+ maxMsgPhyOffsetInCommitLog, offset, this.topic, this.queueId);
+ }
+ return true;
+ }
+
+ long behind = System.currentTimeMillis() - storeTime;
+ if (behind > 10000 && (System.currentTimeMillis() % 10000 == 0)) {
+ String flag = "LEVEL" + (behind / 10000);
+ log.warn("Reput behind {} topic:{} queue:{} offset:{} behind:{}", flag, topic, queueId, offset, behind);
+ }
+
+ this.byteBufferItem.flip();
+ this.byteBufferItem.limit(CQ_STORE_UNIT_SIZE);
+ this.byteBufferItem.putLong(offset);
+ this.byteBufferItem.putInt(size);
+ this.byteBufferItem.putLong(tagsCode);
+ this.byteBufferItem.putLong(storeTime);
+ this.byteBufferItem.putLong(msgBaseOffset);
+ this.byteBufferItem.putShort(batchSize);
+ this.byteBufferItem.putInt(INVALID_POS);
+ this.byteBufferItem.putInt(0); // 4 bytes reserved
+
+ MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile(this.mappedFileQueue.getMaxOffset());
+ if (mappedFile != null) {
+ boolean isNewFile = isNewFile(mappedFile);
+ boolean appendRes = mappedFile.appendMessage(this.byteBufferItem.array());
+ if (appendRes) {
+ maxMsgPhyOffsetInCommitLog = offset;
+ maxOffsetInQueue = msgBaseOffset + batchSize;
+ //only the first time need to correct the minOffsetInQueue
+ //the other correctness is done in correctLogicMinoffsetService
+ if (mappedFile.isFirstCreateInQueue() && minOffsetInQueue == -1) {
+ reviseMinOffsetInQueue();
+ }
+ if (isNewFile) {
+ // cache new file
+ this.cacheBcq(mappedFile);
+ }
+ }
+ return appendRes;
+ }
+ return false;
+ }
+
+ private BatchOffsetIndex getMinMsgOffset(MappedFile mappedFile, boolean getBatchSize, boolean getStoreTime) {
+ if (mappedFile.getReadPosition() < CQ_STORE_UNIT_SIZE) {
+ return null;
+ }
+ return getBatchOffsetIndexByPos(mappedFile, 0, getBatchSize, getStoreTime);
+ }
+
+ private BatchOffsetIndex getBatchOffsetIndexByPos(MappedFile mappedFile, int pos, boolean getBatchSize,
+ boolean getStoreTime) {
+ SelectMappedBufferResult sbr = mappedFile.selectMappedBuffer(pos);
+ try {
+ return new BatchOffsetIndex(mappedFile, pos, sbr.getByteBuffer().getLong(MSG_BASE_OFFSET_INDEX),
+ getBatchSize ? sbr.getByteBuffer().getShort(MSG_BATCH_SIZE_INDEX) : 0,
+ getStoreTime ? sbr.getByteBuffer().getLong(MSG_STORE_TIME_OFFSET_INDEX) : 0);
+ } finally {
+ sbr.release();
+ }
+ }
+
+ private BatchOffsetIndex getMaxMsgOffset(MappedFile mappedFile, boolean getBatchSize, boolean getStoreTime) {
+ if (mappedFile == null || mappedFile.getReadPosition() < CQ_STORE_UNIT_SIZE) {
+ return null;
+ }
+ int pos = mappedFile.getReadPosition() - CQ_STORE_UNIT_SIZE;
+ return getBatchOffsetIndexByPos(mappedFile, pos, getBatchSize, getStoreTime);
+ }
+
+ private static int ceil(int pos) {
+ return (pos / CQ_STORE_UNIT_SIZE) * CQ_STORE_UNIT_SIZE;
+ }
+
+ /**
+ * Gets SelectMappedBufferResult by batch-message offset
+ * Node: the caller is responsible for the release of SelectMappedBufferResult
+ * @param msgOffset
+ * @return SelectMappedBufferResult
+ */
+ public SelectMappedBufferResult getBatchMsgIndexBuffer(final long msgOffset) {
+ if (msgOffset >= maxOffsetInQueue) {
+ return null;
+ }
+ MappedFile targetBcq;
+ BatchOffsetIndex targetMinOffset;
+
+ // first check the last bcq file
+ MappedFile lastBcq = mappedFileQueue.getLastMappedFile();
+ BatchOffsetIndex minForLastBcq = getMinMsgOffset(lastBcq, false, false);
+ if (null != minForLastBcq && minForLastBcq.getMsgOffset() <= msgOffset) {
+ // found, it's the last bcq.
+ targetBcq = lastBcq;
+ targetMinOffset = minForLastBcq;
+ } else {
+ boolean searchBcqByCacheEnable = this.defaultMessageStore.getMessageStoreConfig().isSearchBcqByCacheEnable();
+ if (searchBcqByCacheEnable) {
+ // it's not the last BCQ file, so search it through cache.
+ targetBcq = this.searchOffsetFromCache(msgOffset);
+ // not found in cache
+ if (targetBcq == null) {
+ MappedFile firstBcq = mappedFileQueue.getFirstMappedFile();
+ BatchOffsetIndex minForFirstBcq = getMinMsgOffset(firstBcq, false, false);
+ if (minForFirstBcq != null && minForFirstBcq.getMsgOffset() <= msgOffset && msgOffset < minForLastBcq.getMsgOffset()) {
+ // old search logic
+ targetBcq = this.searchOffsetFromFiles(msgOffset);
+ }
+ log.warn("cache is not working on BCQ [Topic: {}, QueueId: {}] for msgOffset: {}, targetBcq: {}", this.topic, this.queueId, msgOffset, targetBcq);
+ }
+ } else {
+ // old search logic
+ targetBcq = this.searchOffsetFromFiles(msgOffset);
+ }
+
+ if (targetBcq == null) {
+ return null;
+ }
+
+ targetMinOffset = getMinMsgOffset(targetBcq, false, false);
+ }
+
+ BatchOffsetIndex targetMaxOffset = getMaxMsgOffset(targetBcq, false, false);
+ if (null == targetMinOffset || null == targetMaxOffset) {
+ return null;
+ }
+
+ // then use binary search to find the indexed position
+ SelectMappedBufferResult sbr = targetMinOffset.getMappedFile().selectMappedBuffer(0);
+ try {
+ ByteBuffer byteBuffer = sbr.getByteBuffer();
+ int left = targetMinOffset.getIndexPos(), right = targetMaxOffset.getIndexPos();
+ int mid = binarySearch(byteBuffer, left, right, CQ_STORE_UNIT_SIZE, MSG_BASE_OFFSET_INDEX, msgOffset);
+ if (mid != -1) {
+ // return a buffer that needs to be released manually.
+ return targetMinOffset.getMappedFile().selectMappedBuffer(mid);
+ }
+ } finally {
+ sbr.release();
+ }
+ return null;
+ }
+
+ private MappedFile searchOffsetFromFiles(long msgOffset) {
+ MappedFile targetBcq = null;
+ // find the mapped file one by one reversely
+ int mappedFileNum = this.mappedFileQueue.getMappedFiles().size();
+ for (int i = mappedFileNum - 1; i >= 0; i--) {
+ MappedFile mappedFile = mappedFileQueue.getMappedFiles().get(i);
+ BatchOffsetIndex tmpMinMsgOffset = getMinMsgOffset(mappedFile, false, false);
+ if (null != tmpMinMsgOffset && tmpMinMsgOffset.getMsgOffset() <= msgOffset) {
+ targetBcq = mappedFile;
+ break;
+ }
+ }
+
+ return targetBcq;
+ }
+
+ /**
+ * Find the message whose timestamp is the smallest, greater than or equal to the given time.
+ * @param timestamp
+ * @return
+ */
+ @Override
+ public long getOffsetInQueueByTime(final long timestamp) {
+ MappedFile targetBcq;
+ BatchOffsetIndex targetMinOffset;
+
+ // first check the last bcq
+ MappedFile lastBcq = mappedFileQueue.getLastMappedFile();
+ BatchOffsetIndex minForLastBcq = getMinMsgOffset(lastBcq, false, true);
+ if (null != minForLastBcq && minForLastBcq.getStoreTimestamp() <= timestamp) {
+ // found, it's the last bcq.
+ targetBcq = lastBcq;
+ targetMinOffset = minForLastBcq;
+ } else {
+ boolean searchBcqByCacheEnable = this.defaultMessageStore.getMessageStoreConfig().isSearchBcqByCacheEnable();
+ if (searchBcqByCacheEnable) {
+ // it's not the last BCQ file, so search it through cache.
+ targetBcq = this.searchTimeFromCache(timestamp);
+ if (targetBcq == null) {
+ // not found in cache
+ MappedFile firstBcq = mappedFileQueue.getFirstMappedFile();
+ BatchOffsetIndex minForFirstBcq = getMinMsgOffset(firstBcq, false, true);
+ if (minForFirstBcq != null && minForFirstBcq.getStoreTimestamp() <= timestamp && timestamp < minForLastBcq.getStoreTimestamp()) {
+ // old search logic
+ targetBcq = this.searchTimeFromFiles(timestamp);
+ }
+ log.warn("cache is not working on BCQ [Topic: {}, QueueId: {}] for timestamp: {}, targetBcq: {}", this.topic, this.queueId, timestamp, targetBcq);
+ }
+ } else {
+ // old search logic
+ targetBcq = this.searchTimeFromFiles(timestamp);
+ }
+
+ if (targetBcq == null) {
+ return -1;
+ }
+ targetMinOffset = getMinMsgOffset(targetBcq, false, true);
+ }
+
+ BatchOffsetIndex targetMaxOffset = getMaxMsgOffset(targetBcq, false, true);
+ if (null == targetMinOffset || null == targetMaxOffset) {
+ return -1;
+ }
+
+ //then use binary search to find the indexed position
+ SelectMappedBufferResult sbr = targetMinOffset.getMappedFile().selectMappedBuffer(0);
+ try {
+ ByteBuffer byteBuffer = sbr.getByteBuffer();
+ int left = targetMinOffset.getIndexPos(), right = targetMaxOffset.getIndexPos();
+ int mid = binarySearchRight(byteBuffer, left, right, CQ_STORE_UNIT_SIZE, MSG_STORE_TIME_OFFSET_INDEX, timestamp);
+ if (mid != -1) {
+ return byteBuffer.getLong(mid + MSG_BASE_OFFSET_INDEX);
+ }
+ } finally {
+ sbr.release();
+ }
+
+ return -1;
+ }
+
+ private MappedFile searchTimeFromFiles(long timestamp) {
+ MappedFile targetBcq = null;
+
+ int mappedFileNum = this.mappedFileQueue.getMappedFiles().size();
+ for (int i = mappedFileNum - 1; i >= 0; i--) {
+ MappedFile mappedFile = mappedFileQueue.getMappedFiles().get(i);
+ BatchOffsetIndex tmpMinMsgOffset = getMinMsgOffset(mappedFile, false, true);
+ if (tmpMinMsgOffset == null) {
+ //Maybe the new file
+ continue;
+ }
+ BatchOffsetIndex tmpMaxMsgOffset = getMaxMsgOffset(mappedFile, false, true);
+ //Here should not be null
+ if (tmpMaxMsgOffset == null) {
+ break;
+ }
+ if (tmpMaxMsgOffset.getStoreTimestamp() >= timestamp) {
+ if (tmpMinMsgOffset.getStoreTimestamp() <= timestamp) {
+ targetBcq = mappedFile;
+ break;
+ } else {
+ if (i - 1 < 0) {
+ //This is the first file
+ targetBcq = mappedFile;
+ break;
+ } else {
+ //The min timestamp of this file is larger than the given timestamp, so check the next file
+ continue;
+ }
+ }
+ } else {
+ //The max timestamp of this file is smaller than the given timestamp, so double check the previous file
+ if (i + 1 <= mappedFileNum - 1) {
+ mappedFile = mappedFileQueue.getMappedFiles().get(i + 1);
+ targetBcq = mappedFile;
+ break;
+ } else {
+ //There is no timestamp larger than the given timestamp
+ break;
+ }
+ }
+ }
+
+ return targetBcq;
+ }
+
+ /**
+ * Find the offset of which the value is equal or larger than the given targetValue.
+ * If there are many values equal to the target, then find the earliest one.
+ */
+ public static int binarySearchRight(ByteBuffer byteBuffer, int left, int right, final int unitSize, final int unitShift,
+ long targetValue) {
+ int mid = -1;
+ while (left <= right) {
+ mid = ceil((left + right) / 2);
+ long tmpValue = byteBuffer.getLong(mid + unitShift);
+ if (mid == right) {
+ //Means left and the right are the same
+ if (tmpValue >= targetValue) {
+ return mid;
+ } else {
+ return -1;
+ }
+ } else if (mid == left) {
+ //Means the left + unitSize = right
+ if (tmpValue >= targetValue) {
+ return mid;
+ } else {
+ left = mid + unitSize;
+ }
+ } else {
+ //mid is actully in the mid
+ if (tmpValue < targetValue) {
+ left = mid + unitSize;
+ } else {
+ right = mid;
+ }
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Here is vulnerable, the min value of the bytebuffer must be smaller or equal then the given value.
+ * Otherwise it may get -1
+ */
+ private int binarySearch(ByteBuffer byteBuffer, int left, int right, final int unitSize, final int unitShift,
+ long targetValue) {
+ int maxRight = right;
+ int mid = -1;
+ while (left <= right) {
+ mid = ceil((left + right) / 2);
+ long tmpValue = byteBuffer.getLong(mid + unitShift);
+ if (tmpValue == targetValue) {
+ return mid;
+ }
+ if (tmpValue > targetValue) {
+ right = mid - unitSize;
+ } else {
+ if (mid == left) {
+ //the binary search is converging to the left, so maybe the one on the right of mid is the exactly correct one
+ if (mid + unitSize <= maxRight
+ && byteBuffer.getLong(mid + unitSize + unitShift) <= targetValue) {
+ return mid + unitSize;
+ } else {
+ return mid;
+ }
+ } else {
+ left = mid;
+ }
+ }
+ }
+ return -1;
+ }
+
+ private class BatchConsumeQueueIterator implements ReferredIterator<CqUnit> {
+ private SelectMappedBufferResult sbr;
+ private int relativePos = 0;
+
+ public BatchConsumeQueueIterator(SelectMappedBufferResult sbr) {
+ this.sbr = sbr;
+ if (sbr != null && sbr.getByteBuffer() != null) {
+ relativePos = sbr.getByteBuffer().position();
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (sbr == null || sbr.getByteBuffer() == null) {
+ return false;
+ }
+
+ return sbr.getByteBuffer().hasRemaining();
+ }
+
+ @Override
+ public CqUnit next() {
+ if (!hasNext()) {
+ return null;
+ }
+ ByteBuffer tmpBuffer = sbr.getByteBuffer().slice();
+ tmpBuffer.position(MSG_COMPACT_OFFSET_INDEX);
+ ByteBuffer compactOffsetStoreBuffer = tmpBuffer.slice();
+ compactOffsetStoreBuffer.limit(MSG_COMPACT_OFFSET_LENGTH);
+
+ int relativePos = sbr.getByteBuffer().position();
+ long offsetPy = sbr.getByteBuffer().getLong();
+ int sizePy = sbr.getByteBuffer().getInt();
+ long tagsCode = sbr.getByteBuffer().getLong(); //tagscode
+ sbr.getByteBuffer().getLong();//timestamp
+ long msgBaseOffset = sbr.getByteBuffer().getLong();
+ short batchSize = sbr.getByteBuffer().getShort();
+ int compactedOffset = sbr.getByteBuffer().getInt();
+ sbr.getByteBuffer().position(relativePos + CQ_STORE_UNIT_SIZE);
+
+ return new CqUnit(msgBaseOffset, offsetPy, sizePy, tagsCode, batchSize, compactedOffset, compactOffsetStoreBuffer);
+ }
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException("remove");
+ }
+
+ @Override
+ public void release() {
+ if (sbr != null) {
+ sbr.release();
+ sbr = null;
+ }
+ }
+
+ @Override
+ public CqUnit nextAndRelease() {
+ try {
+ return next();
+ } finally {
+ release();
+ }
+ }
+ }
+
+ @Override
+ public String getTopic() {
+ return topic;
+ }
+
+ @Override
+ public int getQueueId() {
+ return queueId;
+ }
+
+ @Override
+ public CQType getCQType() {
+ return CQType.BatchCQ;
+ }
+
+ @Override
+ public void destroy() {
+ this.maxMsgPhyOffsetInCommitLog = -1;
+ this.minOffsetInQueue = -1;
+ this.maxOffsetInQueue = 0;
+ this.mappedFileQueue.destroy();
+ this.destroyCache();
+ }
+
+ @Override
+ public long getMessageTotalInQueue() {
+ return this.getMaxOffsetInQueue() - this.getMinOffsetInQueue();
+ }
+
+ @Override
+ public long rollNextFile(long offset) {
+ return 0;
+ }
+
+ /**
+ * Batch msg offset (deep logic offset)
+ * @return max deep offset
+ */
+ @Override
+ public long getMaxOffsetInQueue() {
+ return maxOffsetInQueue;
+ }
+
+ @Override
+ public long getMinOffsetInQueue() {
+ return minOffsetInQueue;
+ }
+
+ @Override
+ public void checkSelf() {
+ mappedFileQueue.checkSelf();
+ }
+
+ @Override
+ public void swapMap(int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs) {
+ mappedFileQueue.swapMap(reserveNum, forceSwapIntervalMs, normalSwapIntervalMs);
+ }
+
+ @Override
+ public void cleanSwappedMap(long forceCleanSwapIntervalMs) {
+ mappedFileQueue.cleanSwappedMap(forceCleanSwapIntervalMs);
+ }
+
+ public MappedFileQueue getMappedFileQueue() {
+ return mappedFileQueue;
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/BatchOffsetIndex.java b/store/src/main/java/org/apache/rocketmq/store/queue/BatchOffsetIndex.java
new file mode 100644
index 0000000..8ca85c6
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/BatchOffsetIndex.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store.queue;
+
+import org.apache.rocketmq.store.logfile.MappedFile;
+
+public class BatchOffsetIndex {
+
+ private final MappedFile mappedFile;
+ private final int indexPos;
+ private final long msgOffset;
+ private final short batchSize;
+ private final long storeTimestamp;
+
+ public BatchOffsetIndex(MappedFile file, int pos, long msgOffset, short size, long storeTimestamp) {
+ mappedFile = file;
+ indexPos = pos;
+ this.msgOffset = msgOffset;
+ batchSize = size;
+ this.storeTimestamp = storeTimestamp;
+ }
+
+ public MappedFile getMappedFile() {
+ return mappedFile;
+ }
+
+ public int getIndexPos() {
+ return indexPos;
+ }
+
+ public long getMsgOffset() {
+ return msgOffset;
+ }
+
+ public short getBatchSize() {
+ return batchSize;
+ }
+
+ public long getStoreTimestamp() {
+ return storeTimestamp;
+ }
+}
diff --git a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java b/store/src/main/java/org/apache/rocketmq/store/queue/CQType.java
similarity index 55%
copy from common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
copy to store/src/main/java/org/apache/rocketmq/store/queue/CQType.java
index a2713cb..efd65af 100644
--- a/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/CQType.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
@@ -6,7 +6,7 @@
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
@@ -15,26 +15,10 @@
* limitations under the License.
*/
-package org.apache.rocketmq.common.message;
+package org.apache.rocketmq.store.queue;
-import java.nio.ByteBuffer;
-
-public class MessageExtBatch extends MessageExt {
-
- private static final long serialVersionUID = -2353110995348498537L;
-
- public ByteBuffer wrap() {
- assert getBody() != null;
- return ByteBuffer.wrap(getBody(), 0, getBody().length);
- }
-
- private ByteBuffer encodedBuff;
-
- public ByteBuffer getEncodedBuff() {
- return encodedBuff;
- }
-
- public void setEncodedBuff(ByteBuffer encodedBuff) {
- this.encodedBuff = encodedBuff;
- }
+public enum CQType {
+ SimpleCQ,
+ BatchCQ,
+ MillionCQ;
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java
new file mode 100644
index 0000000..48f717d
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store.queue;
+
+public interface ConsumeQueueInterface {
+ /**
+ * Get the topic name
+ * @return the topic this cq belongs to.
+ */
+ String getTopic();
+
+ /**
+ * Get queue id
+ * @return the queue id this cq belongs to.
+ */
+ int getQueueId();
+
+ /**
+ * Get the units from the start offset.
+ *
+ * @param startIndex start index
+ * @return the unit iterateFrom
+ */
+ ReferredIterator<CqUnit> iterateFrom(long startIndex);
+
+ /**
+ * Get cq unit at specified index
+ * @param index index
+ * @return the cq unit at index
+ */
+ CqUnit get(long index);
+
+ /**
+ * Get earliest cq unit
+ * @return earliest cq unit
+ */
+ CqUnit getEarliestUnit();
+
+ /**
+ * Get last cq unit
+ * @return last cq unit
+ */
+ CqUnit getLatestUnit();
+
+ /**
+ * Get last commit log offset
+ * @return last commit log offset
+ */
+ long getLastOffset();
+
+ /**
+ * Get min offset(index) in queue
+ * @return the min offset(index) in queue
+ */
+ long getMinOffsetInQueue();
+
+ /**
+ * Get max offset(index) in queue
+ * @return the max offset(index) in queue
+ */
+ long getMaxOffsetInQueue();
+
+ /**
+ * Get total message count
+ * @return total message count
+ */
+ long getMessageTotalInQueue();
+
+ /**
+ * Get the message whose timestamp is the smallest, greater than or equal to the given time.
+ * @param timestamp timestamp
+ * @return the offset(index)
+ */
+ long getOffsetInQueueByTime(final long timestamp);
+
+ /**
+ * The max physical offset of commitlog has been dispatched to this queue.
+ * It should be exclusive.
+ *
+ * @return the max physical offset point to commitlog
+ */
+ long getMaxPhysicOffset();
+
+ /**
+ * Usually, the cq files are not exactly consistent with the commitlog, there maybe some redundant data in the first
+ * cq file.
+ *
+ * @return the minimal effective pos of the cq file.
+ */
+ long getMinLogicOffset();
+
+ /**
+ * Get cq type
+ * @return cq type
+ */
+ CQType getCQType();
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
new file mode 100644
index 0000000..bf42e74
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.store.queue;
+
+import org.apache.rocketmq.store.ConsumeQueue;
+import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.MessageStore;
+import org.apache.rocketmq.store.StoreUtil;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.config.StorePathConfigHelper;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+public class ConsumeQueueStore {
+
+ protected final MessageStore messageStore;
+ protected final MessageStoreConfig messageStoreConfig;
+ protected final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
+
+ public ConsumeQueueStore(MessageStore messageStore, MessageStoreConfig messageStoreConfig, ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> consumeQueueTable) {
+ this.messageStore = messageStore;
+ this.messageStoreConfig = messageStoreConfig;
+ this.consumeQueueTable = consumeQueueTable;
+ }
+
+ private FileQueueLifeCycle getLifeCycle(String topic, int queueId) {
+ return (FileQueueLifeCycle) findOrCreateConsumeQueue(topic, queueId);
+ }
+
+ public long rollNextFile(ConsumeQueueInterface consumeQueue, final long offset) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.rollNextFile(offset);
+ }
+
+ public void correctMinOffset(ConsumeQueueInterface consumeQueue, long minCommitLogOffset) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.correctMinOffset(minCommitLogOffset);
+ }
+
+ /**
+ * Apply the dispatched request and build the consume queue.
+ * This function should be idempotent.
+ *
+ * @param request
+ */
+ public void putMessagePositionInfoWrapper(ConsumeQueueInterface consumeQueue, DispatchRequest request) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.putMessagePositionInfoWrapper(request);
+ }
+
+ public boolean load(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.load();
+ }
+
+ public void recover(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.recover();
+ }
+
+ public void checkSelf(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.checkSelf();
+ }
+
+ public boolean flush(ConsumeQueueInterface consumeQueue, int flushLeastPages) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.flush(flushLeastPages);
+ }
+
+ public void destroy(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.destroy();
+ }
+
+ public int deleteExpiredFile(ConsumeQueueInterface consumeQueue, long minOffset) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.deleteExpiredFile(minOffset);
+ }
+
+ public void truncateDirtyLogicFiles(ConsumeQueueInterface consumeQueue, long phyOffset) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.truncateDirtyLogicFiles(phyOffset);
+ }
+
+ public void swapMap(ConsumeQueueInterface consumeQueue, int reserveNum, long forceSwapIntervalMs, long normalSwapIntervalMs) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.swapMap(reserveNum, forceSwapIntervalMs, normalSwapIntervalMs);
+ }
+
+ public void cleanSwappedMap(ConsumeQueueInterface consumeQueue, long forceCleanSwapIntervalMs) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ fileQueueLifeCycle.cleanSwappedMap(forceCleanSwapIntervalMs);
+ }
+
+ public boolean isFirstFileAvailable(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.isFirstFileAvailable();
+ }
+
+ public boolean isFirstFileExist(ConsumeQueueInterface consumeQueue) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
+ return fileQueueLifeCycle.isFirstFileExist();
+ }
+
+ public ConsumeQueueInterface findOrCreateConsumeQueue(String topic, int queueId) {
+ return doFindOrCreateConsumeQueue(topic, queueId);
+ }
+
+ private ConsumeQueueInterface doFindOrCreateConsumeQueue(String topic, int queueId) {
+
+ ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
+ if (null == map) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> newMap = new ConcurrentHashMap<Integer, ConsumeQueueInterface>(128);
+ ConcurrentMap<Integer, ConsumeQueueInterface> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
+ if (oldMap != null) {
+ map = oldMap;
+ } else {
+ map = newMap;
+ }
+ }
+
+ ConsumeQueueInterface logic = map.get(queueId);
+ if (logic != null) {
+ return logic;
+ }
+
+ ConsumeQueueInterface newLogic;
+
+ if (StoreUtil.isStreamMode(this.messageStore)) {
+ newLogic = new BatchConsumeQueue(
+ topic,
+ queueId,
+ StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ this.messageStoreConfig.getMapperFileSizeBatchConsumeQueue(),
+ this.messageStore);
+ ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
+ if (oldLogic != null) {
+ logic = oldLogic;
+ } else {
+ logic = newLogic;
+ }
+ } else {
+ newLogic = new ConsumeQueue(
+ topic,
+ queueId,
+ StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
+ (DefaultMessageStore) this.messageStore);
+ ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
+ if (oldLogic != null) {
+ logic = oldLogic;
+ } else {
+ logic = newLogic;
+ }
+ }
+
+ return logic;
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/CqUnit.java b/store/src/main/java/org/apache/rocketmq/store/queue/CqUnit.java
new file mode 100644
index 0000000..b8865fd
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/CqUnit.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
... 2680 lines suppressed ...
[rocketmq] 17/17: [Assignment] Fix the risk of memory overflow caused by excessive popShareQueueNum.
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 0b4adb49b48b5addc0eafc4ef7412f9f72a9fbfa
Author: zhangyang21 <zh...@xiaomi.com>
AuthorDate: Mon Nov 22 18:31:13 2021 +0800
[Assignment] Fix the risk of memory overflow caused by excessive popShareQueueNum.
Signed-off-by: zhangyang21 <zh...@xiaomi.com>
---
.../broker/processor/QueryAssignmentProcessor.java | 65 +++++++++++--------
.../processor/QueryAssignmentProcessorTest.java | 75 ++++++++++++++++++++++
2 files changed, 113 insertions(+), 27 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessor.java
index fdc320d..6fe9210 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessor.java
@@ -208,33 +208,8 @@ public class QueryAssignmentProcessor implements NettyRequestProcessor {
}
if (setMessageRequestModeRequestBody != null && setMessageRequestModeRequestBody.getMode() == MessageRequestMode.POP) {
- if (setMessageRequestModeRequestBody.getPopShareQueueNum() <= 0) {
- //each client pop all messagequeue
- allocateResult = new ArrayList<>(mqAll.size());
- for (MessageQueue mq : mqAll) {
- //must create new MessageQueue in case of change cache in AssignmentManager
- MessageQueue newMq = new MessageQueue(mq.getTopic(), mq.getBrokerName(), -1);
- allocateResult.add(newMq);
- }
-
- } else {
- if (cidAll.size() <= mqAll.size()) {
- //consumer working in pop mode could share the MessageQueues assigned to the N (N = popWorkGroupSize) consumer following it in the cid list
- allocateResult = allocateMessageQueueStrategy.allocate(consumerGroup, clientId, mqAll, cidAll);
- int index = cidAll.indexOf(clientId);
- if (index >= 0) {
- for (int i = 1; i <= setMessageRequestModeRequestBody.getPopShareQueueNum(); i++) {
- index++;
- index = index % cidAll.size();
- List<MessageQueue> tmp = allocateMessageQueueStrategy.allocate(consumerGroup, cidAll.get(index), mqAll, cidAll);
- allocateResult.addAll(tmp);
- }
- }
- } else {
- //make sure each cid is assigned
- allocateResult = allocate(consumerGroup, clientId, mqAll, cidAll);
- }
- }
+ allocateResult = allocate4Pop(allocateMessageQueueStrategy, consumerGroup, clientId, mqAll,
+ cidAll, setMessageRequestModeRequestBody.getPopShareQueueNum());
} else {
allocateResult = allocateMessageQueueStrategy.allocate(consumerGroup, clientId, mqAll, cidAll);
@@ -256,6 +231,42 @@ public class QueryAssignmentProcessor implements NettyRequestProcessor {
return assignedQueueSet;
}
+ public List<MessageQueue> allocate4Pop(AllocateMessageQueueStrategy allocateMessageQueueStrategy,
+ final String consumerGroup, final String clientId, List<MessageQueue> mqAll, List<String> cidAll,
+ int popShareQueueNum) {
+
+ List<MessageQueue> allocateResult;
+ if (popShareQueueNum <= 0 || popShareQueueNum >= cidAll.size() - 1) {
+ //each client pop all messagequeue
+ allocateResult = new ArrayList<>(mqAll.size());
+ for (MessageQueue mq : mqAll) {
+ //must create new MessageQueue in case of change cache in AssignmentManager
+ MessageQueue newMq = new MessageQueue(mq.getTopic(), mq.getBrokerName(), -1);
+ allocateResult.add(newMq);
+ }
+
+ } else {
+ if (cidAll.size() <= mqAll.size()) {
+ //consumer working in pop mode could share the MessageQueues assigned to the N (N = popWorkGroupSize) consumer following it in the cid list
+ allocateResult = allocateMessageQueueStrategy.allocate(consumerGroup, clientId, mqAll, cidAll);
+ int index = cidAll.indexOf(clientId);
+ if (index >= 0) {
+ for (int i = 1; i <= popShareQueueNum; i++) {
+ index++;
+ index = index % cidAll.size();
+ List<MessageQueue> tmp = allocateMessageQueueStrategy.allocate(consumerGroup, cidAll.get(index), mqAll, cidAll);
+ allocateResult.addAll(tmp);
+ }
+ }
+ } else {
+ //make sure each cid is assigned
+ allocateResult = allocate(consumerGroup, clientId, mqAll, cidAll);
+ }
+ }
+
+ return allocateResult;
+ }
+
private List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
if (currentCID == null || currentCID.length() < 1) {
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessorTest.java b/broker/src/test/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessorTest.java
index 681fcc3..b16533b 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessorTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/processor/QueryAssignmentProcessorTest.java
@@ -19,9 +19,15 @@ package org.apache.rocketmq.broker.processor;
import com.google.common.collect.ImmutableSet;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
+import java.util.ArrayList;
+import java.util.List;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ClientChannelInfo;
import org.apache.rocketmq.broker.loadbalance.AssignmentManager;
+import org.apache.rocketmq.client.consumer.AllocateMessageQueueStrategy;
+import org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueAveragely;
+import org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueAveragelyByCircle;
+import org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueConsistentHash;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
@@ -40,6 +46,7 @@ import org.apache.rocketmq.remoting.protocol.LanguageCode;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -67,6 +74,7 @@ public class QueryAssignmentProcessorTest {
@Mock
private Channel channel;
+ private String broker = "defaultBroker";
private String topic = "FooBar";
private String group = "FooBarGroup";
private String clientId = "127.0.0.1";
@@ -118,6 +126,73 @@ public class QueryAssignmentProcessorTest {
assertThat(responseToReturn.getCode()).isEqualTo(ResponseCode.NO_PERMISSION);
}
+
+ @Test
+ public void testAllocate4Pop() {
+ testAllocate4Pop(new AllocateMessageQueueAveragely());
+ testAllocate4Pop(new AllocateMessageQueueAveragelyByCircle());
+ testAllocate4Pop(new AllocateMessageQueueConsistentHash());
+ }
+
+ private void testAllocate4Pop(AllocateMessageQueueStrategy strategy) {
+ int testNum = 16;
+ List<MessageQueue> mqAll = new ArrayList<>();
+ for (int mqSize = 0; mqSize < testNum; mqSize++) {
+ mqAll.add(new MessageQueue(topic, broker, mqSize));
+
+ List<String> cidAll = new ArrayList<>();
+ for (int cidSize = 0; cidSize < testNum; cidSize++) {
+ String clientId = String.valueOf(cidSize);
+ cidAll.add(clientId);
+
+ for (int popShareQueueNum = 0; popShareQueueNum < testNum; popShareQueueNum++) {
+ List<MessageQueue> allocateResult =
+ queryAssignmentProcessor.allocate4Pop(strategy, group, clientId, mqAll, cidAll, popShareQueueNum);
+ Assert.assertTrue(checkAllocateResult(popShareQueueNum, mqAll.size(), cidAll.size(), allocateResult.size(), strategy));
+ }
+ }
+ }
+ }
+
+ private boolean checkAllocateResult(int popShareQueueNum, int mqSize, int cidSize, int allocateSize,
+ AllocateMessageQueueStrategy strategy) {
+
+ //The maximum size of allocations will not exceed mqSize.
+ if (allocateSize > mqSize) {
+ return false;
+ }
+
+ //It is not allowed that the client is not assigned to the consumeQueue.
+ if (allocateSize <= 0) {
+ return false;
+ }
+
+ if (popShareQueueNum <= 0 || popShareQueueNum >= cidSize - 1) {
+ return allocateSize == mqSize;
+ } else if (mqSize < cidSize) {
+ return allocateSize == 1;
+ }
+
+ if (strategy instanceof AllocateMessageQueueAveragely
+ || strategy instanceof AllocateMessageQueueAveragelyByCircle) {
+
+ if (mqSize % cidSize == 0) {
+ return allocateSize == (mqSize / cidSize) * (popShareQueueNum + 1);
+ } else {
+ int avgSize = mqSize / cidSize;
+ return allocateSize >= avgSize * (popShareQueueNum + 1)
+ && allocateSize <= (avgSize + 1) * (popShareQueueNum + 1);
+ }
+ }
+
+ if (strategy instanceof AllocateMessageQueueConsistentHash) {
+ //Just skip
+ return true;
+ }
+
+ return false;
+ }
+
private RemotingCommand createQueryAssignmentRequest() {
QueryAssignmentRequestBody requestBody = new QueryAssignmentRequestBody();
requestBody.setTopic(topic);
[rocketmq] 15/17: [ISSUE #3498] Make messages in reviveTopic more evenly written to different queues #3499
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit fe88fb36a3e1790cdb33ca670306c2adea280ba3
Author: cserwen <cs...@163.com>
AuthorDate: Wed Jan 12 21:00:56 2022 +0800
[ISSUE #3498] Make messages in reviveTopic more evenly written to different queues #3499
---
.../org/apache/rocketmq/broker/processor/PopMessageProcessor.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java
index aa97fc8..fcc972d 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java
@@ -92,6 +92,7 @@ public class PopMessageProcessor implements NettyRequestProcessor {
private PopLongPollingService popLongPollingService;
private PopBufferMergeService popBufferMergeService;
private QueueLockManager queueLockManager;
+ private AtomicLong ckMessageNumber;
public PopMessageProcessor(final BrokerController brokerController) {
this.brokerController = brokerController;
@@ -104,6 +105,7 @@ public class PopMessageProcessor implements NettyRequestProcessor {
this.popLongPollingService = new PopLongPollingService();
this.queueLockManager = new QueueLockManager();
this.popBufferMergeService = new PopBufferMergeService(this.brokerController, this);
+ this.ckMessageNumber = new AtomicLong();
}
public PopLongPollingService getPopLongPollingService() {
@@ -350,7 +352,7 @@ public class PopMessageProcessor implements NettyRequestProcessor {
if (requestHeader.isOrder()) {
reviveQid = KeyBuilder.POP_ORDER_REVIVE_QUEUE;
} else {
- reviveQid = randomQ % this.brokerController.getBrokerConfig().getReviveQueueNum();
+ reviveQid = (int) Math.abs(ckMessageNumber.getAndIncrement() % this.brokerController.getBrokerConfig().getReviveQueueNum());
}
GetMessageResult getMessageResult = new GetMessageResult();
[rocketmq] 04/17: [tools] Fix parameter conflicts
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit a01433094b46fb89dc64b18fbd224897cbe11c86
Author: zhangyang21 <zh...@xiaomi.com>
AuthorDate: Thu Nov 18 15:44:07 2021 +0800
[tools] Fix parameter conflicts
Signed-off-by: zhangyang21 <zh...@xiaomi.com>
---
.../rocketmq/tools/command/consumer/SetConsumeModeSubCommand.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/command/consumer/SetConsumeModeSubCommand.java b/tools/src/main/java/org/apache/rocketmq/tools/command/consumer/SetConsumeModeSubCommand.java
index 9bf5551..9565ace 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/command/consumer/SetConsumeModeSubCommand.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/command/consumer/SetConsumeModeSubCommand.java
@@ -64,7 +64,7 @@ public class SetConsumeModeSubCommand implements SubCommand {
opt.setRequired(true);
options.addOption(opt);
- opt = new Option("n", "popShareQueueNum", true, "num fo queue which share in pop mode");
+ opt = new Option("q", "popShareQueueNum", true, "num of queue which share in pop mode");
opt.setRequired(false);
options.addOption(opt);
@@ -89,8 +89,8 @@ public class SetConsumeModeSubCommand implements SubCommand {
int popShareQueueNum = 0;
- if (commandLine.hasOption('n')) {
- popShareQueueNum = Integer.parseInt(commandLine.getOptionValue('n')
+ if (commandLine.hasOption('q')) {
+ popShareQueueNum = Integer.parseInt(commandLine.getOptionValue('q')
.trim());
}
[rocketmq] 07/17: [ISSUE #3679] Support topic attributes (#3698)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 18010a4d30e43a76e633a01fe0476f44de763c83
Author: Hongjian Fei <er...@163.com>
AuthorDate: Wed Jan 5 16:02:53 2022 +0800
[ISSUE #3679] Support topic attributes (#3698)
* [Feature] Support topic attributes.
* [Feature] Topic attributes unit-test.
---
.../broker/processor/AdminBrokerProcessor.java | 18 +-
.../rocketmq/broker/topic/TopicConfigManager.java | 143 +++++++++
.../broker/topic/TopicConfigManagerTest.java | 318 +++++++++++++++++++++
.../rocketmq/client/impl/MQClientAPIImpl.java | 3 +
common/pom.xml | 13 -
.../apache/rocketmq/common/TopicAttributes.java | 33 ++-
.../org/apache/rocketmq/common/TopicConfig.java | 50 ++--
.../rocketmq/common/attribute/Attribute.java | 38 ++-
.../rocketmq/common/attribute/AttributeParser.java | 79 +++++
.../common/attribute/BooleanAttribute.java | 32 ++-
.../rocketmq/common/attribute/EnumAttribute.java | 32 ++-
.../common/attribute/LongRangeAttribute.java | 35 ++-
.../protocol/header/CreateTopicRequestHeader.java | 11 +
.../common/attribute/AttributeParserTest.java | 70 +++++
.../rocketmq/common/attribute/AttributeTest.java | 70 +++++
.../apache/rocketmq/store/util/QueueTypeUtils.java | 20 ++
.../tools/command/topic/UpdateTopicSubCommand.java | 15 +
17 files changed, 873 insertions(+), 107 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index f30953d..6505263 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -33,12 +33,14 @@ import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
+import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.admin.ConsumeStats;
import org.apache.rocketmq.common.admin.OffsetWrapper;
import org.apache.rocketmq.common.admin.TopicOffset;
import org.apache.rocketmq.common.admin.TopicStatsTable;
+import org.apache.rocketmq.common.attribute.AttributeParser;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
@@ -294,12 +296,18 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
topicConfig.setTopicFilterType(requestHeader.getTopicFilterTypeEnum());
topicConfig.setPerm(requestHeader.getPerm());
topicConfig.setTopicSysFlag(requestHeader.getTopicSysFlag() == null ? 0 : requestHeader.getTopicSysFlag());
+ String attributesModification = requestHeader.getAttributes();
+ topicConfig.setAttributes(AttributeParser.parseToMap(attributesModification));
- this.brokerController.getTopicConfigManager().updateTopicConfig(topicConfig);
-
- this.brokerController.registerIncrementBrokerData(topicConfig, this.brokerController.getTopicConfigManager().getDataVersion());
-
- response.setCode(ResponseCode.SUCCESS);
+ try {
+ this.brokerController.getTopicConfigManager().updateTopicConfig(topicConfig);
+ this.brokerController.registerIncrementBrokerData(topicConfig, this.brokerController.getTopicConfigManager().getDataVersion());
+ response.setCode(ResponseCode.SUCCESS);
+ } catch (Exception e) {
+ log.error("Update / create topic failed for [{}]", request, e);
+ response.setCode(ResponseCode.SYSTEM_ERROR);
+ response.setRemark(e.getMessage());
+ }
return response;
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
index 4af230d..ba40538 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
@@ -16,6 +16,8 @@
*/
package org.apache.rocketmq.broker.topic;
+import java.util.HashMap;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;
@@ -25,11 +27,16 @@ import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.BrokerPathConfigHelper;
+import org.apache.rocketmq.common.attribute.Attribute;
import org.apache.rocketmq.common.ConfigManager;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MixAll;
+import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.constant.PermName;
@@ -40,6 +47,8 @@ import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
+import static com.google.common.base.Preconditions.checkNotNull;
+
public class TopicConfigManager extends ConfigManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private static final long LOCK_TIMEOUT_MILLIS = 3000;
@@ -353,6 +362,18 @@ public class TopicConfigManager extends ConfigManager {
}
public void updateTopicConfig(final TopicConfig topicConfig) {
+ checkNotNull(topicConfig, "topicConfig shouldn't be null");
+
+ Map<String, String> newAttributes = request(topicConfig);
+ Map<String, String> currentAttributes = current(topicConfig.getTopicName());
+
+ Map<String, String> finalAttributes = alterCurrentAttributes(
+ this.topicConfigTable.get(topicConfig.getTopicName()) == null,
+ ImmutableMap.copyOf(currentAttributes),
+ ImmutableMap.copyOf(newAttributes));
+
+ topicConfig.setAttributes(finalAttributes);
+
TopicConfig old = this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
if (old != null) {
log.info("update topic config, old:[{}] new:[{}]", old, topicConfig);
@@ -398,6 +419,11 @@ public class TopicConfigManager extends ConfigManager {
}
}
+ // make it testable
+ public Map<String, Attribute> allAttributes() {
+ return TopicAttributes.ALL;
+ }
+
public boolean isOrderTopic(final String topic) {
TopicConfig topicConfig = this.topicConfigTable.get(topic);
if (topicConfig == null) {
@@ -471,4 +497,121 @@ public class TopicConfigManager extends ConfigManager {
public ConcurrentMap<String, TopicConfig> getTopicConfigTable() {
return topicConfigTable;
}
+
+ private Map<String, String> request(TopicConfig topicConfig) {
+ return topicConfig.getAttributes() == null ? new HashMap<>() : topicConfig.getAttributes();
+ }
+
+ private Map<String, String> current(String topic) {
+ TopicConfig topicConfig = this.topicConfigTable.get(topic);
+ if (topicConfig == null) {
+ return new HashMap<>();
+ } else {
+ Map<String, String> attributes = topicConfig.getAttributes();
+ if (attributes == null) {
+ return new HashMap<>();
+ } else {
+ return attributes;
+ }
+ }
+ }
+
+ private Map<String, String> alterCurrentAttributes(boolean create, ImmutableMap<String, String> currentAttributes, ImmutableMap<String, String> newAttributes) {
+ Map<String, String> init = new HashMap<>();
+ Map<String, String> add = new HashMap<>();
+ Map<String, String> update = new HashMap<>();
+ Map<String, String> delete = new HashMap<>();
+ Set<String> keys = new HashSet<>();
+
+ for (Entry<String, String> attribute : newAttributes.entrySet()) {
+ String key = attribute.getKey();
+ String realKey = realKey(key);
+ String value = attribute.getValue();
+
+ validate(realKey);
+ duplicationCheck(keys, realKey);
+
+ if (create) {
+ if (key.startsWith("+")) {
+ init.put(realKey, value);
+ } else {
+ throw new RuntimeException("only add attribute is supported while creating topic. key: " + realKey);
+ }
+ } else {
+ if (key.startsWith("+")) {
+ if (!currentAttributes.containsKey(realKey)) {
+ add.put(realKey, value);
+ } else {
+ update.put(realKey, value);
+ }
+ } else if (key.startsWith("-")) {
+ if (!currentAttributes.containsKey(realKey)) {
+ throw new RuntimeException("attempt to delete a nonexistent key: " + realKey);
+ }
+ delete.put(realKey, value);
+ } else {
+ throw new RuntimeException("wrong format key: " + realKey);
+ }
+ }
+ }
+
+ validateAlter(init, true, false);
+ validateAlter(add, false, false);
+ validateAlter(update, false, false);
+ validateAlter(delete, false, true);
+
+ log.info("add: {}, update: {}, delete: {}", add, update, delete);
+ HashMap<String, String> finalAttributes = new HashMap<>(currentAttributes);
+ finalAttributes.putAll(init);
+ finalAttributes.putAll(add);
+ finalAttributes.putAll(update);
+ for (String s : delete.keySet()) {
+ finalAttributes.remove(s);
+ }
+ return finalAttributes;
+ }
+
+ private void duplicationCheck(Set<String> keys, String key) {
+ boolean notExist = keys.add(key);
+ if (!notExist) {
+ throw new RuntimeException("alter duplication key. key: " + key);
+ }
+ }
+
+ private void validate(String kvAttribute) {
+ if (Strings.isNullOrEmpty(kvAttribute)) {
+ throw new RuntimeException("kv string format wrong.");
+ }
+
+ if (kvAttribute.contains("+")) {
+ throw new RuntimeException("kv string format wrong.");
+ }
+
+ if (kvAttribute.contains("-")) {
+ throw new RuntimeException("kv string format wrong.");
+ }
+ }
+
+ private void validateAlter(Map<String, String> alter, boolean init, boolean delete) {
+ for (Entry<String, String> entry : alter.entrySet()) {
+ String key = entry.getKey();
+ String value = entry.getValue();
+
+ Attribute attribute = allAttributes().get(key);
+ if (attribute == null) {
+ throw new RuntimeException("unsupported key: " + key);
+ }
+ if (!init && !attribute.isChangeable()) {
+ throw new RuntimeException("attempt to update an unchangeable attribute. key: " + key);
+ }
+
+ if (!delete) {
+ attribute.verify(value);
+ }
+ }
+ }
+
+ private String realKey(String key) {
+ return key.substring(1);
+ }
}
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
new file mode 100644
index 0000000..9853c36
--- /dev/null
+++ b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.broker.topic;
+
+import org.apache.rocketmq.broker.BrokerController;
+import org.apache.rocketmq.common.attribute.Attribute;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.attribute.BooleanAttribute;
+import org.apache.rocketmq.common.attribute.EnumAttribute;
+import org.apache.rocketmq.common.attribute.LongRangeAttribute;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.store.util.QueueTypeUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnitRunner;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.google.common.collect.Sets.newHashSet;
+import static java.util.Arrays.asList;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+@RunWith(MockitoJUnitRunner.class)
+public class TopicConfigManagerTest {
+ private TopicConfigManager topicConfigManager;
+ @Mock
+ private BrokerController brokerController;
+
+ @Before
+ public void init() {
+ BrokerConfig brokerConfig = new BrokerConfig();
+ when(brokerController.getBrokerConfig()).thenReturn(brokerConfig);
+ MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+ when(brokerController.getMessageStoreConfig()).thenReturn(messageStoreConfig);
+
+ topicConfigManager = new TopicConfigManager(brokerController);
+ }
+
+ @Test
+ public void testAddUnsupportedKeyOnCreating() {
+ String unsupportedKey = "key4";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+enum.key", "enum-2");
+ attributes.put("+" + unsupportedKey, "value1");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("unsupported key: " + unsupportedKey, runtimeException.getMessage());
+ }
+
+ @Test
+ public void testAddWrongFormatKeyOnCreating() {
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("++enum.key", "value1");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("kv string format wrong.", runtimeException.getMessage());
+ }
+
+ @Test
+ public void testDeleteKeyOnCreating() {
+ String key = "enum.key";
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("-" + key, "");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("only add attribute is supported while creating topic. key: " + key, runtimeException.getMessage());
+ }
+
+ @Test
+ public void testAddWrongValueOnCreating() {
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+" + TopicAttributes.queueType.getName(), "wrong-value");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("value is not in set: [SimpleCQ, BatchCQ]", runtimeException.getMessage());
+ }
+
+ @Test
+ public void testNormalAddKeyOnCreating() {
+ String topic = "new-topic";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+enum.key", "enum-2");
+ attributes.put("+long.range.key", "16");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName(topic);
+ topicConfig.setAttributes(attributes);
+ topicConfigManager.updateTopicConfig(topicConfig);
+
+ TopicConfig existingTopicConfig = topicConfigManager.getTopicConfigTable().get(topic);
+ Assert.assertEquals("enum-2", existingTopicConfig.getAttributes().get("enum.key"));
+ Assert.assertEquals("16", existingTopicConfig.getAttributes().get("long.range.key"));
+// assert file
+ }
+
+ @Test
+ public void testAddDuplicatedKeyOnUpdating() {
+ String duplicatedKey = "long.range.key";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ createTopic();
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+" + duplicatedKey, "11");
+ attributes.put("-" + duplicatedKey, "");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("alter duplication key. key: " + duplicatedKey, runtimeException.getMessage());
+ }
+
+ private void createTopic() {
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+enum.key", "enum-3");
+ attributes.put("+bool.key", "true");
+ attributes.put("+long.range.key", "12");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ topicConfigManager.updateTopicConfig(topicConfig);
+ }
+
+ @Test
+ public void testDeleteNonexistentKeyOnUpdating() {
+ String key = "nonexisting.key";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+enum.key", "enum-2");
+ attributes.put("+bool.key", "true");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName("new-topic");
+ topicConfig.setAttributes(attributes);
+
+ topicConfigManager.updateTopicConfig(topicConfig);
+
+ attributes = new HashMap<>();
+ attributes.clear();
+ attributes.put("-" + key, "");
+ topicConfig.setAttributes(attributes);
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("attempt to delete a nonexistent key: " + key, runtimeException.getMessage());
+ }
+
+ @Test
+ public void testAlterTopicWithoutChangingAttributes() {
+ String topic = "new-topic";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+enum.key", "enum-2");
+ attributes.put("+bool.key", "true");
+
+ TopicConfig topicConfigInit = new TopicConfig();
+ topicConfigInit.setTopicName(topic);
+ topicConfigInit.setAttributes(attributes);
+
+ topicConfigManager.updateTopicConfig(topicConfigInit);
+ Assert.assertEquals("enum-2", topicConfigManager.getTopicConfigTable().get(topic).getAttributes().get("enum.key"));
+ Assert.assertEquals("true", topicConfigManager.getTopicConfigTable().get(topic).getAttributes().get("bool.key"));
+
+ TopicConfig topicConfigAlter = new TopicConfig();
+ topicConfigAlter.setTopicName(topic);
+ topicConfigAlter.setReadQueueNums(10);
+ topicConfigAlter.setWriteQueueNums(10);
+ topicConfigManager.updateTopicConfig(topicConfigAlter);
+ Assert.assertEquals("enum-2", topicConfigManager.getTopicConfigTable().get(topic).getAttributes().get("enum.key"));
+ Assert.assertEquals("true", topicConfigManager.getTopicConfigTable().get(topic).getAttributes().get("bool.key"));
+ }
+
+ @Test
+ public void testNormalUpdateUnchangeableKeyOnUpdating() {
+ String topic = "exist-topic";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", true, false),
+ new LongRangeAttribute("long.range.key", false, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+long.range.key", "14");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName(topic);
+ topicConfig.setAttributes(attributes);
+
+ topicConfigManager.updateTopicConfig(topicConfig);
+
+ attributes.put("+long.range.key", "16");
+ topicConfig.setAttributes(attributes);
+ RuntimeException runtimeException = Assert.assertThrows(RuntimeException.class, () -> topicConfigManager.updateTopicConfig(topicConfig));
+ Assert.assertEquals("attempt to update an unchangeable attribute. key: long.range.key", runtimeException.getMessage());
+ }
+
+ @Test
+ public void testNormalQueryKeyOnGetting() {
+ String topic = "exist-topic";
+ String unchangeable = "bool.key";
+
+ supportAttributes(asList(
+ new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1"),
+ new BooleanAttribute("bool.key", false, false),
+ new LongRangeAttribute("long.range.key", true, 10, 20, 15)
+ ));
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put("+" + unchangeable, "true");
+
+ TopicConfig topicConfig = new TopicConfig();
+ topicConfig.setTopicName(topic);
+ topicConfig.setAttributes(attributes);
+
+ topicConfigManager.updateTopicConfig(topicConfig);
+
+ TopicConfig topicConfigUpdated = topicConfigManager.getTopicConfigTable().get(topic);
+ Assert.assertEquals(CQType.SimpleCQ, QueueTypeUtils.getCQType(topicConfigUpdated));
+
+ Assert.assertEquals("true", topicConfigUpdated.getAttributes().get(unchangeable));
+ }
+
+ private void supportAttributes(List<Attribute> supportAttributes) {
+ Map<String, Attribute> supportedAttributes = new HashMap<>();
+
+ for (Attribute supportAttribute : supportAttributes) {
+ supportedAttributes.put(supportAttribute.getName(), supportAttribute);
+ }
+
+ topicConfigManager = spy(topicConfigManager);
+ when(topicConfigManager.allAttributes()).thenReturn(supportedAttributes);
+ }
+}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
index 20b8f41..cd0d05b 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
@@ -56,10 +56,12 @@ import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
+import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.admin.ConsumeStats;
import org.apache.rocketmq.common.admin.TopicStatsTable;
+import org.apache.rocketmq.common.attribute.AttributeParser;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageBatch;
import org.apache.rocketmq.common.message.MessageClientIDSetter;
@@ -332,6 +334,7 @@ public class MQClientAPIImpl {
requestHeader.setTopicFilterType(topicConfig.getTopicFilterType().name());
requestHeader.setTopicSysFlag(topicConfig.getTopicSysFlag());
requestHeader.setOrder(topicConfig.isOrder());
+ requestHeader.setAttributes(AttributeParser.parseToString(topicConfig.getAttributes()));
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_AND_CREATE_TOPIC, requestHeader);
diff --git a/common/pom.xml b/common/pom.xml
index dd8ea6a..bbbc713 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -27,19 +27,6 @@
<artifactId>rocketmq-common</artifactId>
<name>rocketmq-common ${project.version}</name>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>6</source>
- <target>6</target>
- </configuration>
- </plugin>
- </plugins>
- </build>
-
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
similarity index 56%
copy from store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
copy to common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
index 18e7f5a..9c1e96f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
@@ -14,22 +14,27 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store.util;
+package org.apache.rocketmq.common;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.Attribute;
+import org.apache.rocketmq.common.attribute.EnumAttribute;
-public class QueueTypeUtils {
+import java.util.HashMap;
+import java.util.Map;
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
- }
+import static com.google.common.collect.Sets.newHashSet;
+
+public class TopicAttributes {
+ public static final EnumAttribute queueType = new EnumAttribute(
+ "queue.type",
+ false,
+ newHashSet("BatchCQ", "SimpleCQ"),
+ "SimpleCQ"
+ );
+ public static final Map<String, Attribute> ALL;
+
+ static {
+ ALL = new HashMap<>();
+ ALL.put(queueType.getName(), queueType);
}
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java b/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
index ec4d54b..610c3e2 100644
--- a/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
+++ b/common/src/main/java/org/apache/rocketmq/common/TopicConfig.java
@@ -18,6 +18,8 @@ package org.apache.rocketmq.common;
import org.apache.rocketmq.common.constant.PermName;
+import java.util.Map;
+
public class TopicConfig {
private static final String SEPARATOR = " ";
public static int defaultReadQueueNums = 16;
@@ -29,6 +31,7 @@ public class TopicConfig {
private TopicFilterType topicFilterType = TopicFilterType.SINGLE_TAG;
private int topicSysFlag = 0;
private boolean order = false;
+ private Map<String, String> attributes;
public TopicConfig() {
}
@@ -72,6 +75,8 @@ public class TopicConfig {
sb.append(SEPARATOR);
sb.append(this.topicFilterType);
+ // Leave the encode/decode [attributes] out for now
+
return sb.toString();
}
@@ -150,29 +155,29 @@ public class TopicConfig {
this.order = isOrder;
}
- @Override
- public boolean equals(final Object o) {
- if (this == o)
- return true;
- if (o == null || getClass() != o.getClass())
- return false;
+ public Map<String, String> getAttributes() {
+ return attributes;
+ }
+
+ public void setAttributes(Map<String, String> attributes) {
+ this.attributes = attributes;
+ }
- final TopicConfig that = (TopicConfig) o;
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
- if (readQueueNums != that.readQueueNums)
- return false;
- if (writeQueueNums != that.writeQueueNums)
- return false;
- if (perm != that.perm)
- return false;
- if (topicSysFlag != that.topicSysFlag)
- return false;
- if (order != that.order)
- return false;
- if (topicName != null ? !topicName.equals(that.topicName) : that.topicName != null)
- return false;
- return topicFilterType == that.topicFilterType;
+ TopicConfig that = (TopicConfig) o;
+ if (readQueueNums != that.readQueueNums) return false;
+ if (writeQueueNums != that.writeQueueNums) return false;
+ if (perm != that.perm) return false;
+ if (topicSysFlag != that.topicSysFlag) return false;
+ if (order != that.order) return false;
+ if (topicName != null ? !topicName.equals(that.topicName) : that.topicName != null) return false;
+ if (topicFilterType != that.topicFilterType) return false;
+ return attributes != null ? attributes.equals(that.attributes) : that.attributes == null;
}
@Override
@@ -184,6 +189,7 @@ public class TopicConfig {
result = 31 * result + (topicFilterType != null ? topicFilterType.hashCode() : 0);
result = 31 * result + topicSysFlag;
result = 31 * result + (order ? 1 : 0);
+ result = 31 * result + (attributes != null ? attributes.hashCode() : 0);
return result;
}
@@ -191,7 +197,7 @@ public class TopicConfig {
public String toString() {
return "TopicConfig [topicName=" + topicName + ", readQueueNums=" + readQueueNums
+ ", writeQueueNums=" + writeQueueNums + ", perm=" + PermName.perm2String(perm)
- + ", topicFilterType=" + topicFilterType + ", topicSysFlag=" + topicSysFlag + ", order="
- + order + "]";
+ + ", topicFilterType=" + topicFilterType + ", topicSysFlag=" + topicSysFlag + ", order=" + order
+ + ", attributes=" + attributes + "]";
}
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/attribute/Attribute.java
similarity index 56%
copy from store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
copy to common/src/main/java/org/apache/rocketmq/common/attribute/Attribute.java
index 18e7f5a..ba9be3b 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/Attribute.java
@@ -14,22 +14,32 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store.util;
+package org.apache.rocketmq.common.attribute;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
+public abstract class Attribute {
+ protected String name;
+ protected boolean changeable;
-public class QueueTypeUtils {
+ public abstract void verify(String value);
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
- }
+ public Attribute(String name, boolean changeable) {
+ this.name = name;
+ this.changeable = changeable;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public boolean isChangeable() {
+ return changeable;
+ }
+
+ public void setChangeable(boolean changeable) {
+ this.changeable = changeable;
}
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/attribute/AttributeParser.java b/common/src/main/java/org/apache/rocketmq/common/attribute/AttributeParser.java
new file mode 100644
index 0000000..7ee7afc
--- /dev/null
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/AttributeParser.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.attribute;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Strings;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class AttributeParser {
+ public static Map<String, String> parseToMap(String attributesModification) {
+ if (Strings.isNullOrEmpty(attributesModification)) {
+ return new HashMap<>();
+ }
+
+ // format: +key1=value1,+key2=value2,-key3,+key4=value4
+ Map<String, String> attributes = new HashMap<>();
+ String arraySeparator = ",";
+ String kvSeparator = "=";
+ String[] kvs = attributesModification.split(arraySeparator);
+ for (String kv : kvs) {
+ String key;
+ String value;
+ if (kv.contains(kvSeparator)) {
+ key = kv.split(kvSeparator)[0];
+ value = kv.split(kvSeparator)[1];
+ if (!key.contains("+")) {
+ throw new RuntimeException("add/alter attribute format is wrong: " + key);
+ }
+ } else {
+ key = kv;
+ value = "";
+ if (!key.contains("-")) {
+ throw new RuntimeException("delete attribute format is wrong: " + key);
+ }
+ }
+ String old = attributes.put(key, value);
+ if (old != null) {
+ throw new RuntimeException("key duplication: " + key);
+ }
+ }
+ return attributes;
+ }
+
+ public static String parseToString(Map<String, String> attributes) {
+ if (attributes == null || attributes.size() == 0) {
+ return "";
+ }
+
+ List<String> kvs = new ArrayList<>();
+ for (Map.Entry<String, String> entry : attributes.entrySet()) {
+
+ String value = entry.getValue();
+ if (Strings.isNullOrEmpty(value)) {
+ kvs.add(entry.getKey());
+ } else {
+ kvs.add(entry.getKey() + "=" + entry.getValue());
+ }
+ }
+ return Joiner.on(",").join(kvs);
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/attribute/BooleanAttribute.java
similarity index 54%
copy from store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
copy to common/src/main/java/org/apache/rocketmq/common/attribute/BooleanAttribute.java
index 18e7f5a..41ad748 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/BooleanAttribute.java
@@ -14,22 +14,28 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store.util;
+package org.apache.rocketmq.common.attribute;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
+import static com.google.common.base.Preconditions.checkNotNull;
-public class QueueTypeUtils {
+public class BooleanAttribute extends Attribute {
+ private final boolean defaultValue;
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
+ public BooleanAttribute(String name, boolean changeable, boolean defaultValue) {
+ super(name, changeable);
+ this.defaultValue = defaultValue;
+ }
+
+ @Override
+ public void verify(String value) {
+ checkNotNull(value);
+
+ if (!"false".equalsIgnoreCase(value) && !"true".equalsIgnoreCase(value)) {
+ throw new RuntimeException("boolean attribute format is wrong.");
}
}
+
+ public boolean getDefaultValue() {
+ return defaultValue;
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/attribute/EnumAttribute.java
similarity index 55%
copy from store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
copy to common/src/main/java/org/apache/rocketmq/common/attribute/EnumAttribute.java
index 18e7f5a..5353b8a 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/EnumAttribute.java
@@ -14,22 +14,28 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store.util;
+package org.apache.rocketmq.common.attribute;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
+import java.util.Set;
-public class QueueTypeUtils {
+public class EnumAttribute extends Attribute {
+ private final Set<String> universe;
+ private final String defaultValue;
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
+ public EnumAttribute(String name, boolean changeable, Set<String> universe, String defaultValue) {
+ super(name, changeable);
+ this.universe = universe;
+ this.defaultValue = defaultValue;
+ }
+
+ @Override
+ public void verify(String value) {
+ if (!this.universe.contains(value)) {
+ throw new RuntimeException("value is not in set: " + this.universe);
}
}
+
+ public String getDefaultValue() {
+ return defaultValue;
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/attribute/LongRangeAttribute.java
similarity index 52%
copy from store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
copy to common/src/main/java/org/apache/rocketmq/common/attribute/LongRangeAttribute.java
index 18e7f5a..eeeda72 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/LongRangeAttribute.java
@@ -14,22 +14,31 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.rocketmq.store.util;
+package org.apache.rocketmq.common.attribute;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
+import static java.lang.String.format;
-public class QueueTypeUtils {
+public class LongRangeAttribute extends Attribute {
+ private final long min;
+ private final long max;
+ private final long defaultValue;
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
+ public LongRangeAttribute(String name, boolean changeable, long min, long max, long defaultValue) {
+ super(name, changeable);
+ this.min = min;
+ this.max = max;
+ this.defaultValue = defaultValue;
+ }
+
+ @Override
+ public void verify(String value) {
+ long l = Long.parseLong(value);
+ if (l < min || l > max) {
+ throw new RuntimeException(format("value is not in range(%d, %d)", min, max));
}
}
+
+ public long getDefaultValue() {
+ return defaultValue;
+ }
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
index 290ec4c..2e381b3 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
@@ -26,6 +26,8 @@ import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
+import java.util.Map;
+
public class CreateTopicRequestHeader implements CommandCustomHeader {
@CFNotNull
private String topic;
@@ -42,6 +44,7 @@ public class CreateTopicRequestHeader implements CommandCustomHeader {
private Integer topicSysFlag;
@CFNotNull
private Boolean order = false;
+ private String attributes;
@CFNullable
private Boolean force = false;
@@ -130,4 +133,12 @@ public class CreateTopicRequestHeader implements CommandCustomHeader {
public void setForce(Boolean force) {
this.force = force;
}
+
+ public String getAttributes() {
+ return attributes;
+ }
+
+ public void setAttributes(String attributes) {
+ this.attributes = attributes;
+ }
}
diff --git a/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeParserTest.java b/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeParserTest.java
new file mode 100644
index 0000000..1239810
--- /dev/null
+++ b/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeParserTest.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.attribute;
+
+import com.google.common.collect.Maps;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.google.common.collect.Maps.newHashMap;
+import static org.junit.Assert.assertTrue;
+
+public class AttributeParserTest {
+ @Test
+ public void testParseToMap() {
+ Assert.assertEquals(0, AttributeParser.parseToMap(null).size());
+ AttributeParser.parseToMap("++=++");
+ AttributeParser.parseToMap("--");
+ Assert.assertThrows(RuntimeException.class, () -> AttributeParser.parseToMap("x"));
+ Assert.assertThrows(RuntimeException.class, () -> AttributeParser.parseToMap("+"));
+ Assert.assertThrows(RuntimeException.class, () -> AttributeParser.parseToMap("++"));
+ }
+
+ @Test
+ public void testParseToString() {
+ Assert.assertEquals("", AttributeParser.parseToString(null));
+ Assert.assertEquals("", AttributeParser.parseToString(newHashMap()));
+ HashMap<String, String> map = new HashMap<>();
+ int addSize = 10;
+ for (int i = 0; i < addSize; i++) {
+ map.put("+add.key" + i, "value" + i);
+ }
+ int deleteSize = 10;
+ for (int i = 0; i < deleteSize; i++) {
+ map.put("-delete.key" + i, "");
+ }
+ Assert.assertEquals(addSize + deleteSize, AttributeParser.parseToString(map).split(",").length);
+ }
+
+ @Test
+ public void testParseBetweenStringAndMapWithoutDistortion() {
+ List<String> testCases = Arrays.asList("-a", "+a=b,+c=d,+z=z,+e=e", "+a=b,-d", "+a=b", "-a,-b");
+ for (String testCase : testCases) {
+ assertTrue(Maps.difference(AttributeParser.parseToMap(testCase), AttributeParser.parseToMap(parse(testCase))).areEqual());
+ }
+ }
+
+ private String parse(String original) {
+ Map<String, String> stringStringMap = AttributeParser.parseToMap(original);
+ return AttributeParser.parseToString(stringStringMap);
+ }
+}
diff --git a/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeTest.java b/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeTest.java
new file mode 100644
index 0000000..39a12b9
--- /dev/null
+++ b/common/src/test/java/org/apache/rocketmq/common/attribute/AttributeTest.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.attribute;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static com.google.common.collect.Sets.newHashSet;
+
+public class AttributeTest {
+
+ @Test
+ public void testEnumAttribute() {
+ EnumAttribute enumAttribute = new EnumAttribute("enum.key", true, newHashSet("enum-1", "enum-2", "enum-3"), "enum-1");
+
+ Assert.assertThrows(RuntimeException.class, () -> enumAttribute.verify(""));
+ Assert.assertThrows(RuntimeException.class, () -> enumAttribute.verify("x"));
+ Assert.assertThrows(RuntimeException.class, () -> enumAttribute.verify("enum-4"));
+
+ enumAttribute.verify("enum-1");
+ enumAttribute.verify("enum-2");
+ enumAttribute.verify("enum-3");
+ }
+
+ @Test
+ public void testLongRangeAttribute() {
+ LongRangeAttribute longRangeAttribute = new LongRangeAttribute("long.range.key", true, 10, 20, 15);
+ Assert.assertThrows(RuntimeException.class, () -> longRangeAttribute.verify(""));
+ Assert.assertThrows(RuntimeException.class, () -> longRangeAttribute.verify(","));
+ Assert.assertThrows(RuntimeException.class, () -> longRangeAttribute.verify("a"));
+ Assert.assertThrows(RuntimeException.class, () -> longRangeAttribute.verify("-1"));
+ Assert.assertThrows(RuntimeException.class, () -> longRangeAttribute.verify("21"));
+
+ longRangeAttribute.verify("11");
+ longRangeAttribute.verify("10");
+ longRangeAttribute.verify("20");
+ }
+
+ @Test
+ public void testBooleanAttribute() {
+ BooleanAttribute booleanAttribute = new BooleanAttribute("bool.key", false, false);
+
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify(""));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify("a"));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify(","));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify("checked"));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify("1"));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify("0"));
+ Assert.assertThrows(RuntimeException.class, () -> booleanAttribute.verify("-1"));
+
+ booleanAttribute.verify("true");
+ booleanAttribute.verify("tRue");
+ booleanAttribute.verify("false");
+ booleanAttribute.verify("falSe");
+ }
+}
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
index 18e7f5a..612bf7e 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
@@ -16,13 +16,18 @@
*/
package org.apache.rocketmq.store.util;
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.StreamMessageStore;
import org.apache.rocketmq.store.queue.CQType;
+import java.util.Map;
+
public class QueueTypeUtils {
+ @Deprecated
public static CQType getCQType(MessageStore messageStore) {
if (messageStore instanceof DefaultMessageStore) {
return CQType.SimpleCQ;
@@ -32,4 +37,19 @@ public class QueueTypeUtils {
throw new RuntimeException("new cq type is not supported now.");
}
}
+
+ public static CQType getCQType(TopicConfig topicConfig) {
+ String attributeName = TopicAttributes.queueType.getName();
+
+ Map<String, String> attributes = topicConfig.getAttributes();
+ if (attributes == null || attributes.size() == 0) {
+ return CQType.valueOf(TopicAttributes.queueType.getDefaultValue());
+ }
+
+ if (attributes.containsKey(attributeName)) {
+ return CQType.valueOf(attributes.get(attributeName));
+ } else {
+ return CQType.valueOf(TopicAttributes.queueType.getDefaultValue());
+ }
+ }
}
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java b/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
index c33ae52..3caa477 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
@@ -16,12 +16,16 @@
*/
package org.apache.rocketmq.tools.command.topic;
+import java.util.Map;
import java.util.Set;
+
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.OptionGroup;
import org.apache.commons.cli.Options;
+import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.attribute.AttributeParser;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
import org.apache.rocketmq.remoting.RPCHook;
import org.apache.rocketmq.srvutil.ServerUtil;
@@ -55,6 +59,10 @@ public class UpdateTopicSubCommand implements SubCommand {
optionGroup.setRequired(true);
options.addOptionGroup(optionGroup);
+ opt = new Option("a", "attributes", true, "attribute(+a=b,+c=d,-e)");
+ opt.setRequired(false);
+ options.addOption(opt);
+
opt = new Option("t", "topic", true, "topic name");
opt.setRequired(true);
options.addOption(opt);
@@ -98,6 +106,12 @@ public class UpdateTopicSubCommand implements SubCommand {
topicConfig.setWriteQueueNums(8);
topicConfig.setTopicName(commandLine.getOptionValue('t').trim());
+ if (commandLine.hasOption('a')) {
+ String attributesModification = commandLine.getOptionValue('a').trim();
+ Map<String, String> attributes = AttributeParser.parseToMap(attributesModification);
+ topicConfig.setAttributes(attributes);
+ }
+
// readQueueNums
if (commandLine.hasOption('r')) {
topicConfig.setReadQueueNums(Integer.parseInt(commandLine.getOptionValue('r').trim()));
@@ -187,4 +201,5 @@ public class UpdateTopicSubCommand implements SubCommand {
defaultMQAdminExt.shutdown();
}
}
+
}
[rocketmq] 11/17: fix check style (#3703)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 5ce1b88cf07ddf656b34046054eaa8eb41eb5a67
Author: cserwen <cs...@163.com>
AuthorDate: Wed Jan 5 20:52:49 2022 +0800
fix check style (#3703)
---
.../rocketmq/broker/topic/TopicConfigManagerTest.java | 2 +-
.../apache/rocketmq/client/impl/MQClientAPIImpl.java | 1 -
.../org/apache/rocketmq/common/TopicAttributes.java | 4 ++--
.../common/statictopic/TopicQueueMappingTest.java | 17 +++++++++++++++++
.../common/statictopic/TopicQueueMappingUtilsTest.java | 17 +++++++++++++++++
.../org/apache/rocketmq/store/PutMessageContext.java | 16 ++++++++++++++++
.../java/org/apache/rocketmq/store/TopicQueueLock.java | 16 ++++++++++++++++
.../org/apache/rocketmq/store/util/QueueTypeUtils.java | 6 +++---
.../apache/rocketmq/store/queue/ConsumeQueueTest.java | 16 ++++++++++++++++
.../org/apache/rocketmq/store/queue/QueueTestBase.java | 16 ++++++++++++++++
.../apache/rocketmq/test/statictopic/StaticTopicIT.java | 17 +++++++++++++++++
11 files changed, 121 insertions(+), 7 deletions(-)
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
index 9853c36..79cc01a 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
@@ -123,7 +123,7 @@ public class TopicConfigManagerTest {
@Test
public void testAddWrongValueOnCreating() {
Map<String, String> attributes = new HashMap<>();
- attributes.put("+" + TopicAttributes.queueType.getName(), "wrong-value");
+ attributes.put("+" + TopicAttributes.QUEUE_TYPE.getName(), "wrong-value");
TopicConfig topicConfig = new TopicConfig();
topicConfig.setTopicName("new-topic");
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
index cd0d05b..066ffd5 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
@@ -56,7 +56,6 @@ import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
-import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.admin.ConsumeStats;
diff --git a/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
index 9c1e96f..7bed217 100644
--- a/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
+++ b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
@@ -25,7 +25,7 @@ import java.util.Map;
import static com.google.common.collect.Sets.newHashSet;
public class TopicAttributes {
- public static final EnumAttribute queueType = new EnumAttribute(
+ public static final EnumAttribute QUEUE_TYPE = new EnumAttribute(
"queue.type",
false,
newHashSet("BatchCQ", "SimpleCQ"),
@@ -35,6 +35,6 @@ public class TopicAttributes {
static {
ALL = new HashMap<>();
- ALL.put(queueType.getName(), queueType);
+ ALL.put(QUEUE_TYPE.getName(), QUEUE_TYPE);
}
}
diff --git a/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingTest.java b/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingTest.java
index f1a2b21..4e93275 100644
--- a/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingTest.java
+++ b/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package org.apache.rocketmq.common.statictopic;
import com.alibaba.fastjson.JSON;
diff --git a/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingUtilsTest.java b/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingUtilsTest.java
index f5cd0ef..c9bd22e 100644
--- a/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingUtilsTest.java
+++ b/common/src/test/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingUtilsTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package org.apache.rocketmq.common.statictopic;
import org.apache.rocketmq.common.TopicConfig;
diff --git a/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
index c3dd89b..d4c160d 100644
--- a/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
+++ b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.rocketmq.store;
public class PutMessageContext {
diff --git a/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
index 0338cf7..97c50b9 100644
--- a/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
+++ b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.rocketmq.store;
import java.util.ArrayList;
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
index 612bf7e..1067337 100644
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ b/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
@@ -39,17 +39,17 @@ public class QueueTypeUtils {
}
public static CQType getCQType(TopicConfig topicConfig) {
- String attributeName = TopicAttributes.queueType.getName();
+ String attributeName = TopicAttributes.QUEUE_TYPE.getName();
Map<String, String> attributes = topicConfig.getAttributes();
if (attributes == null || attributes.size() == 0) {
- return CQType.valueOf(TopicAttributes.queueType.getDefaultValue());
+ return CQType.valueOf(TopicAttributes.QUEUE_TYPE.getDefaultValue());
}
if (attributes.containsKey(attributeName)) {
return CQType.valueOf(attributes.get(attributeName));
} else {
- return CQType.valueOf(TopicAttributes.queueType.getDefaultValue());
+ return CQType.valueOf(TopicAttributes.QUEUE_TYPE.getDefaultValue());
}
}
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
index 9f906d9..4345f0e 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.rocketmq.store.queue;
import org.apache.rocketmq.store.ConsumeQueueExt;
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
index 6df5549..a8e9f94 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.rocketmq.store.queue;
import org.apache.rocketmq.common.BrokerConfig;
diff --git a/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java b/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
index 5b3e5fe..5368cdf 100644
--- a/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
+++ b/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package org.apache.rocketmq.test.statictopic;
import com.google.common.collect.ImmutableList;
[rocketmq] 09/17: Fix test for consumer offset
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit a767cc142423b62fc50fc21fb50c1b741203af2e
Author: dongeforever <do...@apache.org>
AuthorDate: Wed Jan 5 17:11:31 2022 +0800
Fix test for consumer offset
---
.../apache/rocketmq/broker/processor/AdminBrokerProcessor.java | 10 ++++++----
.../org/apache/rocketmq/test/statictopic/StaticTopicIT.java | 3 +++
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index 568a728..5b8a19f 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -1176,6 +1176,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
continue;
}
+ TopicQueueMappingDetail mappingDetail = this.brokerController.getTopicQueueMappingManager().getTopicQueueMapping(topic);
+
{
SubscriptionData findSubscriptionData =
this.brokerController.getConsumerManager().findSubscriptionData(requestHeader.getConsumerGroup(), topic);
@@ -1206,14 +1208,14 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
// the consumerOffset cannot be zero for static topic because of the "double read check" strategy
// just remain the logic for dynamic topic
// maybe we should remove it in the future
- if (consumerOffset < 0)
- consumerOffset = 0;
+ if (mappingDetail == null) {
+ if (consumerOffset < 0)
+ consumerOffset = 0;
+ }
offsetWrapper.setBrokerOffset(brokerOffset);
offsetWrapper.setConsumerOffset(consumerOffset);
- // the consumeOffset is not in this broker for static topic
- // and may get the wrong result
long timeOffset = consumerOffset - 1;
if (timeOffset >= 0) {
long lastTimestamp = this.brokerController.getMessageStore().getMessageStoreTimeStamp(topic, i, timeOffset);
diff --git a/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java b/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
index 3e8f146..5b3e5fe 100644
--- a/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
+++ b/test/src/test/java/org/apache/rocketmq/test/statictopic/StaticTopicIT.java
@@ -292,6 +292,7 @@ public class StaticTopicIT extends BaseConf {
String group = initConsumerGroup();
RMQNormalProducer producer = getProducer(nsAddr, topic);
RMQNormalConsumer consumer = getConsumer(nsAddr, group, topic, "*", new RMQNormalListener());
+ long start = System.currentTimeMillis();
int queueNum = 10;
int msgEachQueue = 100;
@@ -314,6 +315,7 @@ public class StaticTopicIT extends BaseConf {
Assert.assertNotNull(wrapper);
Assert.assertEquals(msgEachQueue, wrapper.getBrokerOffset());
Assert.assertEquals(msgEachQueue, wrapper.getConsumerOffset());
+ Assert.assertTrue(wrapper.getLastTimestamp() > start);
}
List<String> brokers = ImmutableList.of(broker2Name, broker3Name, broker1Name);
@@ -332,6 +334,7 @@ public class StaticTopicIT extends BaseConf {
Assert.assertNotNull(wrapper);
Assert.assertEquals(msgEachQueue + brokers.size() * TopicQueueMappingUtils.DEFAULT_BLOCK_SEQ_SIZE, wrapper.getBrokerOffset());
Assert.assertEquals(msgEachQueue, wrapper.getConsumerOffset());
+ Assert.assertTrue(wrapper.getLastTimestamp() > start);
}
consumer = getConsumer(nsAddr, group, topic, "*", new RMQNormalListener());
consumeMessagesAndCheck(producer, consumer, topic, queueNum, msgEachQueue, 1, brokers.size());
[rocketmq] 10/17: Fix check style
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 372e42ff454836c7787c33316934c641e1359ebf
Author: dongeforever <do...@apache.org>
AuthorDate: Wed Jan 5 17:29:26 2022 +0800
Fix check style
---
.../rocketmq/common/protocol/header/CreateTopicRequestHeader.java | 2 --
.../main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java | 8 +++-----
.../apache/rocketmq/common/statictopic/TopicQueueMappingOne.java | 5 +----
3 files changed, 4 insertions(+), 11 deletions(-)
diff --git a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
index 2e381b3..c3c59d4 100644
--- a/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
+++ b/common/src/main/java/org/apache/rocketmq/common/protocol/header/CreateTopicRequestHeader.java
@@ -26,8 +26,6 @@ import org.apache.rocketmq.remoting.annotation.CFNotNull;
import org.apache.rocketmq.remoting.annotation.CFNullable;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
-import java.util.Map;
-
public class CreateTopicRequestHeader implements CommandCustomHeader {
@CFNotNull
private String topic;
diff --git a/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java b/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
index 3782ab0..c5cbc74 100644
--- a/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
+++ b/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
@@ -18,6 +18,9 @@ package org.apache.rocketmq.common.rpc;
import io.netty.util.concurrent.ImmediateEventExecutor;
import io.netty.util.concurrent.Promise;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Future;
import org.apache.rocketmq.common.admin.TopicStatsTable;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.RequestCode;
@@ -28,7 +31,6 @@ import org.apache.rocketmq.common.protocol.header.GetMinOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.PullMessageResponseHeader;
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.SearchOffsetResponseHeader;
-import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
import org.apache.rocketmq.remoting.InvokeCallback;
@@ -37,10 +39,6 @@ import org.apache.rocketmq.remoting.netty.ResponseFuture;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.Future;
-
public class RpcClientImpl implements RpcClient {
private ClientMetadata clientMetadata;
diff --git a/common/src/main/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingOne.java b/common/src/main/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingOne.java
index 636f1d5..597b6ab 100644
--- a/common/src/main/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingOne.java
+++ b/common/src/main/java/org/apache/rocketmq/common/statictopic/TopicQueueMappingOne.java
@@ -16,11 +16,8 @@
*/
package org.apache.rocketmq.common.statictopic;
-import org.apache.commons.lang3.builder.EqualsBuilder;
-import org.apache.commons.lang3.builder.HashCodeBuilder;
-import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
-
import java.util.List;
+import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
public class TopicQueueMappingOne extends RemotingSerializable {
[rocketmq] 05/17: [benchmark] Add clientRebalanceEnable
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit c998f3fe68fbd7fcf993ed7c094564fa66ef4a77
Author: zhangyang21 <zh...@xiaomi.com>
AuthorDate: Thu Nov 18 15:49:13 2021 +0800
[benchmark] Add clientRebalanceEnable
Signed-off-by: zhangyang21 <zh...@xiaomi.com>
---
.../main/java/org/apache/rocketmq/example/benchmark/Consumer.java | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/example/src/main/java/org/apache/rocketmq/example/benchmark/Consumer.java b/example/src/main/java/org/apache/rocketmq/example/benchmark/Consumer.java
index 7d26509..046ac94 100644
--- a/example/src/main/java/org/apache/rocketmq/example/benchmark/Consumer.java
+++ b/example/src/main/java/org/apache/rocketmq/example/benchmark/Consumer.java
@@ -64,6 +64,7 @@ public class Consumer {
final double failRate = commandLine.hasOption('r') ? Double.parseDouble(commandLine.getOptionValue('r').trim()) : 0.0;
final boolean msgTraceEnable = commandLine.hasOption('m') && Boolean.parseBoolean(commandLine.getOptionValue('m'));
final boolean aclEnable = commandLine.hasOption('a') && Boolean.parseBoolean(commandLine.getOptionValue('a'));
+ final boolean clientRebalanceEnable = commandLine.hasOption('c') ? Boolean.parseBoolean(commandLine.getOptionValue('c')) : true;
String group = groupPrefix;
if (Boolean.parseBoolean(isSuffixEnable)) {
@@ -132,6 +133,7 @@ public class Consumer {
consumer.setConsumeThreadMin(threadCount);
consumer.setConsumeThreadMax(threadCount);
consumer.setInstanceName(Long.toString(System.currentTimeMillis()));
+ consumer.setClientRebalance(clientRebalanceEnable);
if (filterType == null || expression == null) {
consumer.subscribe(topic, "*");
@@ -218,6 +220,10 @@ public class Consumer {
opt.setRequired(false);
options.addOption(opt);
+ opt = new Option("c", "clientRebalanceEnable", true, "Client Rebalance Enable, Default: true");
+ opt.setRequired(false);
+ options.addOption(opt);
+
return options;
}
[rocketmq] 03/17: [RIP-19] Pop Consuming (tools)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 6ba93446b3a2fb6763632fc16aef47c2af14768a
Author: hill007299 <hi...@126.com>
AuthorDate: Tue Mar 9 11:21:48 2021 +0800
[RIP-19] Pop Consuming (tools)
Conflicts:
tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExt.java
tools/src/main/java/org/apache/rocketmq/tools/admin/MQAdminExt.java
tools/src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java
---
.../src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java b/tools/src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java
index bfac860..0207465 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/command/MQAdminStartup.java
@@ -48,6 +48,7 @@ import org.apache.rocketmq.tools.command.consumer.ConsumerProgressSubCommand;
import org.apache.rocketmq.tools.command.consumer.ConsumerStatusSubCommand;
import org.apache.rocketmq.tools.command.consumer.DeleteSubscriptionGroupCommand;
import org.apache.rocketmq.tools.command.consumer.GetConsumerConfigSubCommand;
+import org.apache.rocketmq.tools.command.consumer.SetConsumeModeSubCommand;
import org.apache.rocketmq.tools.command.consumer.StartMonitoringSubCommand;
import org.apache.rocketmq.tools.command.consumer.UpdateSubGroupSubCommand;
import org.apache.rocketmq.tools.command.export.ExportConfigsCommand;
@@ -160,6 +161,7 @@ public class MQAdminStartup {
initCommand(new UpdateTopicSubCommand());
initCommand(new DeleteTopicSubCommand());
initCommand(new UpdateSubGroupSubCommand());
+ initCommand(new SetConsumeModeSubCommand());
initCommand(new DeleteSubscriptionGroupCommand());
initCommand(new UpdateBrokerConfigSubCommand());
initCommand(new UpdateTopicPermSubCommand());
[rocketmq] 12/17: [ISSUE #3708] Both CQ and BCQ need to be supported in DefaultMessageStore. (#3712)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit e8cf1338743a3f778145c7d4b4449769c1e5b18a
Author: Hongjian Fei <er...@163.com>
AuthorDate: Thu Jan 6 10:56:05 2022 +0800
[ISSUE #3708] Both CQ and BCQ need to be supported in DefaultMessageStore. (#3712)
* [RIP-26] Both CQ and BCQ will be supported in DefaultMessageStore.
* [RIP-26] Both CQ and BCQ will be supported in DefaultMessageStore.
---
.../apache/rocketmq/broker/BrokerController.java | 11 +-
.../broker/processor/AdminBrokerProcessor.java | 1 -
.../broker/processor/ConsumerManageProcessor.java | 2 -
.../broker/processor/SendMessageProcessor.java | 7 +-
.../broker/topic/TopicConfigManagerTest.java | 9 +-
.../java/org/apache/rocketmq/client/MQAdmin.java | 14 +-
.../client/consumer/DefaultMQPullConsumer.java | 7 +-
.../client/consumer/DefaultMQPushConsumer.java | 6 +-
.../apache/rocketmq/client/impl/MQAdminImpl.java | 6 +-
.../impl/consumer/DefaultMQPullConsumerImpl.java | 2 +-
.../impl/consumer/DefaultMQPushConsumerImpl.java | 2 +-
.../impl/producer/DefaultMQProducerImpl.java | 2 +-
.../client/producer/DefaultMQProducer.java | 9 +-
.../apache/rocketmq/common/TopicAttributes.java | 4 +-
.../apache/rocketmq/common/attribute}/CQType.java | 2 +-
.../rocketmq/common/utils/QueueTypeUtils.java | 51 +
.../java/org/apache/rocketmq/store/CommitLog.java | 45 +-
.../org/apache/rocketmq/store/ConsumeQueue.java | 19 +-
.../apache/rocketmq/store/DefaultMessageStore.java | 223 +-
.../org/apache/rocketmq/store/MappedFileQueue.java | 16 +-
.../org/apache/rocketmq/store/MessageStore.java | 17 +-
.../apache/rocketmq/store/PutMessageContext.java | 67 +-
.../java/org/apache/rocketmq/store/StoreUtil.java | 4 -
.../apache/rocketmq/store/StreamMessageStore.java | 2573 --------------------
.../org/apache/rocketmq/store/TopicQueueLock.java | 11 +-
.../rocketmq/store/config/MessageStoreConfig.java | 11 -
.../rocketmq/store/dledger/DLedgerCommitLog.java | 3 +-
.../apache/rocketmq/store/logfile/MappedFile.java | 1 -
.../rocketmq/store/queue/BatchConsumeQueue.java | 28 +-
.../store/queue/ConsumeQueueInterface.java | 2 +
.../rocketmq/store/queue/ConsumeQueueStore.java | 290 ++-
.../rocketmq/store/queue/FileQueueLifeCycle.java | 2 +
.../rocketmq/store/queue/QueueOffsetAssigner.java | 60 +
.../apache/rocketmq/store/util/QueueTypeUtils.java | 55 -
.../store/DefaultMessageStoreShutDownTest.java | 3 +-
.../rocketmq/store/DefaultMessageStoreTest.java | 3 +-
.../store/queue/BatchConsumeMessageTest.java | 274 ++-
.../store/queue/BatchConsumeQueueTest.java | 27 +-
.../rocketmq/store/queue/ConsumeQueueTest.java | 3 +-
.../apache/rocketmq/store/queue/QueueTestBase.java | 28 +-
.../rocketmq/test/util/MQAdminTestUtils.java | 8 +-
.../org/apache/rocketmq/test/base/BaseConf.java | 35 +-
.../rocketmq/test/base/IntegrationTestBase.java | 49 +-
.../base/dledger/DLedgerProduceAndConsumeIT.java | 5 +-
.../test/client/producer/batch/BatchSendIT.java | 37 +-
.../rocketmq/tools/admin/DefaultMQAdminExt.java | 8 +-
.../tools/admin/DefaultMQAdminExtImpl.java | 16 +-
.../tools/command/topic/UpdateTopicSubCommand.java | 1 -
48 files changed, 912 insertions(+), 3147 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
index 6f9563b..7cf48c9 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java
@@ -96,11 +96,9 @@ import org.apache.rocketmq.srvutil.FileWatchService;
import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.MessageArrivingListener;
import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.dledger.DLedgerCommitLog;
-import org.apache.rocketmq.store.queue.CQType;
import org.apache.rocketmq.store.stats.BrokerStats;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
@@ -298,14 +296,9 @@ public class BrokerController {
if (result) {
try {
- MessageStore messageStore;
- if (Objects.equals(CQType.BatchCQ.toString(), this.messageStoreConfig.getDefaultCQType())) {
- messageStore = new StreamMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
- } else {
- messageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
- }
+ this.messageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
+ ((DefaultMessageStore) this.messageStore).setTopicConfigTable(topicConfigManager.getTopicConfigTable());
- this.messageStore = messageStore;
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
((DLedgerCommitLog) messageStore.getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index 5b8a19f..c1819a5 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -33,7 +33,6 @@ import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
-import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.admin.ConsumeStats;
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
index a266442..31a7993 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
@@ -32,12 +32,10 @@ import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetResponseHe
import org.apache.rocketmq.common.rpc.RpcClientUtils;
import org.apache.rocketmq.common.rpc.RpcRequest;
import org.apache.rocketmq.common.rpc.RpcResponse;
-import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
-import org.apache.rocketmq.common.sysflag.PullSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.common.RemotingHelper;
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
index 3e3173e..a3f5c4e 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
@@ -18,9 +18,9 @@ package org.apache.rocketmq.broker.processor;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
-import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ThreadLocalRandom;
@@ -42,7 +42,6 @@ import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.help.FAQUrl;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageClientIDSetter;
-import org.apache.rocketmq.common.message.MessageClientIDSetter;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
@@ -56,6 +55,7 @@ import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.netty.RemotingResponseCallback;
@@ -63,7 +63,6 @@ import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.store.MessageExtBatch;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.PutMessageResult;
-import org.apache.rocketmq.store.StoreUtil;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
@@ -684,7 +683,7 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
CompletableFuture<PutMessageResult> putMessageResult;
- if (StoreUtil.isStreamMode(this.brokerController.getMessageStore()) && MessageClientIDSetter.getUniqID(messageExtBatch) != null) {
+ if (QueueTypeUtils.isBatchCq(Optional.of(topicConfig)) && MessageClientIDSetter.getUniqID(messageExtBatch) != null) {
// newly introduced inner-batch message
messageExtBatch.setSysFlag(messageExtBatch.getSysFlag() | MessageSysFlag.NEED_UNWRAP_FLAG);
messageExtBatch.setSysFlag(messageExtBatch.getSysFlag() | MessageSysFlag.INNER_BATCH_FLAG);
diff --git a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
index 79cc01a..6b4f30d 100644
--- a/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
+++ b/broker/src/test/java/org/apache/rocketmq/broker/topic/TopicConfigManagerTest.java
@@ -24,9 +24,9 @@ import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.attribute.BooleanAttribute;
import org.apache.rocketmq.common.attribute.EnumAttribute;
import org.apache.rocketmq.common.attribute.LongRangeAttribute;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.apache.rocketmq.store.queue.CQType;
-import org.apache.rocketmq.store.util.QueueTypeUtils;
+import org.apache.rocketmq.common.attribute.CQType;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
@@ -37,6 +37,7 @@ import org.mockito.junit.MockitoJUnitRunner;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import static com.google.common.collect.Sets.newHashSet;
import static java.util.Arrays.asList;
@@ -123,7 +124,7 @@ public class TopicConfigManagerTest {
@Test
public void testAddWrongValueOnCreating() {
Map<String, String> attributes = new HashMap<>();
- attributes.put("+" + TopicAttributes.QUEUE_TYPE.getName(), "wrong-value");
+ attributes.put("+" + TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName(), "wrong-value");
TopicConfig topicConfig = new TopicConfig();
topicConfig.setTopicName("new-topic");
@@ -300,7 +301,7 @@ public class TopicConfigManagerTest {
topicConfigManager.updateTopicConfig(topicConfig);
TopicConfig topicConfigUpdated = topicConfigManager.getTopicConfigTable().get(topic);
- Assert.assertEquals(CQType.SimpleCQ, QueueTypeUtils.getCQType(topicConfigUpdated));
+ Assert.assertEquals(CQType.SimpleCQ, QueueTypeUtils.getCQType(Optional.of(topicConfigUpdated)));
Assert.assertEquals("true", topicConfigUpdated.getAttributes().get(unchangeable));
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/MQAdmin.java b/client/src/main/java/org/apache/rocketmq/client/MQAdmin.java
index 63b2d14..79386bd 100644
--- a/client/src/main/java/org/apache/rocketmq/client/MQAdmin.java
+++ b/client/src/main/java/org/apache/rocketmq/client/MQAdmin.java
@@ -22,29 +22,31 @@ import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.remoting.exception.RemotingException;
+import java.util.Map;
+
/**
* Base interface for MQ management
*/
public interface MQAdmin {
/**
* Creates an topic
- *
- * @param key accesskey
+ * @param key accesskey
* @param newTopic topic name
* @param queueNum topic's queue number
+ * @param attributes
*/
- void createTopic(final String key, final String newTopic, final int queueNum)
+ void createTopic(final String key, final String newTopic, final int queueNum, Map<String, String> attributes)
throws MQClientException;
/**
* Creates an topic
- *
- * @param key accesskey
+ * @param key accesskey
* @param newTopic topic name
* @param queueNum topic's queue number
* @param topicSysFlag topic system flag
+ * @param attributes
*/
- void createTopic(String key, String newTopic, int queueNum, int topicSysFlag)
+ void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes)
throws MQClientException;
/**
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
index 5829f77..3b893bb 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPullConsumer.java
@@ -17,6 +17,7 @@
package org.apache.rocketmq.client.consumer;
import java.util.HashSet;
+import java.util.Map;
import java.util.Set;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.QueryResult;
@@ -124,8 +125,8 @@ public class DefaultMQPullConsumer extends ClientConfig implements MQPullConsume
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, withNamespace(newTopic), queueNum, 0);
+ public void createTopic(String key, String newTopic, int queueNum, Map<String, String> attributes) throws MQClientException {
+ createTopic(key, withNamespace(newTopic), queueNum, 0, null);
}
/**
@@ -133,7 +134,7 @@ public class DefaultMQPullConsumer extends ClientConfig implements MQPullConsume
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
this.defaultMQPullConsumerImpl.createTopic(key, withNamespace(newTopic), queueNum, topicSysFlag);
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
index fce6b64..5ef2b63 100644
--- a/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
@@ -431,8 +431,8 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, withNamespace(newTopic), queueNum, 0);
+ public void createTopic(String key, String newTopic, int queueNum, Map<String, String> attributes) throws MQClientException {
+ createTopic(key, withNamespace(newTopic), queueNum, 0, null);
}
@Override
@@ -448,7 +448,7 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
this.defaultMQPushConsumerImpl.createTopic(key, withNamespace(newTopic), queueNum, topicSysFlag);
}
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
index 0a6e005..73e35c5 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
@@ -21,6 +21,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
@@ -76,10 +77,10 @@ public class MQAdminImpl {
}
public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, newTopic, queueNum, 0);
+ createTopic(key, newTopic, queueNum, 0, null);
}
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
try {
Validators.checkTopic(newTopic);
Validators.isSystemTopic(newTopic);
@@ -100,6 +101,7 @@ public class MQAdminImpl {
topicConfig.setReadQueueNums(queueNum);
topicConfig.setWriteQueueNums(queueNum);
topicConfig.setTopicSysFlag(topicSysFlag);
+ topicConfig.setAttributes(attributes);
boolean createOK = false;
for (int i = 0; i < 5; i++) {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
index 05d3d48..497b274 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
@@ -100,7 +100,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
this.isRunning();
- this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag);
+ this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag, null);
}
private void isRunning() throws MQClientException {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
index 2fa3830..abe3ca7 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
@@ -184,7 +184,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
}
public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
- this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag);
+ this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag, null);
}
public Set<MessageQueue> fetchSubscribeMessageQueues(String topic) throws MQClientException {
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
index bf2ca28..c1ec308 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
@@ -447,7 +447,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
Validators.checkTopic(newTopic);
Validators.isSystemTopic(newTopic);
- this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag);
+ this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag, null);
}
private void makeSureStateOK() throws MQClientException {
diff --git a/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java b/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
index 230785c..8c046aa 100644
--- a/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
+++ b/client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
@@ -19,6 +19,7 @@ package org.apache.rocketmq.client.producer;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.concurrent.CopyOnWriteArraySet;
import java.util.concurrent.ExecutorService;
@@ -757,12 +758,13 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
* @param key accesskey
* @param newTopic topic name
* @param queueNum topic's queue number
+ * @param attributes
* @throws MQClientException if there is any client error.
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, withNamespace(newTopic), queueNum, 0);
+ public void createTopic(String key, String newTopic, int queueNum, Map<String, String> attributes) throws MQClientException {
+ createTopic(key, withNamespace(newTopic), queueNum, 0, null);
}
/**
@@ -773,11 +775,12 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
* @param newTopic topic name
* @param queueNum topic's queue number
* @param topicSysFlag topic system flag
+ * @param attributes
* @throws MQClientException if there is any client error.
*/
@Deprecated
@Override
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
this.defaultMQProducerImpl.createTopic(key, withNamespace(newTopic), queueNum, topicSysFlag);
}
diff --git a/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
index 7bed217..add383f 100644
--- a/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
+++ b/common/src/main/java/org/apache/rocketmq/common/TopicAttributes.java
@@ -25,7 +25,7 @@ import java.util.Map;
import static com.google.common.collect.Sets.newHashSet;
public class TopicAttributes {
- public static final EnumAttribute QUEUE_TYPE = new EnumAttribute(
+ public static final EnumAttribute QUEUE_TYPE_ATTRIBUTE = new EnumAttribute(
"queue.type",
false,
newHashSet("BatchCQ", "SimpleCQ"),
@@ -35,6 +35,6 @@ public class TopicAttributes {
static {
ALL = new HashMap<>();
- ALL.put(QUEUE_TYPE.getName(), QUEUE_TYPE);
+ ALL.put(QUEUE_TYPE_ATTRIBUTE.getName(), QUEUE_TYPE_ATTRIBUTE);
}
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/CQType.java b/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
similarity index 94%
rename from store/src/main/java/org/apache/rocketmq/store/queue/CQType.java
rename to common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
index efd65af..6bd6ad2 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/CQType.java
+++ b/common/src/main/java/org/apache/rocketmq/common/attribute/CQType.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.rocketmq.store.queue;
+package org.apache.rocketmq.common.attribute;
public enum CQType {
SimpleCQ,
diff --git a/common/src/main/java/org/apache/rocketmq/common/utils/QueueTypeUtils.java b/common/src/main/java/org/apache/rocketmq/common/utils/QueueTypeUtils.java
new file mode 100644
index 0000000..e2f006e
--- /dev/null
+++ b/common/src/main/java/org/apache/rocketmq/common/utils/QueueTypeUtils.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.utils;
+
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.attribute.CQType;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+public class QueueTypeUtils {
+
+ public static boolean isBatchCq(Optional<TopicConfig> topicConfig) {
+ return Objects.equals(CQType.BatchCQ, getCQType(topicConfig));
+ }
+
+ public static CQType getCQType(Optional<TopicConfig> topicConfig) {
+ if (!topicConfig.isPresent()) {
+ return CQType.valueOf(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getDefaultValue());
+ }
+
+ String attributeName = TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName();
+
+ Map<String, String> attributes = topicConfig.get().getAttributes();
+ if (attributes == null || attributes.size() == 0) {
+ return CQType.valueOf(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getDefaultValue());
+ }
+
+ if (attributes.containsKey(attributeName)) {
+ return CQType.valueOf(attributes.get(attributeName));
+ } else {
+ return CQType.valueOf(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getDefaultValue());
+ }
+ }
+}
\ No newline at end of file
diff --git a/store/src/main/java/org/apache/rocketmq/store/CommitLog.java b/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
index 062c269..2ae84eb 100644
--- a/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
+++ b/store/src/main/java/org/apache/rocketmq/store/CommitLog.java
@@ -24,10 +24,10 @@ import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
@@ -36,6 +36,7 @@ import java.util.concurrent.TimeoutException;
import java.util.function.Supplier;
import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.message.MessageAccessor;
@@ -44,6 +45,7 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
@@ -51,9 +53,8 @@ import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.ha.HAService;
import org.apache.rocketmq.store.logfile.MappedFile;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.store.schedule.ScheduleMessageService;
-import org.apache.rocketmq.store.util.QueueTypeUtils;
/**
* Store all metadata downtime for recovery, data protection reliability
@@ -433,8 +434,8 @@ public class CommitLog implements Swappable {
private void setBatchSizeIfNeeded(Map<String, String> propertiesMap, DispatchRequest dispatchRequest) {
if (null != propertiesMap && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_NUM) && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_BASE)) {
- dispatchRequest.setMsgBaseOffset(Long.valueOf(propertiesMap.get(MessageConst.PROPERTY_INNER_BASE)));
- dispatchRequest.setBatchSize(Short.valueOf(propertiesMap.get(MessageConst.PROPERTY_INNER_NUM)));
+ dispatchRequest.setMsgBaseOffset(Long.parseLong(propertiesMap.get(MessageConst.PROPERTY_INNER_BASE)));
+ dispatchRequest.setBatchSize(Short.parseShort(propertiesMap.get(MessageConst.PROPERTY_INNER_NUM)));
}
}
@@ -672,7 +673,7 @@ public class CommitLog implements Swappable {
topicQueueLock.lock(topicQueueKey);
try {
- defaultMessageStore.assignOffset(topicQueueKey, msg, getBatchNum(msg));
+ defaultMessageStore.assignOffset(topicQueueKey, msg, getMessageNum(msg));
PutMessageResult encodeResult = putMessageThreadLocal.getEncoder().encode(msg);
if (encodeResult != null) {
@@ -1031,15 +1032,6 @@ public class CommitLog implements Swappable {
return this.mappedFileQueue.retryDeleteFirstFile(intervalForcibly);
}
- public void removeQueueFromTopicQueueTable(final String topic, final int queueId) {
- String key = topic + "-" + queueId;
- synchronized (this) {
- this.defaultMessageStore.removeOffsetTable(key);
- }
-
- log.info("removeQueueFromTopicQueueTable OK Topic: {} QueueId: {}", topic, queueId);
- }
-
public void checkSelf() {
mappedFileQueue.checkSelf();
}
@@ -1058,19 +1050,24 @@ public class CommitLog implements Swappable {
return diff;
}
- protected short getBatchNum(MessageExtBrokerInner msgInner) {
- short batchNum = 1;
+ protected short getMessageNum(MessageExtBrokerInner msgInner) {
+ short messageNum = 1;
// IF inner batch, build batchQueueOffset and batchNum property.
- CQType cqType = QueueTypeUtils.getCQType(defaultMessageStore);
- if (MessageSysFlag.check(msgInner.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG) || CQType.BatchCQ.equals(cqType)) {
+ CQType cqType = getCqType(msgInner);
+ if (MessageSysFlag.check(msgInner.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG) || CQType.BatchCQ.equals(cqType)) {
if (msgInner.getProperty(MessageConst.PROPERTY_INNER_NUM) != null) {
- batchNum = Short.parseShort(msgInner.getProperty(MessageConst.PROPERTY_INNER_NUM));
- batchNum = batchNum >= 1 ? batchNum : 1;
+ messageNum = Short.parseShort(msgInner.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ messageNum = messageNum >= 1 ? messageNum : 1;
}
}
- return batchNum;
+ return messageNum;
+ }
+
+ private CQType getCqType(MessageExtBrokerInner msgInner) {
+ Optional<TopicConfig> topicConfig = this.defaultMessageStore.getTopicConfig(msgInner.getTopic());
+ return QueueTypeUtils.getCQType(topicConfig);
}
abstract class FlushCommitLogService extends ServiceThread {
@@ -1488,7 +1485,7 @@ public class CommitLog implements Swappable {
Long queueOffset = msgInner.getQueueOffset();
// this msg maybe a inner-batch msg.
- short batchNum = getBatchNum(msgInner);
+ short messageNum = getMessageNum(msgInner);
// Transaction messages that require special handling
final int tranType = MessageSysFlag.getTransactionValue(msgInner.getSysFlag());
@@ -1545,7 +1542,7 @@ public class CommitLog implements Swappable {
CommitLog.this.getMessageStore().getPerfCounter().endTick("WRITE_MEMORY_TIME_MS");
msgInner.setEncodedBuff(null);
return new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgIdSupplier,
- msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills, batchNum);
+ msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills, messageNum);
}
public AppendMessageResult doAppend(final long fileFromOffset, final ByteBuffer byteBuffer, final int maxBlank,
diff --git a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
index 0efe74c..7763a0f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java
@@ -18,6 +18,7 @@ package org.apache.rocketmq.store;
import java.io.File;
import java.nio.ByteBuffer;
+import java.util.HashMap;
import java.util.List;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.logging.InternalLogger;
@@ -25,10 +26,11 @@ import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.logfile.MappedFile;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
import org.apache.rocketmq.store.queue.CqUnit;
import org.apache.rocketmq.store.queue.FileQueueLifeCycle;
+import org.apache.rocketmq.store.queue.QueueOffsetAssigner;
import org.apache.rocketmq.store.queue.ReferredIterator;
public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
@@ -37,7 +39,7 @@ public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
public static final int CQ_STORE_UNIT_SIZE = 20;
private static final InternalLogger LOG_ERROR = InternalLoggerFactory.getLogger(LoggerName.STORE_ERROR_LOGGER_NAME);
- private final DefaultMessageStore defaultMessageStore;
+ private final MessageStore defaultMessageStore;
private final MappedFileQueue mappedFileQueue;
private final String topic;
@@ -55,7 +57,7 @@ public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
final int queueId,
final String storePath,
final int mappedFileSize,
- final DefaultMessageStore defaultMessageStore) {
+ final MessageStore defaultMessageStore) {
this.storePath = storePath;
this.mappedFileSize = mappedFileSize;
this.defaultMessageStore = defaultMessageStore;
@@ -438,6 +440,17 @@ public class ConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCycle {
this.defaultMessageStore.getRunningFlags().makeLogicsQueueError();
}
+ @Override
+ public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageExtBrokerInner msg, short messageNum) {
+ String topicQueueKey = getTopic() + "-" + getQueueId();
+ HashMap<String, Long> topicQueueTable = queueOffsetAssigner.getTopicQueueTable();
+
+ long topicOffset = topicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+ topicQueueTable.put(topicQueueKey, topicOffset + messageNum);
+
+ msg.setQueueOffset(topicOffset);
+ }
+
private boolean putMessagePositionInfo(final long offset, final int size, final long tagsCode,
final long cqOffset) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
index d25b9eb..da47be6 100644
--- a/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
@@ -32,9 +32,9 @@ import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.CopyOnWriteArrayList;
@@ -47,13 +47,16 @@ import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.ServiceThread;
import org.apache.rocketmq.common.SystemClock;
import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.running.RunningStats;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.config.BrokerRole;
@@ -74,8 +77,6 @@ import org.apache.rocketmq.store.schedule.ScheduleMessageService;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
import org.apache.rocketmq.store.util.PerfCounter;
-import static java.lang.String.format;
-
public class DefaultMessageStore implements MessageStore {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
@@ -87,10 +88,6 @@ public class DefaultMessageStore implements MessageStore {
private final ConsumeQueueStore consumeQueueStore;
- private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
-
- protected HashMap<String/* topic-queueid */, Long/* offset */> topicQueueTable = new HashMap<>(1024);
-
private final FlushConsumeQueueService flushConsumeQueueService;
private final CleanCommitLogService cleanCommitLogService;
@@ -154,8 +151,7 @@ public class DefaultMessageStore implements MessageStore {
} else {
this.commitLog = new CommitLog(this);
}
- this.consumeQueueTable = new ConcurrentHashMap<>(32);
- this.consumeQueueStore = new ConsumeQueueStore(this, this.messageStoreConfig, this.consumeQueueTable);
+ this.consumeQueueStore = new ConsumeQueueStore(this, this.messageStoreConfig);
this.flushConsumeQueueService = new FlushConsumeQueueService();
this.cleanCommitLogService = new CleanCommitLogService();
@@ -194,13 +190,7 @@ public class DefaultMessageStore implements MessageStore {
@Override
public void truncateDirtyLogicFiles(long phyOffset) {
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
-
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.truncateDirtyLogicFiles(logic, phyOffset);
- }
- }
+ this.consumeQueueStore.truncateDirty(phyOffset);
}
/**
@@ -211,7 +201,6 @@ public class DefaultMessageStore implements MessageStore {
boolean result = true;
try {
- long start = System.currentTimeMillis();
boolean lastExitOK = !this.isTempFileExist();
log.info("last shutdown {}, root dir: {}", lastExitOK ? "normally" : "abnormally", messageStoreConfig.getStorePathRootDir());
@@ -219,7 +208,7 @@ public class DefaultMessageStore implements MessageStore {
result = result && this.commitLog.load();
// load Consume Queue
- result = result && this.loadConsumeQueue();
+ result = result && this.consumeQueueStore.load();
if (result) {
this.storeCheckpoint =
@@ -269,7 +258,7 @@ public class DefaultMessageStore implements MessageStore {
* 4. Make sure the fall-behind messages to be dispatched before starting the commitlog, especially when the broker role are automatically changed.
*/
long maxPhysicalPosInLogicQueue = commitLog.getMinOffset();
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.getConsumeQueueTable().values()) {
for (ConsumeQueueInterface logic : maps.values()) {
if (logic.getMaxPhysicOffset() > maxPhysicalPosInLogicQueue) {
maxPhysicalPosInLogicQueue = logic.getMaxPhysicOffset();
@@ -387,11 +376,7 @@ public class DefaultMessageStore implements MessageStore {
@Override
public void destroyLogics() {
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.destroy(logic);
- }
- }
+ this.consumeQueueStore.destroy();
}
private PutMessageStatus checkMessage(MessageExtBrokerInner msg) {
@@ -464,6 +449,20 @@ public class DefaultMessageStore implements MessageStore {
return CompletableFuture.completedFuture(new PutMessageResult(msgCheckStatus, null));
}
+ if (msg.getProperties().containsKey(MessageConst.PROPERTY_INNER_NUM)
+ && !MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
+ log.warn("[BUG]The message had property {} but is not an inner batch", MessageConst.PROPERTY_INNER_NUM);
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+
+ if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
+ Optional<TopicConfig> topicConfig = this.getTopicConfig(msg.getTopic());
+ if (!QueueTypeUtils.isBatchCq(topicConfig)) {
+ log.error("[BUG]The message is an inner batch but cq type is not batch cq");
+ return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
+ }
+ }
+
long beginTime = this.getSystemClock().now();
CompletableFuture<PutMessageResult> putResultFuture = this.commitLog.asyncPutMessage(msg);
@@ -649,8 +648,7 @@ public class DefaultMessageStore implements MessageStore {
break;
}
- if (this.isTheBatchFull(sizePy, maxMsgNums, maxPullSize, getResult.getBufferTotalSize(), getResult.getMessageCount(),
- isInDisk)) {
+ if (this.isTheBatchFull(sizePy, cqUnit.getBatchNum(), maxMsgNums, maxPullSize, getResult.getBufferTotalSize(), getResult.getMessageCount(), isInDisk)) {
break;
}
@@ -756,7 +754,7 @@ public class DefaultMessageStore implements MessageStore {
return logic.getMaxOffsetInQueue();
}
} else {
- Long offset = this.topicQueueTable.get(topic + "-" + queueId);
+ Long offset = this.consumeQueueStore.getMaxOffset(topic, queueId);
if (offset != null) {
return offset;
}
@@ -800,7 +798,11 @@ public class DefaultMessageStore implements MessageStore {
public long getOffsetInQueueByTime(String topic, int queueId, long timestamp) {
ConsumeQueueInterface logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
- return logic.getOffsetInQueueByTime(timestamp);
+ long resultOffset = logic.getOffsetInQueueByTime(timestamp);
+ // Make sure the result offset is in valid range.
+ resultOffset = Math.max(resultOffset, logic.getMinOffsetInQueue());
+ resultOffset = Math.min(resultOffset, logic.getMaxOffsetInQueue());
+ return resultOffset;
}
return 0;
@@ -1071,7 +1073,7 @@ public class DefaultMessageStore implements MessageStore {
@Override
public int cleanUnusedTopic(Set<String> topics) {
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
+ Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.getConsumeQueueTable().entrySet().iterator();
while (it.hasNext()) {
Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
String topic = next.getKey();
@@ -1086,7 +1088,7 @@ public class DefaultMessageStore implements MessageStore {
cq.getQueueId()
);
- this.commitLog.removeQueueFromTopicQueueTable(cq.getTopic(), cq.getQueueId());
+ this.consumeQueueStore.removeTopicQueueTable(cq.getTopic(), cq.getQueueId());
}
it.remove();
@@ -1105,45 +1107,7 @@ public class DefaultMessageStore implements MessageStore {
public void cleanExpiredConsumerQueue() {
long minCommitLogOffset = this.commitLog.getMinOffset();
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- String topic = next.getKey();
- if (!topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
- ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
- Iterator<Entry<Integer, ConsumeQueueInterface>> itQT = queueTable.entrySet().iterator();
- while (itQT.hasNext()) {
- Entry<Integer, ConsumeQueueInterface> nextQT = itQT.next();
- long maxCLOffsetInConsumeQueue = nextQT.getValue().getLastOffset();
-
- if (maxCLOffsetInConsumeQueue == -1) {
- log.warn("maybe ConsumeQueue was created just now. topic={} queueId={} maxPhysicOffset={} minLogicOffset={}.",
- nextQT.getValue().getTopic(),
- nextQT.getValue().getQueueId(),
- nextQT.getValue().getMaxPhysicOffset(),
- nextQT.getValue().getMinLogicOffset());
- } else if (maxCLOffsetInConsumeQueue < minCommitLogOffset) {
- log.info(
- "cleanExpiredConsumerQueue: {} {} consumer queue destroyed, minCommitLogOffset: {} maxCLOffsetInConsumeQueue: {}",
- topic,
- nextQT.getKey(),
- minCommitLogOffset,
- maxCLOffsetInConsumeQueue);
-
- DefaultMessageStore.this.commitLog.removeQueueFromTopicQueueTable(nextQT.getValue().getTopic(),
- nextQT.getValue().getQueueId());
-
- this.consumeQueueStore.destroy(nextQT.getValue());
- itQT.remove();
- }
- }
-
- if (queueTable.isEmpty()) {
- log.info("cleanExpiredConsumerQueue: {},topic destroyed", topic);
- it.remove();
- }
- }
- }
+ this.consumeQueueStore.cleanExpired(minCommitLogOffset);
}
public Map<String, Long> getMessageIds(final String topic, final int queueId, long minOffset, long maxOffset,
@@ -1269,13 +1233,13 @@ public class DefaultMessageStore implements MessageStore {
return (maxOffsetPy - offsetPy) > memory;
}
- private boolean isTheBatchFull(int sizePy, int maxMsgNums, long maxMsgSize, int bufferTotal, int messageTotal, boolean isInDisk) {
+ private boolean isTheBatchFull(int sizePy, int unitBatchNum, int maxMsgNums, long maxMsgSize, int bufferTotal, int messageTotal, boolean isInDisk) {
if (0 == bufferTotal || 0 == messageTotal) {
return false;
}
- if (maxMsgNums <= messageTotal) {
+ if (messageTotal + unitBatchNum > maxMsgNums) {
return true;
}
@@ -1392,16 +1356,7 @@ public class DefaultMessageStore implements MessageStore {
private void checkSelf() {
this.commitLog.checkSelf();
-
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- Iterator<Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
- while (itNext.hasNext()) {
- Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
- this.consumeQueueStore.checkSelf(cq.getValue());
- }
- }
+ this.consumeQueueStore.checkSelf();
}
private boolean isTempFileExist() {
@@ -1410,53 +1365,6 @@ public class DefaultMessageStore implements MessageStore {
return file.exists();
}
- private boolean loadConsumeQueue() {
- checkOtherConsumeQueue();
-
- File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
- File[] fileTopicList = dirLogic.listFiles();
- if (fileTopicList != null) {
-
- for (File fileTopic : fileTopicList) {
- String topic = fileTopic.getName();
-
- File[] fileQueueIdList = fileTopic.listFiles();
- if (fileQueueIdList != null) {
- for (File fileQueueId : fileQueueIdList) {
- int queueId;
- try {
- queueId = Integer.parseInt(fileQueueId.getName());
- } catch (NumberFormatException e) {
- continue;
- }
- ConsumeQueueInterface logic = new ConsumeQueue(
- topic,
- queueId,
- StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
- this.getMessageStoreConfig().getMappedFileSizeConsumeQueue(),
- this);
- this.putConsumeQueue(topic, queueId, logic);
- if (!this.consumeQueueStore.load(logic)) {
- return false;
- }
- }
- }
- }
- }
-
- log.info("load logics queue all over, OK");
-
- return true;
- }
-
- private void checkOtherConsumeQueue() {
- File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
- if (dirLogic.exists()) {
- throw new RuntimeException(format("Batch consume queue directory: [%s] exist. Can not load consume queue while batch consume queue exists.",
- StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir())));
- }
- }
-
private void recover(final boolean lastExitOK) {
long recoverCqStart = System.currentTimeMillis();
long maxPhyOffsetOfConsumeQueue = this.recoverConsumeQueue();
@@ -1485,43 +1393,13 @@ public class DefaultMessageStore implements MessageStore {
return transientStorePool;
}
- private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueueInterface consumeQueue) {
- ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface> map = this.consumeQueueTable.get(topic);
- if (null == map) {
- map = new ConcurrentHashMap<Integer/* queueId */, ConsumeQueueInterface>();
- map.put(queueId, consumeQueue);
- this.consumeQueueTable.put(topic, map);
- } else {
- map.put(queueId, consumeQueue);
- }
- }
-
private long recoverConsumeQueue() {
- long maxPhysicOffset = -1;
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.recover(logic);
- if (logic.getMaxPhysicOffset() > maxPhysicOffset) {
- maxPhysicOffset = logic.getMaxPhysicOffset();
- }
- }
- }
-
- return maxPhysicOffset;
+ return this.consumeQueueStore.recover();
}
public void recoverTopicQueueTable() {
- HashMap<String/* topic-queueid */, Long/* offset */> table = new HashMap<String, Long>(1024);
long minPhyOffset = this.commitLog.getMinOffset();
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- String key = logic.getTopic() + "-" + logic.getQueueId();
- table.put(key, logic.getMaxOffsetInQueue());
- this.consumeQueueStore.correctMinOffset(logic, minPhyOffset);
- }
- }
-
- this.topicQueueTable = table;
+ this.consumeQueueStore.recoverOffsetTable(minPhyOffset);
}
@Override
@@ -1539,7 +1417,7 @@ public class DefaultMessageStore implements MessageStore {
}
public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> getConsumeQueueTable() {
- return consumeQueueTable;
+ return consumeQueueStore.getConsumeQueueTable();
}
@Override
@@ -1569,8 +1447,7 @@ public class DefaultMessageStore implements MessageStore {
}
public void putMessagePositionInfo(DispatchRequest dispatchRequest) {
- ConsumeQueueInterface cq = this.findConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
- this.consumeQueueStore.putMessagePositionInfoWrapper(cq, dispatchRequest);
+ this.consumeQueueStore.putMessagePositionInfoWrapper(dispatchRequest);
}
@Override
@@ -1606,7 +1483,7 @@ public class DefaultMessageStore implements MessageStore {
@Override
public ConsumeQueueInterface getConsumeQueue(String topic, int queueId) {
- ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
+ ConcurrentMap<Integer, ConsumeQueueInterface> map = this.getConsumeQueueTable().get(topic);
if (map == null) {
return null;
}
@@ -1656,19 +1533,21 @@ public class DefaultMessageStore implements MessageStore {
}
@Override
- public void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
+ public void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short messageNum) {
final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
- long topicOffset = this.topicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
- msg.setQueueOffset(topicOffset);
- this.topicQueueTable.put(topicQueueKey, topicOffset + batchNum);
+ this.consumeQueueStore.assignQueueOffset(msg, messageNum);
}
}
@Override
- public void removeOffsetTable(String topicQueueKey) {
- this.topicQueueTable.remove(topicQueueKey);
+ public Optional<TopicConfig> getTopicConfig(String topic) {
+ return this.consumeQueueStore.getTopicConfig(topic);
+ }
+
+ public void setTopicConfigTable(ConcurrentMap<String, TopicConfig> topicConfigTable) {
+ this.consumeQueueStore.setTopicConfigTable(topicConfigTable);
}
class CommitLogDispatcherBuildConsumeQueue implements CommitLogDispatcher {
@@ -1975,7 +1854,7 @@ public class DefaultMessageStore implements MessageStore {
if (minOffset > this.lastPhysicalMinOffset) {
this.lastPhysicalMinOffset = minOffset;
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.getConsumeQueueTable();
for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
for (ConsumeQueueInterface logic : maps.values()) {
@@ -2021,7 +1900,7 @@ public class DefaultMessageStore implements MessageStore {
logicsMsgTimestamp = DefaultMessageStore.this.getStoreCheckpoint().getLogicsMsgTimestamp();
}
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.consumeQueueTable;
+ ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = DefaultMessageStore.this.getConsumeQueueTable();
for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
for (ConsumeQueueInterface cq : maps.values()) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java b/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
index a7d5083..2ad6ee4 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MappedFileQueue.java
@@ -171,8 +171,8 @@ public class MappedFileQueue implements Swappable {
return true;
}
- try {
- MappedFile mappedFile = new DefaultMappedFile(file.getPath(), mappedFileSize);
+ try {
+ MappedFile mappedFile = new DefaultMappedFile(file.getPath(), mappedFileSize);
mappedFile.setWrotePosition(this.mappedFileSize);
mappedFile.setFlushedPosition(this.mappedFileSize);
@@ -234,13 +234,13 @@ public class MappedFileQueue implements Swappable {
if (this.allocateMappedFileService != null) {
mappedFile = this.allocateMappedFileService.putRequestAndReturnMappedFile(nextFilePath,
nextNextFilePath, this.mappedFileSize);
- } else {
- try {
- mappedFile = new DefaultMappedFile(nextFilePath, this.mappedFileSize);
- } catch (IOException e) {
- log.error("create mappedFile exception", e);
- }
+ } else {
+ try {
+ mappedFile = new DefaultMappedFile(nextFilePath, this.mappedFileSize);
+ } catch (IOException e) {
+ log.error("create mappedFile exception", e);
}
+ }
if (mappedFile != null) {
if (this.mappedFiles.isEmpty()) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/MessageStore.java b/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
index 51d8a24..341a29f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/MessageStore.java
@@ -18,10 +18,12 @@ package org.apache.rocketmq.store;
import java.util.HashMap;
import java.util.LinkedList;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import org.apache.rocketmq.common.SystemClock;
+import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.MessageStoreConfig;
@@ -562,16 +564,19 @@ public interface MessageStore {
boolean isSyncMaster();
/**
- * assign an queue offset and increase it.
+ * Assign an queue offset and increase it.
+ * If there is a race condition, you need to lock/unlock this method yourself.
+ *
* @param topicQueueKey topic-queue key
* @param msg message
- * @param batchNum batch num
+ * @param messageNum message num
*/
- void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum);
+ void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short messageNum);
/**
- * remove offset table
- * @param topicQueueKey topic-queue key
+ * get topic config
+ * @param topic topic name
+ * @return topic config info
*/
- void removeOffsetTable(String topicQueueKey);
+ Optional<TopicConfig> getTopicConfig(String topic);
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
index d4c160d..bf8832d 100644
--- a/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
+++ b/store/src/main/java/org/apache/rocketmq/store/PutMessageContext.java
@@ -8,40 +8,41 @@
*
* http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
+
package org.apache.rocketmq.store;
public class PutMessageContext {
- private String topicQueueTableKey;
- private long[] phyPos;
- private int batchSize;
-
- public PutMessageContext(String topicQueueTableKey) {
- this.topicQueueTableKey = topicQueueTableKey;
- }
-
- public String getTopicQueueTableKey() {
- return topicQueueTableKey;
- }
-
- public long[] getPhyPos() {
- return phyPos;
- }
-
- public void setPhyPos(long[] phyPos) {
- this.phyPos = phyPos;
- }
-
- public int getBatchSize() {
- return batchSize;
- }
-
- public void setBatchSize(int batchSize) {
- this.batchSize = batchSize;
- }
- }
\ No newline at end of file
+ private String topicQueueTableKey;
+ private long[] phyPos;
+ private int batchSize;
+
+ public PutMessageContext(String topicQueueTableKey) {
+ this.topicQueueTableKey = topicQueueTableKey;
+ }
+
+ public String getTopicQueueTableKey() {
+ return topicQueueTableKey;
+ }
+
+ public long[] getPhyPos() {
+ return phyPos;
+ }
+
+ public void setPhyPos(long[] phyPos) {
+ this.phyPos = phyPos;
+ }
+
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+}
\ No newline at end of file
diff --git a/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java b/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
index 81b407e..e910c2a 100644
--- a/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
+++ b/store/src/main/java/org/apache/rocketmq/store/StoreUtil.java
@@ -44,10 +44,6 @@ public class StoreUtil {
return physicalTotal;
}
- public static boolean isStreamMode(MessageStore messageStore) {
- return messageStore instanceof StreamMessageStore;
- }
-
public static void fileAppend(MappedFile file, ByteBuffer data) {
boolean success = file.appendMessage(data);
if (!success) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java b/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java
deleted file mode 100644
index d9277ff..0000000
--- a/store/src/main/java/org/apache/rocketmq/store/StreamMessageStore.java
+++ /dev/null
@@ -1,2573 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.store;
-
-import org.apache.rocketmq.common.BrokerConfig;
-import org.apache.rocketmq.common.MixAll;
-import org.apache.rocketmq.common.ServiceThread;
-import org.apache.rocketmq.common.SystemClock;
-import org.apache.rocketmq.common.ThreadFactoryImpl;
-import org.apache.rocketmq.common.UtilAll;
-import org.apache.rocketmq.common.constant.LoggerName;
-import org.apache.rocketmq.common.message.MessageAccessor;
-import org.apache.rocketmq.common.message.MessageConst;
-import org.apache.rocketmq.common.message.MessageDecoder;
-import org.apache.rocketmq.common.message.MessageExt;
-import org.apache.rocketmq.common.running.RunningStats;
-import org.apache.rocketmq.common.sysflag.MessageSysFlag;
-import org.apache.rocketmq.common.topic.TopicValidator;
-import org.apache.rocketmq.logging.InternalLogger;
-import org.apache.rocketmq.logging.InternalLoggerFactory;
-import org.apache.rocketmq.store.config.BrokerRole;
-import org.apache.rocketmq.store.config.FlushDiskType;
-import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.apache.rocketmq.store.config.StorePathConfigHelper;
-import org.apache.rocketmq.store.ha.HAService;
-import org.apache.rocketmq.store.index.IndexService;
-import org.apache.rocketmq.store.index.QueryOffsetResult;
-import org.apache.rocketmq.store.logfile.DefaultMappedFile;
-import org.apache.rocketmq.store.logfile.MappedFile;
-import org.apache.rocketmq.store.queue.BatchConsumeQueue;
-import org.apache.rocketmq.store.queue.CQType;
-import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
-import org.apache.rocketmq.store.queue.ConsumeQueueStore;
-import org.apache.rocketmq.store.queue.CqUnit;
-import org.apache.rocketmq.store.queue.ReferredIterator;
-import org.apache.rocketmq.store.schedule.ScheduleMessageService;
-import org.apache.rocketmq.store.stats.BrokerStatsManager;
-import org.apache.rocketmq.store.util.PerfCounter;
-import org.apache.rocketmq.store.util.QueueTypeUtils;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.ByteBuffer;
-import java.nio.channels.FileLock;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Set;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.LinkedBlockingDeque;
-import java.util.concurrent.RejectedExecutionException;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ThreadFactory;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicLong;
-
-import static java.lang.String.format;
-
-public class StreamMessageStore implements MessageStore {
- private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
-
- public final PerfCounter.Ticks perfs = new PerfCounter.Ticks(log);
-
- private final MessageStoreConfig messageStoreConfig;
- // CommitLog
- private final CommitLog commitLog;
-
- private final ConsumeQueueStore consumeQueueStore;
-
- private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
-
- protected HashMap<String/* topic-queueid */, Long/* offset */> topicQueueTable = new HashMap<String, Long>(1024);
-
- protected HashMap<String/* topic-queueid */, Long/* offset */> batchTopicQueueTable = new HashMap<String, Long>(1024);
-
- private final FlushConsumeQueueService flushConsumeQueueService;
-
- private final CleanCommitLogService cleanCommitLogService;
-
- private final CleanConsumeQueueService cleanConsumeQueueService;
-
- private final CorrectLogicOffsetService correctLogicOffsetService;
-
- private final IndexService indexService;
-
- private final AllocateMappedFileService allocateMappedFileService;
-
- private final ReputMessageService reputMessageService;
-
- private final HAService haService;
-
- private final ScheduleMessageService scheduleMessageService;
-
- private final StoreStatsService storeStatsService;
-
- private final TransientStorePool transientStorePool;
-
- private final RunningFlags runningFlags = new RunningFlags();
- private final SystemClock systemClock = new SystemClock();
-
- private final ScheduledExecutorService scheduledExecutorService =
- Executors.newSingleThreadScheduledExecutor(new ThreadFactoryImpl("StoreScheduledThread"));
- private final BrokerStatsManager brokerStatsManager;
- private final MessageArrivingListener messageArrivingListener;
- private final BrokerConfig brokerConfig;
-
- private volatile boolean shutdown = true;
-
- private StoreCheckpoint storeCheckpoint;
-
- private AtomicLong printTimes = new AtomicLong(0);
-
- private final LinkedList<CommitLogDispatcher> dispatcherList;
-
- private RandomAccessFile lockFile;
-
- private FileLock lock;
-
- boolean shutDownNormal = false;
-
- //polish for reput
- private ThreadPoolExecutor[] reputExecutors;
-
- private BlockingQueue<Runnable>[] reputQueues;
-
- private boolean isDispatchFromSenderThread;
-
- private static final Future EMPTY_FUTURE = new Future() {
- @Override
- public boolean cancel(final boolean mayInterruptIfRunning) {
- return false;
- }
-
- @Override
- public boolean isCancelled() {
- return false;
- }
-
- @Override
- public boolean isDone() {
- return true;
- }
-
- @Override
- public Object get() {
- return null;
- }
-
- @Override
- public Object get(final long timeout, final TimeUnit unit) {
- return null;
- }
- };
-
- // Max pull msg size
- private final static int MAX_PULL_MSG_SIZE = 128 * 1024 * 1024;
-
- public StreamMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager,
- final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException {
- this.messageArrivingListener = messageArrivingListener;
- this.brokerConfig = brokerConfig;
- this.messageStoreConfig = messageStoreConfig;
- this.brokerStatsManager = brokerStatsManager;
- this.allocateMappedFileService = new AllocateMappedFileService(this);
- if (messageStoreConfig.isEnableDLegerCommitLog()) {
- throw new RuntimeException("dleger is not supported in this message store.");
- }
- this.isDispatchFromSenderThread = messageStoreConfig.isDispatchFromSenderThread();
- this.commitLog = new CommitLog(this);
- this.consumeQueueTable = new ConcurrentHashMap<>(32);
- this.consumeQueueStore = new ConsumeQueueStore(this, this.messageStoreConfig, this.consumeQueueTable);
-
- this.flushConsumeQueueService = new FlushConsumeQueueService();
- this.cleanCommitLogService = new CleanCommitLogService();
- this.cleanConsumeQueueService = new CleanConsumeQueueService();
- this.correctLogicOffsetService = new CorrectLogicOffsetService();
- this.storeStatsService = new StoreStatsService();
- this.indexService = new IndexService(this);
- if (!messageStoreConfig.isEnableDLegerCommitLog()) {
- this.haService = new HAService(this);
- } else {
- this.haService = null;
- }
- if (isDispatchFromSenderThread) {
- this.reputMessageService = new SyncReputMessageService();
- } else {
- this.reputMessageService = new ReputMessageService();
- }
-
- this.scheduleMessageService = new ScheduleMessageService(this);
-
- this.transientStorePool = new TransientStorePool(messageStoreConfig);
-
- if (messageStoreConfig.isTransientStorePoolEnable()) {
- this.transientStorePool.init();
- }
-
- this.allocateMappedFileService.start();
-
- this.indexService.start();
-
- this.dispatcherList = new LinkedList<>();
- this.dispatcherList.addLast(new CommitLogDispatcherBuildConsumeQueue());
- this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex());
-
- File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir()));
- DefaultMappedFile.ensureDirOK(file.getParent());
- lockFile = new RandomAccessFile(file, "rw");
- initAsyncReputThreads(messageStoreConfig.getDispatchCqThreads(), messageStoreConfig.getDispatchCqCacheNum());
- }
-
- /**
- * @throws IOException
- */
- @Override
- public boolean load() {
- boolean result = true;
-
- try {
- long start = System.currentTimeMillis();
- boolean lastExitOK = !this.isTempFileExist();
- log.info("last shutdown {}, root dir: {}", lastExitOK ? "normally" : "abnormally", messageStoreConfig.getStorePathRootDir());
-
- if (null != scheduleMessageService) {
- result = result && this.scheduleMessageService.load();
- }
-
- // load Commit Log
- result = result && this.commitLog.load();
-
- // load Batch Consume Queue
- result = result && this.loadBatchConsumeQueue();
-
- if (result) {
- this.storeCheckpoint =
- new StoreCheckpoint(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
-
- this.indexService.load(lastExitOK);
-
- this.recover(lastExitOK);
-
- log.info("load over, and the max phy offset = {} cost = {}", this.getMaxPhyOffset(), System.currentTimeMillis() - start);
- }
- } catch (Exception e) {
- log.error("load exception", e);
- result = false;
- }
-
- if (!result) {
- this.allocateMappedFileService.shutdown();
- }
-
- return result;
- }
-
- /**
- * @throws Exception
- */
- @Override
- public void start() throws Exception {
-
- lock = lockFile.getChannel().tryLock(0, 1, false);
- if (lock == null || lock.isShared() || !lock.isValid()) {
- throw new RuntimeException("Lock failed,MQ already started");
- }
-
- lockFile.getChannel().write(ByteBuffer.wrap("lock".getBytes()));
- lockFile.getChannel().force(true);
-
- if (this.getMessageStoreConfig().isDuplicationEnable()) {
- this.reputMessageService.setReputFromOffset(this.commitLog.getConfirmOffset());
- } else {
- this.reputMessageService.setReputFromOffset(this.commitLog.getMaxOffset());
- }
- this.reputMessageService.start();
-
- if (!messageStoreConfig.isEnableDLegerCommitLog()) {
- this.haService.start();
- this.handleScheduleMessageService(messageStoreConfig.getBrokerRole());
- }
-
- this.flushConsumeQueueService.start();
- this.commitLog.start();
- this.storeStatsService.start();
-
- this.createTempFile();
- this.addScheduleTask();
- this.perfs.start();
- this.shutdown = false;
- }
-
- @Override
- public void shutdown() {
- if (!this.shutdown) {
- this.shutdown = true;
-
- this.scheduledExecutorService.shutdown();
-
- try {
-
- Thread.sleep(1000 * 3);
- } catch (InterruptedException e) {
- log.error("shutdown Exception, ", e);
- }
-
- if (this.scheduleMessageService != null) {
- this.scheduleMessageService.shutdown();
- }
- if (this.haService != null) {
- this.haService.shutdown();
- }
-
- this.storeStatsService.shutdown();
- this.indexService.shutdown();
- this.commitLog.shutdown();
- this.reputMessageService.shutdown();
- this.flushConsumeQueueService.shutdown();
- this.allocateMappedFileService.shutdown();
- this.storeCheckpoint.flush();
- this.storeCheckpoint.shutdown();
-
- this.perfs.shutdown();
-
- if (this.runningFlags.isWriteable() && dispatchBehindBytes() == 0) {
- this.deleteFile(StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
- shutDownNormal = true;
- } else {
- log.warn("the store may be wrong, so shutdown abnormally, and keep abort file. writable: {}, dispatchBehindBytes: {}, abort file: {}",
- this.runningFlags.isWriteable(), dispatchBehindBytes(), StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
- }
- }
-
- this.transientStorePool.destroy();
-
- if (lockFile != null && lock != null) {
- try {
- lock.release();
- lockFile.close();
- } catch (IOException e) {
- }
- }
- }
-
- @Override
- public void destroy() {
- this.destroyLogics();
- this.commitLog.destroy();
- this.indexService.destroy();
- this.deleteFile(StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir()));
- this.deleteFile(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
- }
-
- @Override
- public void destroyLogics() {
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.destroy(logic);
- }
- }
- }
-
- @Override
- public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
- if (this.shutdown) {
- log.warn("message store has shutdown, so putMessage is forbidden");
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- }
-
- if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
- long value = this.printTimes.getAndIncrement();
- if ((value % 50000) == 0) {
- log.warn("message store is slave mode, so putMessage is forbidden ");
- }
-
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- }
-
- if (msg.getProperties().containsKey(MessageConst.PROPERTY_INNER_NUM)
- && !MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
- log.warn("[BUG]The message had property {} but is not an inner batch", MessageConst.PROPERTY_INNER_NUM);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
- CQType cqType = QueueTypeUtils.getCQType(this);
-
- if (!CQType.BatchCQ.equals(cqType)) {
- log.warn("[BUG]The message is an inner batch but cq type is not batch consume queue");
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
- }
-
- if (!this.runningFlags.isWriteable()) {
- long value = this.printTimes.getAndIncrement();
- if ((value % 50000) == 0) {
- log.warn("message store is not writeable, so putMessage is forbidden " + this.runningFlags.getFlagBits());
- }
-
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- } else {
- this.printTimes.set(0);
- }
-
- int topicLen = msg.getTopic().length();
- if (topicLen > this.messageStoreConfig.getMaxTopicLength()) {
- log.warn("putMessage message topic[{}] length too long {}", msg.getTopic(), topicLen);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (topicLen > Byte.MAX_VALUE) {
- log.warn("putMessage message topic[{}] length too long {}, but it is not supported by broker",
- msg.getTopic(), topicLen);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (msg.getPropertiesString() != null && msg.getPropertiesString().length() > Short.MAX_VALUE) {
- log.warn("putMessage message properties length too long " + msg.getPropertiesString().length());
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.PROPERTIES_SIZE_EXCEEDED, null));
- }
-
- if (this.isOSPageCacheBusy()) {
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null));
- }
-
- long beginTime = this.getSystemClock().now();
- perfs.startTick("PUT_MESSAGE_TIME_MS");
- CompletableFuture<PutMessageResult> result = this.commitLog.asyncPutMessage(msg);
- perfs.endTick("PUT_MESSAGE_TIME_MS");
-
- long eclipseTime = this.getSystemClock().now() - beginTime;
- if (eclipseTime > 500) {
- log.warn("putMessage not in lock eclipse time(ms)={}, bodyLength={}", eclipseTime, msg.getBody().length);
- }
-
- return result;
- }
-
- @Override
- public PutMessageResult putMessage(MessageExtBrokerInner msg) {
- CompletableFuture<PutMessageResult> future = asyncPutMessage(msg);
- try {
- return future.get(3, TimeUnit.SECONDS);
- } catch (Throwable t) {
- log.error("Get async put result failed", t);
- return new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null);
- }
- }
-
- @Override
- public PutMessageResult putMessages(MessageExtBatch messageExtBatch) {
- CompletableFuture<PutMessageResult> future = asyncPutMessages(messageExtBatch);
- try {
- return future.get(3, TimeUnit.SECONDS);
- } catch (Throwable t) {
- log.error("Get async put result failed", t);
- return new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null);
- }
- }
-
- @Override
- public CompletableFuture<PutMessageResult> asyncPutMessages(MessageExtBatch messageExtBatch) {
- if (this.shutdown) {
- log.warn("StreamMessageStore has shutdown, so putMessages is forbidden");
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- }
-
- if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
- long value = this.printTimes.getAndIncrement();
- if ((value % 50000) == 0) {
- log.warn("StreamMessageStore is in slave mode, so putMessages is forbidden ");
- }
-
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- }
-
- if (!this.runningFlags.isWriteable()) {
- long value = this.printTimes.getAndIncrement();
- if ((value % 50000) == 0) {
- log.warn("StreamMessageStore is not writable, so putMessages is forbidden " + this.runningFlags.getFlagBits());
- }
-
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.SERVICE_NOT_AVAILABLE, null));
- } else {
- this.printTimes.set(0);
- }
-
- int topicLen = messageExtBatch.getTopic().length();
- if (topicLen > this.messageStoreConfig.getMaxTopicLength()) {
- log.warn("putMessage batch message topic[{}] length too long {}", messageExtBatch.getTopic(), topicLen);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (topicLen > Byte.MAX_VALUE) {
- log.warn("putMessage batch message topic[{}] length too long {}, but it is not supported by broker",
- messageExtBatch.getTopic(), topicLen);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (messageExtBatch.getBody().length > messageStoreConfig.getMaxMessageSize()) {
- log.warn("PutMessages body length too long " + messageExtBatch.getBody().length);
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null));
- }
-
- if (this.isOSPageCacheBusy()) {
- return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.OS_PAGECACHE_BUSY, null));
- }
-
- long beginTime = this.getSystemClock().now();
- CompletableFuture<PutMessageResult> result = this.commitLog.asyncPutMessages(messageExtBatch);
-
- long eclipseTime = this.getSystemClock().now() - beginTime;
- if (eclipseTime > 500) {
- log.warn("not in lock eclipse time(ms)={}, bodyLength={}", eclipseTime, messageExtBatch.getBody().length);
- }
- this.storeStatsService.setPutMessageEntireTimeMax(eclipseTime);
-
- return result;
- }
-
- @Override
- public boolean isOSPageCacheBusy() {
- long begin = this.getCommitLog().getBeginTimeInLock();
- long diff = this.systemClock.now() - begin;
-
- return diff < 10000000
- && diff > this.messageStoreConfig.getOsPageCacheBusyTimeOutMills();
- }
-
- @Override
- public long lockTimeMills() {
- return this.commitLog.lockTimeMills();
- }
-
- @Override
- public SystemClock getSystemClock() {
- return systemClock;
- }
-
- @Override
- public CommitLog getCommitLog() {
- return commitLog;
- }
-
- public boolean isDispatchFromSenderThread() {
- return isDispatchFromSenderThread;
- }
-
- @Override
- public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
- final int maxMsgNums,
- final MessageFilter messageFilter) {
- return getMessage(group, topic, queueId, offset, maxMsgNums, MAX_PULL_MSG_SIZE, messageFilter);
- }
-
- @Override
- public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
- final int maxMsgNums,
- final int maxTotalMsgSize,
- final MessageFilter messageFilter) {
- if (this.shutdown) {
- log.warn("message store has shutdown, so getMessage is forbidden");
- return null;
- }
-
- if (!this.runningFlags.isReadable()) {
- log.warn("message store is not readable, so getMessage is forbidden " + this.runningFlags.getFlagBits());
- return null;
- }
-
- long beginTime = this.getSystemClock().now();
-
- GetMessageStatus status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
- long nextBeginOffset = offset;
- long minOffset = 0;
- long maxOffset = 0;
-
- GetMessageResult getResult = new GetMessageResult();
-
- final long maxOffsetPy = this.commitLog.getMaxOffset();
-
- ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
- if (consumeQueue != null) {
- minOffset = consumeQueue.getMinOffsetInQueue();
- maxOffset = consumeQueue.getMaxOffsetInQueue();
-
- if (maxOffset == 0) {
- status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
- nextBeginOffset = nextOffsetCorrection(offset, 0);
- } else if (offset < minOffset) {
- status = GetMessageStatus.OFFSET_TOO_SMALL;
- nextBeginOffset = nextOffsetCorrection(offset, minOffset);
- } else if (offset == maxOffset) {
- status = GetMessageStatus.OFFSET_OVERFLOW_ONE;
- nextBeginOffset = nextOffsetCorrection(offset, offset);
- } else if (offset > maxOffset) {
- status = GetMessageStatus.OFFSET_OVERFLOW_BADLY;
- if (0 == minOffset) {
- nextBeginOffset = nextOffsetCorrection(offset, minOffset);
- } else {
- nextBeginOffset = nextOffsetCorrection(offset, maxOffset);
- }
- } else {
- final int maxFilterMessageCount = Math.max(messageStoreConfig.getPullBatchMaxMessageCount(), maxMsgNums);
- final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
-
- long maxPullSize = Math.max(maxTotalMsgSize, 100);
- if (maxPullSize > MAX_PULL_MSG_SIZE) {
- log.warn("The max pull size is too large maxPullSize={} topic={} queueId={}", maxPullSize, topic, queueId);
- maxPullSize = MAX_PULL_MSG_SIZE;
- }
- status = GetMessageStatus.NO_MATCHED_MESSAGE;
- long maxPhyOffsetPulling = 0;
- int cqFileNum = 0;
-
- while (getResult.getBufferTotalSize() <= 0
- && nextBeginOffset < maxOffset
- && cqFileNum++ < this.messageStoreConfig.getTravelCqFileNumWhenGetMessage()) {
- ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(nextBeginOffset);
-
- if (bufferConsumeQueue == null) {
- status = GetMessageStatus.OFFSET_FOUND_NULL;
- nextBeginOffset = nextOffsetCorrection(nextBeginOffset, this.consumeQueueStore.rollNextFile(consumeQueue, nextBeginOffset));
- log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
- + maxOffset + ", but access logic queue failed. Correct nextBeginOffset to " + nextBeginOffset);
- break;
- }
-
- try {
- long nextPhyFileStartOffset = Long.MIN_VALUE;
- while (bufferConsumeQueue.hasNext()
- && nextBeginOffset < maxOffset) {
- CqUnit cqUnit = bufferConsumeQueue.next();
- long offsetPy = cqUnit.getPos();
- int sizePy = cqUnit.getSize();
-
- boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
-
- if (cqUnit.getQueueOffset() - offset > maxFilterMessageCount) {
- break;
- }
-
- if (this.isTheBatchFull(sizePy, cqUnit.getBatchNum(), maxMsgNums, maxPullSize, getResult.getBufferTotalSize(), getResult.getMessageCount(),
- isInDisk)) {
- break;
- }
-
- if (getResult.getBufferTotalSize() >= maxPullSize) {
- break;
- }
-
- maxPhyOffsetPulling = offsetPy;
-
- //Be careful, here should before the isTheBatchFull
- nextBeginOffset = cqUnit.getQueueOffset() + cqUnit.getBatchNum();
-
- if (nextPhyFileStartOffset != Long.MIN_VALUE) {
- if (offsetPy < nextPhyFileStartOffset) {
- continue;
- }
- }
-
- if (messageFilter != null
- && !messageFilter.isMatchedByConsumeQueue(cqUnit.getValidTagsCodeAsLong(), cqUnit.getCqExtUnit())) {
- if (getResult.getBufferTotalSize() == 0) {
- status = GetMessageStatus.NO_MATCHED_MESSAGE;
- }
-
- continue;
- }
-
- SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
- if (null == selectResult) {
- if (getResult.getBufferTotalSize() == 0) {
- status = GetMessageStatus.MESSAGE_WAS_REMOVING;
- }
-
- nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy);
- continue;
- }
-
- if (messageFilter != null
- && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
- if (getResult.getBufferTotalSize() == 0) {
- status = GetMessageStatus.NO_MATCHED_MESSAGE;
- }
- // release...
- selectResult.release();
- continue;
- }
-
- this.storeStatsService.getGetMessageTransferedMsgCount().add(1);
- getResult.addMessage(selectResult, cqUnit.getQueueOffset(), cqUnit.getBatchNum());
- status = GetMessageStatus.FOUND;
- nextPhyFileStartOffset = Long.MIN_VALUE;
- }
- } finally {
- bufferConsumeQueue.release();
- }
- }
-
- if (diskFallRecorded) {
- long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
- brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
- }
-
- long diff = maxOffsetPy - maxPhyOffsetPulling;
- long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE
- * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
- getResult.setSuggestPullingFromSlave(diff > memory);
- }
- } else {
- status = GetMessageStatus.NO_MATCHED_LOGIC_QUEUE;
- nextBeginOffset = nextOffsetCorrection(offset, 0);
- }
-
- if (GetMessageStatus.FOUND == status) {
- this.storeStatsService.getGetMessageTimesTotalFound().add(1);
- } else {
- this.storeStatsService.getGetMessageTimesTotalMiss().add(1);
- }
- long elapsedTime = this.getSystemClock().now() - beginTime;
- this.storeStatsService.setGetMessageEntireTimeMax(elapsedTime);
-
- getResult.setStatus(status);
- getResult.setNextBeginOffset(nextBeginOffset);
- getResult.setMaxOffset(maxOffset);
- getResult.setMinOffset(minOffset);
- return getResult;
- }
-
- @Override
- public long getMaxOffsetInQueue(String topic, int queueId) {
- ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
- if (logic != null) {
- return logic.getMaxOffsetInQueue();
- }
-
- return 0;
- }
-
- @Override
- public long getMaxOffsetInQueue(String topic, int queueId, boolean committed) {
- if (committed) {
- ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
- if (logic != null) {
- return logic.getMaxOffsetInQueue();
- }
- } else {
- Long offset = this.batchTopicQueueTable.get(topic + "-" + queueId);
- if (offset != null) {
- return offset;
- }
- }
-
- return 0;
- }
-
- @Override
- public long getMinOffsetInQueue(String topic, int queueId) {
- ConsumeQueueInterface logic = this.getConsumeQueue(topic, queueId);
- if (logic != null) {
- return logic.getMinOffsetInQueue();
- }
-
- return -1;
- }
-
- @Override
- public long getCommitLogOffsetInQueue(String topic, int queueId, long consumeQueueOffset) {
- ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
- if (consumeQueue != null) {
-
- ReferredIterator<CqUnit> bufferConsumeQueue = consumeQueue.iterateFrom(consumeQueueOffset);
- if (bufferConsumeQueue != null) {
- try {
- if (bufferConsumeQueue.hasNext()) {
- long offsetPy = bufferConsumeQueue.next().getPos();
- return offsetPy;
- }
- } finally {
- bufferConsumeQueue.release();
- }
- }
- }
-
- return 0;
- }
-
- @Override
- public long getOffsetInQueueByTime(String topic, int queueId, long timestamp) {
- ConsumeQueueInterface logic = getConsumeQueue(topic, queueId);
- if (logic != null) {
- long resultOffset = logic.getOffsetInQueueByTime(timestamp);
- // -1 means no msg found.
- if (resultOffset == -1) {
- return -1;
- }
- // Make sure the result offset should in valid range.
- resultOffset = Math.max(resultOffset, logic.getMinOffsetInQueue());
- resultOffset = Math.min(resultOffset, logic.getMaxOffsetInQueue());
- return resultOffset;
- }
-
- // logic is null means there is no message in this queue, return -1.
- return -1;
- }
-
- @Override
- public MessageExt lookMessageByOffset(long commitLogOffset) {
- SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, 4);
- if (null != sbr) {
- try {
- // 1 TOTALSIZE
- int size = sbr.getByteBuffer().getInt();
- return lookMessageByOffset(commitLogOffset, size);
- } finally {
- sbr.release();
- }
- }
-
- return null;
- }
-
- @Override
- public SelectMappedBufferResult selectOneMessageByOffset(long commitLogOffset) {
- SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, 4);
- if (null != sbr) {
- try {
- // 1 TOTALSIZE
- int size = sbr.getByteBuffer().getInt();
- return this.commitLog.getMessage(commitLogOffset, size);
- } finally {
- sbr.release();
- }
- }
-
- return null;
- }
-
- @Override
- public SelectMappedBufferResult selectOneMessageByOffset(long commitLogOffset, int msgSize) {
- return this.commitLog.getMessage(commitLogOffset, msgSize);
- }
-
- @Override
- public String getRunningDataInfo() {
- return this.storeStatsService.toString();
- }
-
- private String getStorePathPhysic() {
- String storePathPhysic = StreamMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
- return storePathPhysic;
- }
-
- @Override
- public HashMap<String, String> getRuntimeInfo() {
- HashMap<String, String> result = this.storeStatsService.getRuntimeInfo();
-
- {
- String storePathPhysic = this.getMessageStoreConfig().getStorePathCommitLog();
- double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
- result.put(RunningStats.commitLogDiskRatio.name(), String.valueOf(physicRatio));
-
- }
-
- {
-
- String storePathLogics = StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir());
- double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
- result.put(RunningStats.consumeQueueDiskRatio.name(), String.valueOf(logicsRatio));
- }
-
- {
- if (this.scheduleMessageService != null) {
- this.scheduleMessageService.buildRunningStats(result);
- }
- }
-
- result.put(RunningStats.commitLogMinOffset.name(), String.valueOf(this.getMinPhyOffset()));
- result.put(RunningStats.commitLogMaxOffset.name(), String.valueOf(this.getMaxPhyOffset()));
-
- return result;
- }
-
- @Override
- public long getMaxPhyOffset() {
- return this.commitLog.getMaxOffset();
- }
-
- @Override
- public long getMinPhyOffset() {
- return this.commitLog.getMinOffset();
- }
-
- @Override
- public long getEarliestMessageTime(String topic, int queueId) {
- ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
- if (logicQueue != null) {
- return getStoreTime(logicQueue.getEarliestUnit());
- }
-
- return -1;
- }
-
- private long getStoreTime(CqUnit result) {
- if (result != null) {
- try {
- final long phyOffset = result.getPos();
- final int size = result.getSize();
- long storeTime = this.getCommitLog().pickupStoreTimestamp(phyOffset, size);
- return storeTime;
- } catch (Exception e) {
- }
- }
- return -1;
- }
-
- @Override
- public long getEarliestMessageTime() {
- final long minPhyOffset = this.getMinPhyOffset();
- final int size = this.messageStoreConfig.getMaxMessageSize() * 2;
- return this.getCommitLog().pickupStoreTimestamp(minPhyOffset, size);
- }
-
- @Override
- public long getMessageStoreTimeStamp(String topic, int queueId, long consumeQueueOffset) {
- ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
- if (logicQueue != null) {
- return getStoreTime(logicQueue.get(consumeQueueOffset));
- }
-
- return -1;
- }
-
- @Override
- public long getMessageTotalInQueue(String topic, int queueId) {
- ConsumeQueueInterface logicQueue = this.getConsumeQueue(topic, queueId);
- if (logicQueue != null) {
- return logicQueue.getMessageTotalInQueue();
- }
-
- return -1;
- }
-
- @Override
- public SelectMappedBufferResult getCommitLogData(final long offset) {
- if (this.shutdown) {
- log.warn("message store has shutdown, so getPhyQueueData is forbidden");
- return null;
- }
-
- return this.commitLog.getData(offset);
- }
-
- @Override
- public boolean appendToCommitLog(long startOffset, byte[] data, int dataStart, int dataLength) {
- if (this.shutdown) {
- log.warn("message store has shutdown, so appendToPhyQueue is forbidden");
- return false;
- }
-
- boolean result = this.commitLog.appendData(startOffset, data, dataStart, dataLength);
- if (result) {
- this.reputMessageService.wakeup();
- } else {
- log.error("appendToPhyQueue failed " + startOffset + " " + data.length);
- }
-
- return result;
- }
-
- @Override
- public void executeDeleteFilesManually() {
- this.cleanCommitLogService.excuteDeleteFilesManualy();
- }
-
- @Override
- public QueryMessageResult queryMessage(String topic, String key, int maxNum, long begin, long end) {
- QueryMessageResult queryMessageResult = new QueryMessageResult();
-
- long lastQueryMsgTime = end;
-
- for (int i = 0; i < 3; i++) {
- QueryOffsetResult queryOffsetResult = this.indexService.queryOffset(topic, key, maxNum, begin, lastQueryMsgTime);
- if (queryOffsetResult.getPhyOffsets().isEmpty()) {
- break;
- }
-
- Collections.sort(queryOffsetResult.getPhyOffsets());
-
- queryMessageResult.setIndexLastUpdatePhyoffset(queryOffsetResult.getIndexLastUpdatePhyoffset());
- queryMessageResult.setIndexLastUpdateTimestamp(queryOffsetResult.getIndexLastUpdateTimestamp());
-
- for (int m = 0; m < queryOffsetResult.getPhyOffsets().size(); m++) {
- long offset = queryOffsetResult.getPhyOffsets().get(m);
-
- try {
-
- boolean match = true;
- MessageExt msg = this.lookMessageByOffset(offset);
- if (0 == m) {
- lastQueryMsgTime = msg.getStoreTimestamp();
- }
-
-// String[] keyArray = msg.getKeys().split(MessageConst.KEY_SEPARATOR);
-// if (topic.equals(msg.getTopic())) {
-// for (String k : keyArray) {
-// if (k.equals(key)) {
-// match = true;
-// break;
-// }
-// }
-// }
-
- if (match) {
- SelectMappedBufferResult result = this.commitLog.getData(offset, false);
- if (result != null) {
- int size = result.getByteBuffer().getInt(0);
- result.getByteBuffer().limit(size);
- result.setSize(size);
- queryMessageResult.addMessage(result);
- }
- } else {
- log.warn("queryMessage hash duplicate, {} {}", topic, key);
- }
- } catch (Exception e) {
- log.error("queryMessage exception", e);
- }
- }
-
- if (queryMessageResult.getBufferTotalSize() > 0) {
- break;
- }
-
- if (lastQueryMsgTime < begin) {
- break;
- }
- }
-
- return queryMessageResult;
- }
-
- @Override
- public void updateHaMasterAddress(String newAddr) {
- this.haService.updateMasterAddress(newAddr);
- }
-
- @Override
- public long slaveFallBehindMuch() {
- return this.commitLog.getMaxOffset() - this.haService.getPush2SlaveMaxOffset().get();
- }
-
- @Override
- public long now() {
- return this.systemClock.now();
- }
-
- @Override
- public int cleanUnusedTopic(Set<String> topics) {
- Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- String topic = next.getKey();
-
- if (!topics.contains(topic) && !topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
- ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
- for (ConsumeQueueInterface cq : queueTable.values()) {
- this.consumeQueueStore.destroy(cq);
- log.info("cleanUnusedTopic: {} {} ConsumeQueue cleaned",
- cq.getTopic(),
- cq.getQueueId()
- );
-
- this.commitLog.removeQueueFromTopicQueueTable(cq.getTopic(), cq.getQueueId());
- }
- it.remove();
- log.info("cleanUnusedTopic: {},topic consumeQueue destroyed", topic);
- }
- }
- return 0;
- }
-
- @Override
- public void cleanExpiredConsumerQueue() {
- long minCommitLogOffset = this.commitLog.getMinOffset();
-
- Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- String topic = next.getKey();
- if (!topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
- ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
- Iterator<Map.Entry<Integer, ConsumeQueueInterface>> itQT = queueTable.entrySet().iterator();
- while (itQT.hasNext()) {
- Map.Entry<Integer, ConsumeQueueInterface> nextQT = itQT.next();
- long maxCLOffsetInConsumeQueue = nextQT.getValue().getMaxPhysicOffset();
-
- if (maxCLOffsetInConsumeQueue == -1) {
- log.warn("maybe ConsumeQueue was created just now. topic={} queueId={} maxPhysicOffset={} minLogicOffset={}.",
- nextQT.getValue().getTopic(),
- nextQT.getValue().getQueueId(),
- nextQT.getValue().getMaxPhysicOffset(),
- nextQT.getValue().getMinLogicOffset());
- } else if (maxCLOffsetInConsumeQueue < minCommitLogOffset) {
- log.info(
- "cleanExpiredConsumerQueue: {} {} consumer queue destroyed, minCommitLogOffset: {} maxCLOffsetInConsumeQueue: {}",
- topic,
- nextQT.getKey(),
- minCommitLogOffset,
- maxCLOffsetInConsumeQueue);
-
- this.commitLog.removeQueueFromTopicQueueTable(nextQT.getValue().getTopic(),
- nextQT.getValue().getQueueId());
-
- this.consumeQueueStore.destroy(nextQT.getValue());
- itQT.remove();
- }
- }
-
- if (queueTable.isEmpty()) {
- log.info("cleanExpiredConsumerQueue: {},topic destroyed", topic);
- it.remove();
- }
- }
- }
- }
-
- public double getDiskSpaceWarningLevelRatio() {
- return cleanCommitLogService.getDiskSpaceWarningLevelRatio();
- }
-
- @Override
- public boolean checkInDiskByConsumeOffset(final String topic, final int queueId, long consumeOffset) {
-
- final long maxOffsetPy = this.commitLog.getMaxOffset();
-
- ConsumeQueueInterface consumeQueue = getConsumeQueue(topic, queueId);
- if (consumeQueue != null) {
- CqUnit cqUnit = consumeQueue.get(consumeOffset);
-
- if (cqUnit != null) {
- long offsetPy = cqUnit.getPos();
- return checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
- } else {
- return false;
- }
- }
- return false;
- }
-
- @Override
- public long dispatchBehindBytes() {
- return this.reputMessageService.behind();
- }
-
- @Override
- public long flush() {
- return this.commitLog.flush();
- }
-
- @Override
- public boolean resetWriteOffset(long phyOffset) {
- return this.commitLog.resetOffset(phyOffset);
- }
-
- @Override
- public long getConfirmOffset() {
- return this.commitLog.getConfirmOffset();
- }
-
- @Override
- public void setConfirmOffset(long phyOffset) {
- this.commitLog.setConfirmOffset(phyOffset);
- }
-
- @Override
- public MessageExt lookMessageByOffset(long commitLogOffset, int size) {
- SelectMappedBufferResult sbr = this.commitLog.getMessage(commitLogOffset, size);
- if (null != sbr) {
- try {
- return MessageDecoder.decode(sbr.getByteBuffer(), true, false);
- } finally {
- sbr.release();
- }
- }
-
- return null;
- }
-
- private long nextOffsetCorrection(long oldOffset, long newOffset) {
- long nextOffset = oldOffset;
- if (this.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE || this.getMessageStoreConfig().isOffsetCheckInSlave()) {
- nextOffset = newOffset;
- }
- return nextOffset;
- }
-
- private boolean checkInDiskByCommitOffset(long offsetPy, long maxOffsetPy) {
- long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
- return (maxOffsetPy - offsetPy) > memory;
- }
-
- private boolean isTheBatchFull(int sizePy, int unitBatchNum, int maxMsgNums, long maxMsgSize, int bufferTotal,
- int messageTotal, boolean isInDisk) {
-
- //At least has one message(batch)
- if (0 == bufferTotal || 0 == messageTotal) {
- return false;
- }
-
- if (messageTotal + unitBatchNum > maxMsgNums) {
- return true;
- }
-
- if (bufferTotal + sizePy > maxMsgSize) {
- return true;
- }
-
- if (isInDisk) {
- if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInDisk()) {
- return true;
- }
- if (messageTotal + unitBatchNum > this.messageStoreConfig.getMaxTransferCountOnMessageInDisk()) {
- return true;
- }
- } else {
- if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInMemory()) {
- return true;
- }
-
- if (messageTotal + unitBatchNum > this.messageStoreConfig.getMaxTransferCountOnMessageInMemory()) {
- return true;
- }
- }
-
- return false;
- }
-
- private void deleteFile(final String fileName) {
- File file = new File(fileName);
- boolean result = file.delete();
- log.info(fileName + (result ? " delete OK" : " delete Failed"));
- }
-
- /**
- * @throws IOException
- */
- private void createTempFile() throws IOException {
- String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
- File file = new File(fileName);
- DefaultMappedFile.ensureDirOK(file.getParent());
- boolean result = file.createNewFile();
- log.info(fileName + (result ? " create OK" : " already exists"));
- }
-
- private void addScheduleTask() {
-
- this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- @Override
- public void run() {
- StreamMessageStore.this.cleanFilesPeriodically();
- }
- }, 1000 * 60, this.messageStoreConfig.getCleanResourceInterval(), TimeUnit.MILLISECONDS);
-
- this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- @Override
- public void run() {
- StreamMessageStore.this.checkSelf();
- }
- }, 1, 10, TimeUnit.MINUTES);
-
- this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- @Override
- public void run() {
- try {
- if (!getMessageStoreConfig().isMappedFileSwapEnable()) {
- log.warn("Swap is not enabled.");
- return ;
- }
- StreamMessageStore.this.commitLog.swapMap(getMessageStoreConfig().getCommitLogSwapMapReserveFileNum(),
- getMessageStoreConfig().getCommitLogForceSwapMapInterval(), getMessageStoreConfig().getCommitLogSwapMapInterval());
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : StreamMessageStore.this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- StreamMessageStore.this.consumeQueueStore.swapMap(logic, getMessageStoreConfig().getLogicQueueSwapMapReserveFileNum(),
- getMessageStoreConfig().getLogicQueueForceSwapMapInterval(), getMessageStoreConfig().getLogicQueueSwapMapInterval());
- }
- }
- } catch (Exception e) {
- log.error("swap map exception", e);
- }
- }
- }, 1, 5, TimeUnit.MINUTES);
-
- this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- @Override
- public void run() {
- try {
- StreamMessageStore.this.commitLog.cleanSwappedMap(getMessageStoreConfig().getCleanSwapedMapInterval());
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : StreamMessageStore.this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- StreamMessageStore.this.consumeQueueStore.cleanSwappedMap(logic, getMessageStoreConfig().getCleanSwapedMapInterval());
- }
- }
- } catch (Exception e) {
- log.error("clean swap map exception", e);
- }
- }
- }, 1, 5, TimeUnit.MINUTES);
-
- this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- @Override
- public void run() {
- if (StreamMessageStore.this.getMessageStoreConfig().isDebugLockEnable()) {
- try {
- if (StreamMessageStore.this.commitLog.getBeginTimeInLock() != 0) {
- long lockTime = System.currentTimeMillis() - StreamMessageStore.this.commitLog.getBeginTimeInLock();
- if (lockTime > 1000 && lockTime < 10000000) {
-
- String stack = UtilAll.jstack();
- final String fileName = System.getProperty("user.home") + File.separator + "debug/lock/stack-"
- + StreamMessageStore.this.commitLog.getBeginTimeInLock() + "-" + lockTime;
- MixAll.string2FileNotSafe(stack, fileName);
- }
- }
- } catch (Exception e) {
- }
- }
- }
- }, 1, 1, TimeUnit.SECONDS);
-
- // this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
- // @Override
- // public void run() {
- // StreamMessageStore.this.cleanExpiredConsumerQueue();
- // }
- // }, 1, 1, TimeUnit.HOURS);
- }
-
- private void cleanFilesPeriodically() {
- this.cleanCommitLogService.run();
- this.cleanConsumeQueueService.run();
- this.correctLogicOffsetService.run();
- }
-
- private void checkSelf() {
- this.commitLog.checkSelf();
-
- Iterator<Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
- while (it.hasNext()) {
- Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
- Iterator<Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
- while (itNext.hasNext()) {
- Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
- this.consumeQueueStore.checkSelf(cq.getValue());
- }
- }
- }
-
- private boolean isTempFileExist() {
- String fileName = StorePathConfigHelper.getAbortFile(this.messageStoreConfig.getStorePathRootDir());
- File file = new File(fileName);
- return file.exists();
- }
-
- protected boolean loadBatchConsumeQueue() {
- checkOtherConsumeQueue();
-
- File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
- File[] fileTopicList = dirLogic.listFiles();
- if (fileTopicList != null) {
-
- for (File fileTopic : fileTopicList) {
- String topic = fileTopic.getName();
-
- File[] fileQueueIdList = fileTopic.listFiles();
- if (fileQueueIdList != null) {
- for (File fileQueueId : fileQueueIdList) {
- int queueId;
- try {
- queueId = Integer.parseInt(fileQueueId.getName());
- } catch (NumberFormatException e) {
- continue;
- }
- ConsumeQueueInterface logic = new BatchConsumeQueue(
- topic,
- queueId,
- StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
- this.getMessageStoreConfig().getMapperFileSizeBatchConsumeQueue(),
- this);
- this.putConsumeQueue(topic, queueId, logic);
- if (!this.consumeQueueStore.load(logic)) {
- return false;
- }
- }
- }
- }
- }
-
- log.info("load logics queue all over, OK");
-
- return true;
- }
-
- private void checkOtherConsumeQueue() {
- File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
- if (dirLogic.exists()) {
- throw new RuntimeException(format("Consume queue directory: [%s] exist. Can not load batch consume queue while consume queue exists.",
- StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir())));
- }
- }
-
- private void recover(final boolean lastExitOK) {
- long recoverCqStart = System.currentTimeMillis();
- long maxPhyOffsetOfConsumeQueue = this.recoverConsumeQueue();
- long recoverCqEnd = System.currentTimeMillis();
-
- if (lastExitOK) {
- this.commitLog.recoverNormally(maxPhyOffsetOfConsumeQueue);
- } else {
- this.commitLog.recoverAbnormally(maxPhyOffsetOfConsumeQueue);
- }
- long recoverClogEnd = System.currentTimeMillis();
- this.recoverTopicQueueTable();
- long recoverOffsetEnd = System.currentTimeMillis();
-
- log.info("Recover end total:{} recoverCq:{} recoverClog:{} recoverOffset:{}",
- recoverOffsetEnd - recoverCqStart, recoverCqEnd - recoverCqStart, recoverClogEnd - recoverCqEnd, recoverOffsetEnd - recoverClogEnd);
- }
-
- @Override
- public MessageStoreConfig getMessageStoreConfig() {
- return messageStoreConfig;
- }
-
- @Override
- public TransientStorePool getTransientStorePool() {
- return transientStorePool;
- }
-
- private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueueInterface consumeQueue) {
- ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface> map = this.consumeQueueTable.get(topic);
- if (null == map) {
- map = new ConcurrentHashMap<Integer/* queueId */, ConsumeQueueInterface>();
- map.put(queueId, consumeQueue);
- this.consumeQueueTable.put(topic, map);
- } else {
- map.put(queueId, consumeQueue);
- }
- }
-
- private long recoverConsumeQueue() {
- long maxPhysicOffset = -1;
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.recover(logic);
- if (logic.getMaxPhysicOffset() > maxPhysicOffset) {
- maxPhysicOffset = logic.getMaxPhysicOffset();
- }
- }
- }
-
- return maxPhysicOffset;
- }
-
- public void recoverTopicQueueTable() {
- HashMap<String/* topic-queueid */, Long/* offset */> table = new HashMap<String, Long>(1024);
- long minPhyOffset = this.commitLog.getMinOffset();
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- String key = logic.getTopic() + "-" + logic.getQueueId();
- table.put(key, logic.getMaxOffsetInQueue());
- this.consumeQueueStore.correctMinOffset(logic, minPhyOffset);
- }
- }
-
- this.batchTopicQueueTable = table;
- }
-
- @Override
- public AllocateMappedFileService getAllocateMappedFileService() {
- return allocateMappedFileService;
- }
-
- @Override
- public void truncateDirtyLogicFiles(long phyOffset) {
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
-
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- this.consumeQueueStore.truncateDirtyLogicFiles(logic, phyOffset);
- }
- }
- }
-
- @Override
- public StoreStatsService getStoreStatsService() {
- return storeStatsService;
- }
-
- public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> getConsumeQueueTable() {
- return consumeQueueTable;
- }
-
- @Override
- public StoreCheckpoint getStoreCheckpoint() {
- return storeCheckpoint;
- }
-
- @Override
- public HAService getHaService() {
- return haService;
- }
-
- @Override
- public void registerCleanFileHook(CleanFilesHook logicalQueueCleanHook) {
-
- }
-
- @Override
- public ScheduleMessageService getScheduleMessageService() {
- return scheduleMessageService;
- }
-
- @Override
- public RunningFlags getRunningFlags() {
- return runningFlags;
- }
-
- public void initAsyncReputThreads(int tsNum, int cacheNum) {
- if (tsNum <= 0) {
- tsNum = 1;
- }
- if (cacheNum < 512) {
- cacheNum = 512;
- }
- reputExecutors = new ThreadPoolExecutor[tsNum];
- reputQueues = new BlockingQueue[tsNum];
-
- for (int i = 0; i < tsNum; i++) {
- final int tmp = i;
- reputQueues[i] = new LinkedBlockingDeque<>(cacheNum);
- //Each executor can only have one thread, otherwise the cq index will get wrong
- reputExecutors[i] = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
- reputQueues[i],
- new ThreadFactory() {
- @Override
- public Thread newThread(Runnable r) {
- return new Thread(r, "MQDispatchThread-" + tmp);
- }
- });
- }
- for (ThreadPoolExecutor executorService: reputExecutors) {
- if (executorService.getMaximumPoolSize() != 1 ||
- executorService.getCorePoolSize() != 1) {
- throw new RuntimeException("The MQDispatchThreadPoll can only have one thread");
- }
- }
-
- }
-
- public Future doDispatch(final DispatchRequest request) {
- return doDispatch(request, false);
- }
-
- public Future doDispatch(final DispatchRequest request, boolean async) {
- Runnable task = new Runnable() {
- @Override
- public void run() {
- for (CommitLogDispatcher dispatcher : StreamMessageStore.this.dispatcherList) {
- dispatcher.dispatch(request);
- }
- }
- };
- if (!async) {
- task.run();
- return EMPTY_FUTURE;
- }
- int hash = Math.abs((request.getTopic() + request.getQueueId()).hashCode());
- int slot = hash % reputExecutors.length;
- try {
- return reputExecutors[slot].submit(task);
- } catch (RejectedExecutionException ignored) {
- int tryNum = 0;
- while (tryNum++ < Integer.MAX_VALUE) {
- try {
- Thread.sleep(1);
- } catch (Throwable ignored2) {
-
- }
- try {
- return reputExecutors[slot].submit(task);
- } catch (RejectedExecutionException e) {
- log.warn("DispatchReject topic:{} queue:{} pyOffset:{} tryNum:{}", request.getTopic(), request.getQueueId(), request.getCommitLogOffset(), tryNum, e);
- }
- }
- }
- return EMPTY_FUTURE;
- }
-
- public void syncProcessDispatchRequest(DispatchRequest request, boolean isRecover) throws InterruptedException {
- if (!isDispatchFromSenderThread) {
- log.error("addDispatchRequestQueue operation not supported while isCreateDispatchRequestAsync is true");
- } else {
- if (isRecover) {
- ((SyncReputMessageService) this.reputMessageService).processDispatchRequestForRecover(request);
- } else {
- ((SyncReputMessageService) this.reputMessageService).processDispatchRequest(request);
- }
- }
- }
-
- public boolean dispatched(long physicalOffset) {
- return reputMessageService.dispatched(physicalOffset);
- }
-
- public void putMessagePositionInfo(DispatchRequest dispatchRequest) {
- ConsumeQueueInterface cq = this.findOrCreateConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
- this.consumeQueueStore.putMessagePositionInfoWrapper(cq, dispatchRequest);
- }
-
- private ConsumeQueueInterface findOrCreateConsumeQueue(String topic, int queueId) {
- return this.consumeQueueStore.findOrCreateConsumeQueue(topic, queueId);
- }
-
- @Override
- public BrokerStatsManager getBrokerStatsManager() {
- return brokerStatsManager;
- }
-
- @Override
- public void handleScheduleMessageService(final BrokerRole brokerRole) {
- if (this.scheduleMessageService != null) {
- if (brokerRole == BrokerRole.SLAVE) {
- this.scheduleMessageService.shutdown();
- } else {
- this.scheduleMessageService.start();
- }
- }
-
- }
-
- public int remainTransientStoreBufferNumbs() {
- return this.transientStorePool.availableBufferNums();
- }
-
- @Override
- public boolean isTransientStorePoolDeficient() {
- return remainTransientStoreBufferNumbs() == 0;
- }
-
- @Override
- public LinkedList<CommitLogDispatcher> getDispatcherList() {
- return this.dispatcherList;
- }
-
- @Override
- public ConsumeQueueInterface getConsumeQueue(String topic, int queueId) {
- ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
- if (map == null) {
- return null;
- }
- return map.get(queueId);
- }
-
- @Override
- public void unlockMappedFile(final MappedFile mappedFile) {
- this.scheduledExecutorService.schedule(new Runnable() {
- @Override
- public void run() {
- mappedFile.munlock();
- }
- }, 6, TimeUnit.SECONDS);
- }
-
- @Override
- public PerfCounter.Ticks getPerfCounter() {
- return perfs;
- }
-
- @Override
- public ConsumeQueueStore getQueueStore() {
- return consumeQueueStore;
- }
-
- @Override
- public boolean isSyncDiskFlush() {
- return FlushDiskType.SYNC_FLUSH == this.getMessageStoreConfig().getFlushDiskType();
- }
-
- @Override
- public boolean isSyncMaster() {
- return BrokerRole.SYNC_MASTER == this.getMessageStoreConfig().getBrokerRole();
- }
-
- @Override
- public void assignOffset(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
- final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
-
- if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
- assignOffsetForCq(topicQueueKey, msg);
- assignOffsetForBcq(topicQueueKey, msg, batchNum);
- }
- }
-
- private void assignOffsetForCq(String topicQueueKey, MessageExtBrokerInner msg) {
- // not supported yet
- }
-
- private void assignOffsetForBcq(String topicQueueKey, MessageExtBrokerInner msg, short batchNum) {
- Long batchTopicOffset = this.batchTopicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
- CQType cqType = QueueTypeUtils.getCQType(this);
- if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG) || CQType.BatchCQ.equals(cqType)) {
- MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_BASE, String.valueOf(batchTopicOffset));
- msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
- }
- msg.setQueueOffset(batchTopicOffset);
- this.batchTopicQueueTable.put(topicQueueKey, batchTopicOffset + batchNum);
- }
-
- @Override
- public void removeOffsetTable(String topicQueueKey) {
- this.topicQueueTable.remove(topicQueueKey);
- this.batchTopicQueueTable.remove(topicQueueKey);
- }
-
- @Override
- public void onCommitLogAppend(MessageExtBrokerInner msg, AppendMessageResult result, MappedFile commitLogFile) {
- DispatchRequest dispatchRequest;
- switch (result.getStatus()) {
- case PUT_OK:
- dispatchRequest = constructDispatchRequest(msg, result);
- onCommitLogDispatch(dispatchRequest, this.isDispatchFromSenderThread(), commitLogFile, false, false);
- break;
- case END_OF_FILE:
- dispatchRequest = new DispatchRequest(0, true);
- onCommitLogDispatch(dispatchRequest, this.isDispatchFromSenderThread(), commitLogFile, false, true);
- break;
- default:
- throw new RuntimeException("");
- }
- }
-
- @Override
- public void onCommitLogDispatch(DispatchRequest dispatchRequest, boolean doDispatch, MappedFile commitLogFile, boolean isRecover, boolean isFileEnd) {
- if (isFileEnd) {
- if (doDispatch) {
- long nextReputFromOffset = this.getCommitLog().rollNextFile(commitLogFile.getFileFromOffset());
- dispatchRequest.setNextReputFromOffset(nextReputFromOffset);
- syncDispatch(dispatchRequest, isRecover);
- }
- } else {
- if (doDispatch) {
- dispatchRequest.setNextReputFromOffset(dispatchRequest.getCommitLogOffset() + dispatchRequest.getMsgSize());
- syncDispatch(dispatchRequest, isRecover);
- }
- }
- }
-
- private DispatchRequest constructDispatchRequest(MessageExtBrokerInner msg, AppendMessageResult appendResult) {
- long tagsCode = 0;
- String keys = "";
- String uniqKey = null;
- int sysFlag = msg.getSysFlag();
- String topic = msg.getTopic();
- long storeTimestamp = msg.getStoreTimestamp();
- int queueId = msg.getQueueId();
- long preparedTransactionOffset = msg.getPreparedTransactionOffset();
- Map<String, String> propertiesMap = msg.getProperties();
-
- if (msg.getProperties() != null && msg.getProperties().size() > 0) {
-
- keys = propertiesMap.get(MessageConst.PROPERTY_KEYS);
-
- uniqKey = propertiesMap.get(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX);
-
- String tags = propertiesMap.get(MessageConst.PROPERTY_TAGS);
- if (tags != null && tags.length() > 0) {
- tagsCode = MessageExtBrokerInner.tagsString2tagsCode(MessageExt.parseTopicFilterType(sysFlag), tags);
- }
-
- // Timing message processing
- {
- String t = propertiesMap.get(MessageConst.PROPERTY_DELAY_TIME_LEVEL);
- if (TopicValidator.RMQ_SYS_SCHEDULE_TOPIC.equals(topic) && t != null) {
- int delayLevel = Integer.parseInt(t);
-
- if (delayLevel > this.getScheduleMessageService().getMaxDelayLevel()) {
- delayLevel = this.getScheduleMessageService().getMaxDelayLevel();
- }
-
- if (delayLevel > 0) {
- tagsCode = this.getScheduleMessageService().computeDeliverTimestamp(delayLevel,
- storeTimestamp);
- }
- }
- }
- }
-
- DispatchRequest dispatchRequest = new DispatchRequest(
- topic,
- queueId,
- appendResult.getWroteOffset(),
- appendResult.getWroteBytes(),
- tagsCode,
- storeTimestamp,
- appendResult.getLogicsOffset(),
- keys,
- uniqKey,
- sysFlag,
- preparedTransactionOffset,
- propertiesMap
- );
-
- if (null != propertiesMap && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_NUM) && propertiesMap.containsKey(MessageConst.PROPERTY_INNER_BASE)) {
- dispatchRequest.setMsgBaseOffset(Long.parseLong(propertiesMap.get(MessageConst.PROPERTY_INNER_BASE)));
- dispatchRequest.setBatchSize(Short.parseShort(propertiesMap.get(MessageConst.PROPERTY_INNER_NUM)));
- }
- return dispatchRequest;
- }
-
- private void syncDispatch(DispatchRequest dispatchRequest, boolean isRecover) {
- try {
- this.syncProcessDispatchRequest(dispatchRequest, isRecover);
- } catch (InterruptedException e) {
- log.error("OnCommitlogAppend sync dispatch failed, addDispatchRequestQueue interrupted. DispatchRequest:{}", dispatchRequest);
- }
- }
-
- class CommitLogDispatcherBuildConsumeQueue implements CommitLogDispatcher {
-
- @Override
- public void dispatch(DispatchRequest dispatchRequest) {
- final int tranType = MessageSysFlag.getTransactionValue(dispatchRequest.getSysFlag());
- switch (tranType) {
- case MessageSysFlag.TRANSACTION_NOT_TYPE:
- case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
- StreamMessageStore.this.putMessagePositionInfo(dispatchRequest);
- if (BrokerRole.SLAVE != StreamMessageStore.this.getMessageStoreConfig().getBrokerRole()
- && StreamMessageStore.this.brokerConfig.isLongPollingEnable()) {
- StreamMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
- dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
- dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
- dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
- }
- if (StreamMessageStore.this.getMessageStoreConfig().getBrokerRole() == BrokerRole.SLAVE) {
- StreamMessageStore.this.storeStatsService
- .getSinglePutMessageTopicTimesTotal(dispatchRequest.getTopic()).add(1);
- StreamMessageStore.this.storeStatsService
- .getSinglePutMessageTopicSizeTotal(dispatchRequest.getTopic())
- .add(dispatchRequest.getMsgSize());
- }
- break;
- case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
- case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
- break;
- }
- }
- }
-
- class CommitLogDispatcherBuildIndex implements CommitLogDispatcher {
-
- @Override
- public void dispatch(DispatchRequest request) {
- if (StreamMessageStore.this.messageStoreConfig.isMessageIndexEnable()) {
- StreamMessageStore.this.indexService.buildIndex(request);
- }
- }
- }
-
- class CleanCommitLogService {
-
- private final static int MAX_MANUAL_DELETE_FILE_TIMES = 20;
- private final String diskSpaceWarningLevelRatio =
- System.getProperty("rocketmq.broker.diskSpaceWarningLevelRatio", "");
-
- private final String diskSpaceCleanForciblyRatio =
- System.getProperty("rocketmq.broker.diskSpaceCleanForciblyRatio", "");
- private long lastRedeleteTimestamp = 0;
-
- private volatile int manualDeleteFileSeveralTimes = 0;
-
- private volatile boolean cleanImmediately = false;
-
- double getDiskSpaceWarningLevelRatio() {
- double finalDiskSpaceWarningLevelRatio;
- if ("".equals(diskSpaceWarningLevelRatio)) {
- finalDiskSpaceWarningLevelRatio = StreamMessageStore.this.getMessageStoreConfig().getDiskSpaceWarningLevelRatio() / 100.0;
- } else {
- finalDiskSpaceWarningLevelRatio = Double.parseDouble(diskSpaceWarningLevelRatio);
- }
-
- if (finalDiskSpaceWarningLevelRatio > 0.90) {
- finalDiskSpaceWarningLevelRatio = 0.90;
- }
- if (finalDiskSpaceWarningLevelRatio < 0.35) {
- finalDiskSpaceWarningLevelRatio = 0.35;
- }
-
- return finalDiskSpaceWarningLevelRatio;
- }
-
- double getDiskSpaceCleanForciblyRatio() {
- double finalDiskSpaceCleanForciblyRatio;
- if ("".equals(diskSpaceCleanForciblyRatio)) {
- finalDiskSpaceCleanForciblyRatio = StreamMessageStore.this.getMessageStoreConfig().getDiskSpaceCleanForciblyRatio() / 100.0;
- } else {
- finalDiskSpaceCleanForciblyRatio = Double.parseDouble(diskSpaceCleanForciblyRatio);
- }
-
- if (finalDiskSpaceCleanForciblyRatio > 0.85) {
- finalDiskSpaceCleanForciblyRatio = 0.85;
- }
- if (finalDiskSpaceCleanForciblyRatio < 0.30) {
- finalDiskSpaceCleanForciblyRatio = 0.30;
- }
-
- return finalDiskSpaceCleanForciblyRatio;
- }
-
- public void excuteDeleteFilesManualy() {
- this.manualDeleteFileSeveralTimes = MAX_MANUAL_DELETE_FILE_TIMES;
- StreamMessageStore.log.info("executeDeleteFilesManually was invoked");
- }
-
- public long run() {
- int deleteCount = 0;
- try {
- deleteCount = this.deleteExpiredFiles();
-
- this.redeleteHangedFile();
- } catch (Throwable e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- }
- return deleteCount;
- }
-
- private int deleteExpiredFiles() {
- int deleteCount = 0;
- long fileReservedTime = StreamMessageStore.this.getMessageStoreConfig().getFileReservedTime();
- int deletePhysicFilesInterval = StreamMessageStore.this.getMessageStoreConfig().getDeleteCommitLogFilesInterval();
- int destroyMapedFileIntervalForcibly = StreamMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly();
- int maxBatchDeleteFilesNum = StreamMessageStore.this.getMessageStoreConfig().getMaxBatchDeleteFilesNum();
- if (maxBatchDeleteFilesNum < 10) {
- maxBatchDeleteFilesNum = 10;
- }
-
- boolean timeup = this.isTimeToDelete();
- boolean spacefull = this.isSpaceToDelete();
- boolean manualDelete = this.manualDeleteFileSeveralTimes > 0;
-
- boolean needDelete = timeup || spacefull || manualDelete;
-
- if (needDelete) {
-
- if (manualDelete)
- this.manualDeleteFileSeveralTimes--;
-
- boolean cleanAtOnce = StreamMessageStore.this.getMessageStoreConfig().isCleanFileForciblyEnable() && this.cleanImmediately;
-
- String storePathPhysic = StreamMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
- double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
- long totalSpace = UtilAll.getTotalSpace(storePathPhysic);
- double realRatio = (StreamMessageStore.this.commitLog.getMaxOffset() - StreamMessageStore.this.commitLog.getMinOffset()) / (totalSpace + 0.001);
-
- cleanAtOnce = cleanAtOnce && (realRatio > 0.3);
-
- log.info("begin to delete before {} hours file. timeup:{} spacefull:{} manualDeleteFileSeveralTimes:{} cleanAtOnce:{} maxBatchDeleteFilesNum:{} physicRatio:{}",
- fileReservedTime,
- timeup,
- spacefull,
- manualDeleteFileSeveralTimes,
- cleanAtOnce,
- maxBatchDeleteFilesNum,
- physicRatio);
-
- fileReservedTime *= 60 * 60 * 1000;
-
- deleteCount = StreamMessageStore.this.commitLog.deleteExpiredFile(fileReservedTime, deletePhysicFilesInterval,
- destroyMapedFileIntervalForcibly, cleanAtOnce);
- if (deleteCount > 0) {
- } else if (spacefull) {
- log.warn("disk space will be full soon, but delete file failed. timeup:{} manualDelete:{} cleanAtOnce:{} physicRatio:{}",
- timeup,
- manualDelete,
- cleanAtOnce,
- physicRatio);
- }
- }
- return deleteCount;
- }
-
- private void redeleteHangedFile() {
- int interval = StreamMessageStore.this.getMessageStoreConfig().getRedeleteHangedFileInterval();
- long currentTimestamp = System.currentTimeMillis();
- if ((currentTimestamp - this.lastRedeleteTimestamp) > interval) {
- this.lastRedeleteTimestamp = currentTimestamp;
- int destroyMapedFileIntervalForcibly =
- StreamMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly();
- if (StreamMessageStore.this.commitLog.retryDeleteFirstFile(destroyMapedFileIntervalForcibly)) {
- }
- }
- }
-
- public String getServiceName() {
- return CleanCommitLogService.class.getSimpleName();
- }
-
- private boolean isTimeToDelete() {
- String when = StreamMessageStore.this.getMessageStoreConfig().getDeleteWhen();
- if (UtilAll.isItTimeToDo(when)) {
- StreamMessageStore.log.info("it's time to reclaim disk space, " + when);
- return true;
- }
-
- return false;
- }
-
- private boolean isSpaceToDelete() {
- double ratio = StreamMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 100.0;
-
- cleanImmediately = false;
-
- {
- double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(getStorePathPhysic());
- if (physicRatio > getDiskSpaceWarningLevelRatio()) {
- boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskFull();
- if (diskok) {
- StreamMessageStore.log.error("physic disk maybe full soon " + physicRatio + ", so mark disk full");
- }
-
- cleanImmediately = true;
- } else if (physicRatio > getDiskSpaceCleanForciblyRatio()) {
- cleanImmediately = true;
- } else {
- boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskOK();
- if (!diskok) {
- StreamMessageStore.log.info("physic disk space OK " + physicRatio + ", so mark disk ok");
- }
- }
-
- if (physicRatio < 0 || physicRatio > ratio) {
- StreamMessageStore.log.info("physic disk maybe full soon, so reclaim space, {}, cleanImmediately {}", physicRatio, cleanImmediately);
- return true;
- }
- }
-
- {
- String storePathLogics = StorePathConfigHelper
- .getStorePathConsumeQueue(StreamMessageStore.this.getMessageStoreConfig().getStorePathRootDir());
- double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
- if (logicsRatio > getDiskSpaceWarningLevelRatio()) {
- boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskFull();
- if (diskok) {
- StreamMessageStore.log.error("logics disk maybe full soon " + logicsRatio + ", so mark disk full");
- }
-
- cleanImmediately = true;
- } else if (logicsRatio > getDiskSpaceCleanForciblyRatio()) {
- cleanImmediately = true;
- } else {
- boolean diskok = StreamMessageStore.this.runningFlags.getAndMakeDiskOK();
- if (!diskok) {
- StreamMessageStore.log.info("logics disk space OK " + logicsRatio + ", so mark disk ok");
- }
- }
-
- if (logicsRatio < 0 || logicsRatio > ratio) {
- StreamMessageStore.log.info("logics disk maybe full soon, so reclaim space, " + logicsRatio);
- return true;
- }
- }
-
- return false;
- }
-
- public int getManualDeleteFileSeveralTimes() {
- return manualDeleteFileSeveralTimes;
- }
-
- public void setManualDeleteFileSeveralTimes(int manualDeleteFileSeveralTimes) {
- this.manualDeleteFileSeveralTimes = manualDeleteFileSeveralTimes;
- }
- }
-
- class CleanConsumeQueueService {
- private long lastPhysicalMinOffset = 0;
-
- public void run() {
- try {
- this.deleteExpiredFiles();
- } catch (Throwable e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- }
- }
-
- private void deleteExpiredFiles() {
- int deleteLogicsFilesInterval = StreamMessageStore.this.getMessageStoreConfig().getDeleteConsumeQueueFilesInterval();
-
- long minOffset = StreamMessageStore.this.commitLog.getMinOffset();
- if (minOffset > this.lastPhysicalMinOffset) {
- this.lastPhysicalMinOffset = minOffset;
-
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
-
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- int deleteCount = StreamMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minOffset);
-
- if (deleteCount > 0 && deleteLogicsFilesInterval > 0) {
- try {
- Thread.sleep(deleteLogicsFilesInterval);
- } catch (InterruptedException ignored) {
- }
- }
- }
- }
-
- StreamMessageStore.this.indexService.deleteExpiredFile(minOffset);
- }
- }
-
- public String getServiceName() {
- return CleanConsumeQueueService.class.getSimpleName();
- }
- }
-
- class CorrectLogicOffsetService {
-
- private long lastForceCorrectTime = -1L;
-
- public void run() {
- try {
- this.correctLogicMinOffset();
- } catch (Throwable e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- }
- }
-
- private boolean needCorrect(ConsumeQueueInterface logic, long minPhyOffset, long lastForeCorrectTimeCurRun) {
- if (logic == null) {
- return false;
- }
- // If first exist and not available, it means first file may destroy failed, delete it.
- if (StreamMessageStore.this.consumeQueueStore.isFirstFileExist(logic) && !StreamMessageStore.this.consumeQueueStore.isFirstFileAvailable(logic)) {
- log.error("CorrectLogicOffsetService.needCorrect. first file not available, trigger correct." +
- " topic:{}, queue:{}, maxPhyOffset in queue:{}, minPhyOffset " +
- "in commit log:{}, minOffset in queue:{}, maxOffset in queue:{}, cqType:{}"
- , logic.getTopic(), logic.getQueueId(), logic.getMaxPhysicOffset()
- , minPhyOffset, logic.getMinOffsetInQueue(), logic.getMaxOffsetInQueue(), logic.getCQType());
- return true;
- }
-
- // logic.getMaxPhysicOffset() or minPhyOffset = -1
- // means there is no message in current queue, so no need to correct.
- if (logic.getMaxPhysicOffset() == -1 || minPhyOffset == -1) {
- return false;
- }
-
- if (logic.getMaxPhysicOffset() < minPhyOffset) {
- if (logic.getMinOffsetInQueue() < logic.getMaxOffsetInQueue()) {
- log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is less than min phy offset: {}, " +
- "but min offset: {} is less than max offset: {}. topic:{}, queue:{}, cqType:{}."
- , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
- , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
- return true;
- } else if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
- return false;
- } else {
- log.error("CorrectLogicOffsetService.needCorrect. It should not happen, logic max phy offset: {} is less than min phy offset: {}," +
- " but min offset: {} is larger than max offset: {}. topic:{}, queue:{}, cqType:{}"
- , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
- , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
- return false;
- }
- }
- //the logic.getMaxPhysicOffset() >= minPhyOffset
- int forceCorrectInterval = StreamMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetForceInterval();
- if ((System.currentTimeMillis() - lastForeCorrectTimeCurRun) > forceCorrectInterval) {
- lastForceCorrectTime = System.currentTimeMillis();
- CqUnit cqUnit = logic.getEarliestUnit();
- if (cqUnit == null) {
- if (logic.getMinOffsetInQueue() == logic.getMaxOffsetInQueue()) {
- return false;
- } else {
- log.error("CorrectLogicOffsetService.needCorrect. cqUnit is null, logic max phy offset: {} is greater than min phy offset: {}, " +
- "but min offset: {} is not equal to max offset: {}. topic:{}, queue:{}, cqType:{}."
- , logic.getMaxPhysicOffset(), minPhyOffset, logic.getMinOffsetInQueue()
- , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
- return true;
- }
- }
-
- if (cqUnit.getPos() < minPhyOffset) {
- log.error("CorrectLogicOffsetService.needCorrect. logic max phy offset: {} is greater than min phy offset: {}, " +
- "but minPhyPos in cq is: {}. min offset in queue: {}, max offset in queue: {}, topic:{}, queue:{}, cqType:{}."
- , logic.getMaxPhysicOffset(), minPhyOffset, cqUnit.getPos(), logic.getMinOffsetInQueue()
- , logic.getMaxOffsetInQueue(), logic.getTopic(), logic.getQueueId(), logic.getCQType());
- return true;
- }
-
- if (cqUnit.getPos() >= minPhyOffset) {
-
- // Normal case, do not need correct.
- return false;
- }
- }
-
- return false;
- }
-
- private void correctLogicMinOffset() {
-
- long lastForeCorrectTimeCurRun = lastForceCorrectTime;
- long minPhyOffset = getMinPhyOffset();
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
- for (ConsumeQueueInterface logic : maps.values()) {
- if (needCorrect(logic, minPhyOffset, lastForeCorrectTimeCurRun)) {
- doCorrect(logic, minPhyOffset);
- }
- }
- }
- }
-
- private void doCorrect(ConsumeQueueInterface logic, long minPhyOffset) {
- StreamMessageStore.this.consumeQueueStore.deleteExpiredFile(logic, minPhyOffset);
- int sleepIntervalWhenCorrectMinOffset = StreamMessageStore.this.getMessageStoreConfig().getCorrectLogicMinOffsetSleepInterval();
- if (sleepIntervalWhenCorrectMinOffset > 0) {
- try {
- Thread.sleep(sleepIntervalWhenCorrectMinOffset);
- } catch (InterruptedException ignored) {
- }
- }
- }
-
- public String getServiceName() {
- return CorrectLogicOffsetService.class.getSimpleName();
- }
- }
-
- class FlushConsumeQueueService extends ServiceThread {
- private static final int RETRY_TIMES_OVER = 3;
- private long lastFlushTimestamp = 0;
-
- private void doFlush(int retryTimes) {
- int flushConsumeQueueLeastPages = StreamMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueLeastPages();
-
- if (retryTimes == RETRY_TIMES_OVER) {
- flushConsumeQueueLeastPages = 0;
- }
-
- long logicsMsgTimestamp = 0;
-
- int flushConsumeQueueThoroughInterval = StreamMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueThoroughInterval();
- long currentTimeMillis = System.currentTimeMillis();
- if (currentTimeMillis >= (this.lastFlushTimestamp + flushConsumeQueueThoroughInterval)) {
- this.lastFlushTimestamp = currentTimeMillis;
- flushConsumeQueueLeastPages = 0;
- logicsMsgTimestamp = StreamMessageStore.this.getStoreCheckpoint().getLogicsMsgTimestamp();
- }
-
- ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> tables = StreamMessageStore.this.consumeQueueTable;
-
- for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : tables.values()) {
- for (ConsumeQueueInterface cq : maps.values()) {
- boolean result = false;
- for (int i = 0; i < retryTimes && !result; i++) {
- result = StreamMessageStore.this.consumeQueueStore.flush(cq, flushConsumeQueueLeastPages);
- }
- }
- }
-
- if (0 == flushConsumeQueueLeastPages) {
- if (logicsMsgTimestamp > 0) {
- StreamMessageStore.this.getStoreCheckpoint().setLogicsMsgTimestamp(logicsMsgTimestamp);
- }
- StreamMessageStore.this.getStoreCheckpoint().flush();
- }
- }
-
- @Override
- public void run() {
- StreamMessageStore.log.info(this.getServiceName() + " service started");
-
- while (!this.isStopped()) {
- try {
- int interval = StreamMessageStore.this.getMessageStoreConfig().getFlushIntervalConsumeQueue();
- this.waitForRunning(interval);
- this.doFlush(1);
- } catch (Exception e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- }
- }
-
- this.doFlush(RETRY_TIMES_OVER);
-
- StreamMessageStore.log.info(this.getServiceName() + " service end");
- }
-
- @Override
- public String getServiceName() {
- return FlushConsumeQueueService.class.getSimpleName();
- }
-
- @Override
- public long getJointime() {
- return 1000 * 60;
- }
- }
-
- private class SyncReputMessageServiceFutureItem {
- private Future future;
- private long nextReputFromOffset;
-
- public SyncReputMessageServiceFutureItem(Future future, long nextReputFromOffset) {
- this.future = future;
- this.nextReputFromOffset = nextReputFromOffset;
- }
-
- public Future getFuture() {
- return future;
- }
-
- public void setFuture(Future future) {
- this.future = future;
- }
-
- public long getNextReputFromOffset() {
- return nextReputFromOffset;
- }
-
- public void setNextReputFromOffset(long nextReputFromOffset) {
- this.nextReputFromOffset = nextReputFromOffset;
- }
- }
-
- class SyncReputMessageService extends ReputMessageService {
-
- private BlockingQueue<SyncReputMessageServiceFutureItem> reputResultQueue;
-
- SyncReputMessageService() {
- reputResultQueue = new LinkedBlockingDeque<>(messageStoreConfig.getDispatchCqThreads() * messageStoreConfig.getDispatchCqCacheNum());
- }
-
- void processDispatchRequestForRecover(DispatchRequest dispatchRequest) {
- int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
- if (dispatchRequest.isSuccess() && size > 0) {
- StreamMessageStore.this.doDispatch(dispatchRequest);
- this.reputFromOffset = Math.max(dispatchRequest.getNextReputFromOffset(), this.reputFromOffset);
- }
- }
-
- void processDispatchRequest(DispatchRequest dispatchRequest) throws InterruptedException {
- int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
- if (dispatchRequest.isSuccess() && size > 0) {
- Future future = StreamMessageStore.this.doDispatch(dispatchRequest, messageStoreConfig.isEnableAsyncReput());
- reputResultQueue.put(new SyncReputMessageServiceFutureItem(future, dispatchRequest.getNextReputFromOffset()));
- }
- }
-
- @Override
- public void shutdown() {
- for (int i = 0; i < 30 && this.reputResultQueue.size() > 0; i++) {
- try {
- Thread.sleep(1000);
- } catch (InterruptedException ignored) {
- }
- }
-
- if (this.reputResultQueue.size() > 0) {
- log.warn("shutdown SyncReputMessageService, but reputResultQueue not all processed, CLMaxOffset: {} reputFromOffset: {}",
- StreamMessageStore.this.commitLog.getMaxOffset(), this.reputFromOffset);
- }
-
- super.shutdown();
- }
- @Override
- public void run() {
- StreamMessageStore.log.info(this.getServiceName() + " service started");
- while (!this.isStopped()) {
- try {
- SyncReputMessageServiceFutureItem item = reputResultQueue.poll(1, TimeUnit.SECONDS);
- if (null == item) {
- continue;
- }
- item.getFuture().get();
- this.reputFromOffset = Math.max(item.getNextReputFromOffset(), this.reputFromOffset);
-
- } catch (Exception e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- waitForRunning(100);
- }
- }
-
- StreamMessageStore.log.info(this.getServiceName() + " service end");
- }
-
- @Override
- public String getServiceName() {
- return SyncReputMessageService.class.getSimpleName();
- }
- }
-
- class ReputMessageService extends ServiceThread {
-
- protected volatile long reputFromOffset = 0;
-
- public long getReputFromOffset() {
- return reputFromOffset;
- }
-
- public void setReputFromOffset(long reputFromOffset) {
- this.reputFromOffset = reputFromOffset;
- }
-
- public boolean dispatched(long physicalOffset) {
- return this.reputFromOffset > physicalOffset;
- }
-
- @Override
- public void shutdown() {
- for (int i = 0; i < 30 && this.isCommitLogAvailable(); i++) {
- try {
- Thread.sleep(1000);
- } catch (InterruptedException ignored) {
- }
- }
-
- if (this.isCommitLogAvailable()) {
- log.warn("shutdown ReputMessageService, but commitlog have not finish to be dispatched, CL: {} reputFromOffset: {}",
- StreamMessageStore.this.commitLog.getMaxOffset(), this.reputFromOffset);
- }
-
- super.shutdown();
- }
-
- public long behind() {
- return StreamMessageStore.this.commitLog.getMaxOffset() - this.reputFromOffset;
- }
-
- protected boolean isCommitLogAvailable() {
- return this.reputFromOffset < StreamMessageStore.this.commitLog.getMaxOffset();
- }
-
- private void doReput() throws Exception {
- if (this.reputFromOffset < StreamMessageStore.this.commitLog.getMinOffset()) {
- log.warn("The reputFromOffset={} is smaller than minPyOffset={}, this usually indicate that the dispatch behind too much and the commitlog has expired.",
- this.reputFromOffset, StreamMessageStore.this.commitLog.getMinOffset());
- this.reputFromOffset = StreamMessageStore.this.commitLog.getMinOffset();
- }
- for (boolean doNext = true; this.isCommitLogAvailable() && doNext; ) {
-
- if (StreamMessageStore.this.getMessageStoreConfig().isDuplicationEnable()
- && this.reputFromOffset >= StreamMessageStore.this.getConfirmOffset()) {
- break;
- }
-
- SelectMappedBufferResult result = StreamMessageStore.this.commitLog.getData(reputFromOffset);
- boolean reputAsync = messageStoreConfig.isEnableAsyncReput();
- if (result != null) {
- long cacheReputFromOffset = this.reputFromOffset;
- List<Future> futures = new LinkedList<>();
- try {
- this.reputFromOffset = result.getStartOffset();
- cacheReputFromOffset = this.reputFromOffset;
-
- for (int readSize = 0; readSize < result.getSize() && doNext; ) {
- DispatchRequest dispatchRequest =
- StreamMessageStore.this.commitLog.checkMessageAndReturnSize(result.getByteBuffer(), false, false);
- int size = dispatchRequest.getBufferSize() == -1 ? dispatchRequest.getMsgSize() : dispatchRequest.getBufferSize();
-
- if (dispatchRequest.isSuccess()) {
- if (size > 0) {
- futures.add(StreamMessageStore.this.doDispatch(dispatchRequest, reputAsync));
-
- if (BrokerRole.SLAVE != StreamMessageStore.this.getMessageStoreConfig().getBrokerRole()
- && StreamMessageStore.this.brokerConfig.isLongPollingEnable()) {
- StreamMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
- dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
- dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
- dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
- }
-
- cacheReputFromOffset += size;
- readSize += size;
- if (StreamMessageStore.this.getMessageStoreConfig().getBrokerRole() == BrokerRole.SLAVE) {
- StreamMessageStore.this.storeStatsService
- .getSinglePutMessageTopicTimesTotal(dispatchRequest.getTopic()).add(1);
- StreamMessageStore.this.storeStatsService
- .getSinglePutMessageTopicSizeTotal(dispatchRequest.getTopic())
- .add(dispatchRequest.getMsgSize());
- }
- } else if (size == 0) {
- cacheReputFromOffset = StreamMessageStore.this.commitLog.rollNextFile(cacheReputFromOffset);
- readSize = result.getSize();
- }
- } else if (!dispatchRequest.isSuccess()) {
-
- if (size > 0) {
- log.error("[BUG]read total count not equals msg total size. reputFromOffset={}", cacheReputFromOffset);
- cacheReputFromOffset += size;
- } else {
- doNext = false;
- if (StreamMessageStore.this.brokerConfig.getBrokerId() == MixAll.MASTER_ID) {
- log.error("[BUG]the master dispatch message to consume queue error, COMMITLOG OFFSET: {}",
- cacheReputFromOffset);
- cacheReputFromOffset += result.getSize() - readSize;
- }
- }
- }
- }
- for (Future future: futures) {
- future.get();
- }
- futures.clear();
- this.reputFromOffset = cacheReputFromOffset;
- } finally {
- result.release();
- }
- } else {
- doNext = false;
- }
- }
- }
-
- @Override
- public void run() {
- StreamMessageStore.log.info(this.getServiceName() + " service started");
-
- while (!this.isStopped()) {
- try {
- Thread.sleep(1);
- this.doReput();
- } catch (Exception e) {
- StreamMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
- }
- }
-
- StreamMessageStore.log.info(this.getServiceName() + " service end");
- }
-
- @Override
- public String getServiceName() {
- return ReputMessageService.class.getSimpleName();
- }
-
- }
-}
diff --git a/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
index 97c50b9..a78eeed 100644
--- a/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
+++ b/store/src/main/java/org/apache/rocketmq/store/TopicQueueLock.java
@@ -8,12 +8,13 @@
*
* http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
+
package org.apache.rocketmq.store;
import java.util.ArrayList;
diff --git a/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java b/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
index 7a41c27..e58e695 100644
--- a/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
+++ b/store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java
@@ -19,7 +19,6 @@ package org.apache.rocketmq.store.config;
import org.apache.rocketmq.common.annotation.ImportantField;
import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.queue.BatchConsumeQueue;
-import org.apache.rocketmq.store.queue.CQType;
import java.io.File;
@@ -206,9 +205,6 @@ public class MessageStoreConfig {
@ImportantField
private boolean enableCleanExpiredOffset = false;
- @ImportantField
- private String defaultCQType = CQType.SimpleCQ.toString();
-
private int maxAsyncPutMessageRequests = 5000;
private int pullBatchMaxMessageCount = 160;
@@ -799,13 +795,6 @@ public class MessageStoreConfig {
this.enableCleanExpiredOffset = enableCleanExpiredOffset;
}
- public String getDefaultCQType() {
- return defaultCQType;
- }
-
- public void setDefaultCQType(String defaultCQType) {
- this.defaultCQType = defaultCQType;
- }
public String getReadOnlyCommitLogStorePaths() {
return readOnlyCommitLogStorePaths;
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
index 50078c9..cbe794e 100644
--- a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
+++ b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
@@ -33,7 +33,6 @@ import io.openmessaging.storage.dledger.utils.DLedgerUtils;
import java.net.Inet6Address;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
-import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
@@ -431,7 +430,7 @@ public class DLedgerCommitLog extends CommitLog {
String topicQueueKey = msg.getTopic() + "-" + msg.getQueueId();
topicQueueLock.lock(topicQueueKey);
try {
- defaultMessageStore.assignOffset(topicQueueKey, msg, getBatchNum(msg));
+ defaultMessageStore.assignOffset(topicQueueKey, msg, getMessageNum(msg));
encodeResult = this.messageSerializer.serialize(msg);
if (encodeResult.status != AppendMessageStatus.PUT_OK) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java b/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java
index 0b6a9dd..7d1feba 100644
--- a/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java
+++ b/store/src/main/java/org/apache/rocketmq/store/logfile/MappedFile.java
@@ -18,7 +18,6 @@ package org.apache.rocketmq.store.logfile;
import org.apache.rocketmq.store.AppendMessageCallback;
import org.apache.rocketmq.store.AppendMessageResult;
-import org.apache.rocketmq.store.CommitLog;
import org.apache.rocketmq.store.MessageExtBatch;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.PutMessageContext;
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
index de0e636..648a472 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java
@@ -17,11 +17,17 @@
package org.apache.rocketmq.store.queue;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.message.MessageAccessor;
+import org.apache.rocketmq.common.message.MessageConst;
+import org.apache.rocketmq.common.message.MessageDecoder;
+import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.DispatchRequest;
import org.apache.rocketmq.store.MappedFileQueue;
+import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.config.BrokerRole;
@@ -29,6 +35,7 @@ import org.apache.rocketmq.store.logfile.MappedFile;
import java.io.File;
import java.nio.ByteBuffer;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentSkipListMap;
@@ -386,9 +393,9 @@ public class BatchConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCy
}
@Override
- public int deleteExpiredFile(long offset) {
- int cnt = this.mappedFileQueue.deleteExpiredFileByOffset(offset, CQ_STORE_UNIT_SIZE);
- this.correctMinOffset(offset);
+ public int deleteExpiredFile(long minCommitLogPos) {
+ int cnt = this.mappedFileQueue.deleteExpiredFileByOffset(minCommitLogPos, CQ_STORE_UNIT_SIZE);
+ this.correctMinOffset(minCommitLogPos);
return cnt;
}
@@ -473,6 +480,21 @@ public class BatchConsumeQueue implements ConsumeQueueInterface, FileQueueLifeCy
this.defaultMessageStore.getRunningFlags().makeLogicsQueueError();
}
+ @Override
+ public void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageExtBrokerInner msg, short messageNum) {
+ HashMap<String, Long> batchTopicQueueTable = queueOffsetAssigner.getBatchTopicQueueTable();
+ String topicQueueKey = getTopic() + "-" + getQueueId();
+
+ Long topicOffset = batchTopicQueueTable.computeIfAbsent(topicQueueKey, k -> 0L);
+
+ if (MessageSysFlag.check(msg.getSysFlag(), MessageSysFlag.INNER_BATCH_FLAG)) {
+ MessageAccessor.putProperty(msg, MessageConst.PROPERTY_INNER_BASE, String.valueOf(topicOffset));
+ msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
+ }
+ msg.setQueueOffset(topicOffset);
+ batchTopicQueueTable.put(topicQueueKey, topicOffset + messageNum);
+ }
+
boolean putBatchMessagePositionInfo(final long offset, final int size, final long tagsCode, final long storeTime,
final long msgBaseOffset, final short batchSize) {
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java
index 48f717d..5232a74 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java
@@ -17,6 +17,8 @@
package org.apache.rocketmq.store.queue;
+import org.apache.rocketmq.common.attribute.CQType;
+
public interface ConsumeQueueInterface {
/**
* Get the topic name
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
index bf42e74..e8146ff 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
@@ -16,27 +16,49 @@
*/
package org.apache.rocketmq.store.queue;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.attribute.CQType;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.topic.TopicValidator;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.ConsumeQueue;
-import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StoreUtil;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
public class ConsumeQueueStore {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
protected final MessageStore messageStore;
protected final MessageStoreConfig messageStoreConfig;
+ protected final QueueOffsetAssigner queueOffsetAssigner = new QueueOffsetAssigner();
protected final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface>> consumeQueueTable;
- public ConsumeQueueStore(MessageStore messageStore, MessageStoreConfig messageStoreConfig, ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> consumeQueueTable) {
+ // Should be careful, do not change the topic config
+ // TopicConfigManager is more suitable here.
+ private ConcurrentMap<String, TopicConfig> topicConfigTable;
+
+ public ConsumeQueueStore(MessageStore messageStore, MessageStoreConfig messageStoreConfig) {
this.messageStore = messageStore;
this.messageStoreConfig = messageStoreConfig;
- this.consumeQueueTable = consumeQueueTable;
+ this.consumeQueueTable = new ConcurrentHashMap<>(32);
+ }
+
+ public void setTopicConfigTable(ConcurrentMap<String, TopicConfig> topicConfigTable) {
+ this.topicConfigTable = topicConfigTable;
}
private FileQueueLifeCycle getLifeCycle(String topic, int queueId) {
@@ -64,21 +86,143 @@ public class ConsumeQueueStore {
fileQueueLifeCycle.putMessagePositionInfoWrapper(request);
}
+ public void putMessagePositionInfoWrapper(DispatchRequest dispatchRequest) {
+ ConsumeQueueInterface cq = this.findOrCreateConsumeQueue(dispatchRequest.getTopic(), dispatchRequest.getQueueId());
+ this.putMessagePositionInfoWrapper(cq, dispatchRequest);
+ }
+
public boolean load(ConsumeQueueInterface consumeQueue) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
return fileQueueLifeCycle.load();
}
+ public boolean load() {
+ return loadConsumeQueues() && loadBatchConsumeQueues();
+ }
+
+ private boolean loadBatchConsumeQueues() {
+ File dirLogic = new File(StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ File[] fileTopicList = dirLogic.listFiles();
+ if (fileTopicList != null) {
+
+ for (File fileTopic : fileTopicList) {
+ String topic = fileTopic.getName();
+
+ File[] fileQueueIdList = fileTopic.listFiles();
+ if (fileQueueIdList != null) {
+ for (File fileQueueId : fileQueueIdList) {
+ int queueId;
+ try {
+ queueId = Integer.parseInt(fileQueueId.getName());
+ } catch (NumberFormatException e) {
+ continue;
+ }
+
+ TopicConfig topicConfig = this.topicConfigTable == null ? null : this.topicConfigTable.get(topic);
+
+ // For batch consume queue, the topic config must exist
+ if (topicConfig == null) {
+ log.warn("topic: {} has no topic config.", topic);
+ continue;
+ }
+
+ if (!Objects.equals(CQType.BatchCQ, QueueTypeUtils.getCQType(Optional.of(topicConfig)))) {
+ log.error("[BUG]topic: {} should be BCQ.", topic);
+ }
+
+ ConsumeQueueInterface logic = new BatchConsumeQueue(
+ topic,
+ queueId,
+ StorePathConfigHelper.getStorePathBatchConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
+ this.messageStore);
+ this.putConsumeQueue(topic, queueId, logic);
+ if (!this.load(logic)) {
+ return false;
+ }
+ }
+ }
+ }
+ }
+
+ log.info("load batch consume queue all over, OK");
+
+ return true;
+ }
+
+ private boolean loadConsumeQueues() {
+ File dirLogic = new File(StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()));
+ File[] fileTopicList = dirLogic.listFiles();
+ if (fileTopicList != null) {
+
+ for (File fileTopic : fileTopicList) {
+ String topic = fileTopic.getName();
+
+ File[] fileQueueIdList = fileTopic.listFiles();
+ if (fileQueueIdList != null) {
+ for (File fileQueueId : fileQueueIdList) {
+ int queueId;
+ try {
+ queueId = Integer.parseInt(fileQueueId.getName());
+ } catch (NumberFormatException e) {
+ continue;
+ }
+ ConsumeQueueInterface logic = new ConsumeQueue(
+ topic,
+ queueId,
+ StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
+ this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
+ this.messageStore);
+ this.putConsumeQueue(topic, queueId, logic);
+ if (!this.load(logic)) {
+ return false;
+ }
+ }
+ }
+ }
+ }
+
+ log.info("load logics queue all over, OK");
+
+ return true;
+ }
+
public void recover(ConsumeQueueInterface consumeQueue) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
fileQueueLifeCycle.recover();
}
+ public long recover() {
+ long maxPhysicOffset = -1;
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.recover(logic);
+ if (logic.getMaxPhysicOffset() > maxPhysicOffset) {
+ maxPhysicOffset = logic.getMaxPhysicOffset();
+ }
+ }
+ }
+
+ return maxPhysicOffset;
+ }
+
public void checkSelf(ConsumeQueueInterface consumeQueue) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
fileQueueLifeCycle.checkSelf();
}
+ public void checkSelf() {
+ Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ Iterator<Map.Entry<Integer, ConsumeQueueInterface>> itNext = next.getValue().entrySet().iterator();
+ while (itNext.hasNext()) {
+ Map.Entry<Integer, ConsumeQueueInterface> cq = itNext.next();
+ this.checkSelf(cq.getValue());
+ }
+ }
+ }
+
public boolean flush(ConsumeQueueInterface consumeQueue, int flushLeastPages) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
return fileQueueLifeCycle.flush(flushLeastPages);
@@ -89,9 +233,9 @@ public class ConsumeQueueStore {
fileQueueLifeCycle.destroy();
}
- public int deleteExpiredFile(ConsumeQueueInterface consumeQueue, long minOffset) {
+ public int deleteExpiredFile(ConsumeQueueInterface consumeQueue, long minCommitLogPos) {
FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(consumeQueue.getTopic(), consumeQueue.getQueueId());
- return fileQueueLifeCycle.deleteExpiredFile(minOffset);
+ return fileQueueLifeCycle.deleteExpiredFile(minCommitLogPos);
}
public void truncateDirtyLogicFiles(ConsumeQueueInterface consumeQueue, long phyOffset) {
@@ -124,10 +268,9 @@ public class ConsumeQueueStore {
}
private ConsumeQueueInterface doFindOrCreateConsumeQueue(String topic, int queueId) {
-
ConcurrentMap<Integer, ConsumeQueueInterface> map = consumeQueueTable.get(topic);
if (null == map) {
- ConcurrentMap<Integer, ConsumeQueueInterface> newMap = new ConcurrentHashMap<Integer, ConsumeQueueInterface>(128);
+ ConcurrentMap<Integer, ConsumeQueueInterface> newMap = new ConcurrentHashMap<>(128);
ConcurrentMap<Integer, ConsumeQueueInterface> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
if (oldMap != null) {
map = oldMap;
@@ -143,7 +286,9 @@ public class ConsumeQueueStore {
ConsumeQueueInterface newLogic;
- if (StoreUtil.isStreamMode(this.messageStore)) {
+ Optional<TopicConfig> topicConfig = this.getTopicConfig(topic);
+ // TODO maybe the topic has been deleted.
+ if (Objects.equals(CQType.BatchCQ, QueueTypeUtils.getCQType(topicConfig))) {
newLogic = new BatchConsumeQueue(
topic,
queueId,
@@ -162,7 +307,7 @@ public class ConsumeQueueStore {
queueId,
StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
this.messageStoreConfig.getMappedFileSizeConsumeQueue(),
- (DefaultMessageStore) this.messageStore);
+ this.messageStore);
ConsumeQueueInterface oldLogic = map.putIfAbsent(queueId, newLogic);
if (oldLogic != null) {
logic = oldLogic;
@@ -173,4 +318,129 @@ public class ConsumeQueueStore {
return logic;
}
+
+ public Long getMaxOffset(String topic, int queueId) {
+ return this.queueOffsetAssigner.getTopicQueueTable().get(topic + "-" + queueId);
+ }
+
+ public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
+ this.queueOffsetAssigner.setTopicQueueTable(topicQueueTable);
+ }
+
+ public void setBatchTopicQueueTable(HashMap<String, Long> batchTopicQueueTable) {
+ this.queueOffsetAssigner.setBatchTopicQueueTable(batchTopicQueueTable);
+ }
+
+ public void assignQueueOffset(MessageExtBrokerInner msg, short messageNum) {
+ FileQueueLifeCycle fileQueueLifeCycle = getLifeCycle(msg.getTopic(), msg.getQueueId());
+ fileQueueLifeCycle.assignQueueOffset(this.queueOffsetAssigner, msg, messageNum);
+ }
+
+ public void removeTopicQueueTable(String topic, Integer queueId) {
+ this.queueOffsetAssigner.remove(topic, queueId);
+ }
+
+ public ConcurrentMap<String, ConcurrentMap<Integer, ConsumeQueueInterface>> getConsumeQueueTable() {
+ return consumeQueueTable;
+ }
+
+ private void putConsumeQueue(final String topic, final int queueId, final ConsumeQueueInterface consumeQueue) {
+ ConcurrentMap<Integer/* queueId */, ConsumeQueueInterface> map = this.consumeQueueTable.get(topic);
+ if (null == map) {
+ map = new ConcurrentHashMap<>();
+ map.put(queueId, consumeQueue);
+ this.consumeQueueTable.put(topic, map);
+ } else {
+ map.put(queueId, consumeQueue);
+ }
+ }
+
+ public void recoverOffsetTable(long minPhyOffset) {
+ HashMap<String, Long> cqOffsetTable = new HashMap<>(1024);
+ HashMap<String, Long> bcqOffsetTable = new HashMap<>(1024);
+
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ String key = logic.getTopic() + "-" + logic.getQueueId();
+
+ long maxOffsetInQueue = logic.getMaxOffsetInQueue();
+ if (Objects.equals(CQType.BatchCQ, logic.getCQType())) {
+ bcqOffsetTable.put(key, maxOffsetInQueue);
+ } else {
+ cqOffsetTable.put(key, maxOffsetInQueue);
+ }
+
+ this.correctMinOffset(logic, minPhyOffset);
+ }
+ }
+
+ this.setTopicQueueTable(cqOffsetTable);
+ this.setBatchTopicQueueTable(bcqOffsetTable);
+ }
+
+ public void destroy() {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.destroy(logic);
+ }
+ }
+ }
+
+ public void cleanExpired(long minCommitLogOffset) {
+ Iterator<Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>>> it = this.consumeQueueTable.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, ConcurrentMap<Integer, ConsumeQueueInterface>> next = it.next();
+ String topic = next.getKey();
+ if (!topic.equals(TopicValidator.RMQ_SYS_SCHEDULE_TOPIC)) {
+ ConcurrentMap<Integer, ConsumeQueueInterface> queueTable = next.getValue();
+ Iterator<Map.Entry<Integer, ConsumeQueueInterface>> itQT = queueTable.entrySet().iterator();
+ while (itQT.hasNext()) {
+ Map.Entry<Integer, ConsumeQueueInterface> nextQT = itQT.next();
+ long maxCLOffsetInConsumeQueue = nextQT.getValue().getLastOffset();
+
+ if (maxCLOffsetInConsumeQueue == -1) {
+ log.warn("maybe ConsumeQueue was created just now. topic={} queueId={} maxPhysicOffset={} minLogicOffset={}.",
+ nextQT.getValue().getTopic(),
+ nextQT.getValue().getQueueId(),
+ nextQT.getValue().getMaxPhysicOffset(),
+ nextQT.getValue().getMinLogicOffset());
+ } else if (maxCLOffsetInConsumeQueue < minCommitLogOffset) {
+ log.info(
+ "cleanExpiredConsumerQueue: {} {} consumer queue destroyed, minCommitLogOffset: {} maxCLOffsetInConsumeQueue: {}",
+ topic,
+ nextQT.getKey(),
+ minCommitLogOffset,
+ maxCLOffsetInConsumeQueue);
+
+ removeTopicQueueTable(nextQT.getValue().getTopic(),
+ nextQT.getValue().getQueueId());
+
+ this.destroy(nextQT.getValue());
+ itQT.remove();
+ }
+ }
+
+ if (queueTable.isEmpty()) {
+ log.info("cleanExpiredConsumerQueue: {},topic destroyed", topic);
+ it.remove();
+ }
+ }
+ }
+ }
+
+ public void truncateDirty(long phyOffset) {
+ for (ConcurrentMap<Integer, ConsumeQueueInterface> maps : this.consumeQueueTable.values()) {
+ for (ConsumeQueueInterface logic : maps.values()) {
+ this.truncateDirtyLogicFiles(logic, phyOffset);
+ }
+ }
+ }
+
+ public Optional<TopicConfig> getTopicConfig(String topic) {
+ if (this.topicConfigTable == null) {
+ return Optional.empty();
+ }
+
+ return Optional.ofNullable(this.topicConfigTable.get(topic));
+ }
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/FileQueueLifeCycle.java b/store/src/main/java/org/apache/rocketmq/store/queue/FileQueueLifeCycle.java
index dd8c86d..05c217f 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/FileQueueLifeCycle.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/FileQueueLifeCycle.java
@@ -17,6 +17,7 @@
package org.apache.rocketmq.store.queue;
import org.apache.rocketmq.store.DispatchRequest;
+import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.Swappable;
public interface FileQueueLifeCycle extends Swappable {
@@ -32,4 +33,5 @@ public interface FileQueueLifeCycle extends Swappable {
boolean isFirstFileExist();
void correctMinOffset(long minCommitLogOffset);
void putMessagePositionInfoWrapper(DispatchRequest request);
+ void assignQueueOffset(QueueOffsetAssigner queueOffsetAssigner, MessageExtBrokerInner msg, short messageNum);
}
diff --git a/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java b/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java
new file mode 100644
index 0000000..09e18ec
--- /dev/null
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetAssigner.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store.queue;
+
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+
+import java.util.HashMap;
+
+/**
+ * QueueOffsetAssigner is a component for assigning queue.
+ *
+ */
+public class QueueOffsetAssigner {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);
+
+ private HashMap<String, Long> topicQueueTable = new HashMap<>(1024);
+ private HashMap<String, Long> batchTopicQueueTable = new HashMap<>(1024);
+
+ public HashMap<String, Long> getTopicQueueTable() {
+ return topicQueueTable;
+ }
+
+ public void setTopicQueueTable(HashMap<String, Long> topicQueueTable) {
+ this.topicQueueTable = topicQueueTable;
+ }
+
+ public HashMap<String, Long> getBatchTopicQueueTable() {
+ return batchTopicQueueTable;
+ }
+
+ public void setBatchTopicQueueTable(HashMap<String, Long> batchTopicQueueTable) {
+ this.batchTopicQueueTable = batchTopicQueueTable;
+ }
+
+ public synchronized void remove(String topic, Integer queueId) {
+ String topicQueueKey = topic + "-" + queueId;
+ // Beware of thread-safety
+ this.topicQueueTable.remove(topicQueueKey);
+ this.batchTopicQueueTable.remove(topicQueueKey);
+
+ log.info("removeQueueFromTopicQueueTable OK Topic: {} QueueId: {}", topic, queueId);
+ }
+}
\ No newline at end of file
diff --git a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java b/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
deleted file mode 100644
index 1067337..0000000
--- a/store/src/main/java/org/apache/rocketmq/store/util/QueueTypeUtils.java
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.rocketmq.store.util;
-
-import org.apache.rocketmq.common.TopicAttributes;
-import org.apache.rocketmq.common.TopicConfig;
-import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.MessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
-
-import java.util.Map;
-
-public class QueueTypeUtils {
-
- @Deprecated
- public static CQType getCQType(MessageStore messageStore) {
- if (messageStore instanceof DefaultMessageStore) {
- return CQType.SimpleCQ;
- } else if (messageStore instanceof StreamMessageStore) {
- return CQType.BatchCQ;
- } else {
- throw new RuntimeException("new cq type is not supported now.");
- }
- }
-
- public static CQType getCQType(TopicConfig topicConfig) {
- String attributeName = TopicAttributes.QUEUE_TYPE.getName();
-
- Map<String, String> attributes = topicConfig.getAttributes();
- if (attributes == null || attributes.size() == 0) {
- return CQType.valueOf(TopicAttributes.QUEUE_TYPE.getDefaultValue());
- }
-
- if (attributes.containsKey(attributeName)) {
- return CQType.valueOf(attributes.get(attributeName));
- } else {
- return CQType.valueOf(TopicAttributes.QUEUE_TYPE.getDefaultValue());
- }
- }
-}
diff --git a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreShutDownTest.java b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreShutDownTest.java
index 1ccb974..e27483e 100644
--- a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreShutDownTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreShutDownTest.java
@@ -23,7 +23,7 @@ import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
import org.junit.After;
import org.junit.Before;
@@ -72,7 +72,6 @@ public class DefaultMessageStoreShutDownTest {
messageStoreConfig.setMaxIndexNum(100 * 100);
messageStoreConfig.setFlushDiskType(FlushDiskType.SYNC_FLUSH);
messageStoreConfig.setHaListenPort(StoreTestBase.nextPort());
- messageStoreConfig.setDefaultCQType(CQType.SimpleCQ.name());
return new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), null, new BrokerConfig());
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreTest.java b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreTest.java
index 41cf92f..932fc2f 100644
--- a/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreTest.java
@@ -38,7 +38,7 @@ import org.apache.rocketmq.store.config.FlushDiskType;
import org.apache.rocketmq.store.config.MessageStoreConfig;
import org.apache.rocketmq.store.config.StorePathConfigHelper;
import org.apache.rocketmq.store.logfile.DefaultMappedFile;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.store.queue.ConsumeQueueInterface;
import org.apache.rocketmq.store.queue.CqUnit;
import org.apache.rocketmq.store.stats.BrokerStatsManager;
@@ -115,7 +115,6 @@ public class DefaultMessageStoreTest {
messageStoreConfig.setFlushDiskType(FlushDiskType.SYNC_FLUSH);
messageStoreConfig.setFlushIntervalConsumeQueue(1);
messageStoreConfig.setHaListenPort(StoreTestBase.nextPort());
- messageStoreConfig.setDefaultCQType(CQType.SimpleCQ.name());
return new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MyMessageArrivingListener(), new BrokerConfig());
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
index 3ebe148..500cd81 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeMessageTest.java
@@ -17,9 +17,17 @@
package org.apache.rocketmq.store.queue;
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.attribute.CQType;
+import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.sysflag.MessageSysFlag;
+import org.apache.rocketmq.common.utils.QueueTypeUtils;
+import org.apache.rocketmq.store.DefaultMessageStore;
import org.apache.rocketmq.store.GetMessageResult;
import org.apache.rocketmq.store.GetMessageStatus;
import org.apache.rocketmq.store.MessageExtBrokerInner;
@@ -27,21 +35,168 @@ import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
+import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
+import java.io.File;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
import java.util.UUID;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.awaitility.Awaitility.await;
public class BatchConsumeMessageTest extends QueueTestBase {
+ private MessageStore messageStore;
- @Test
- public void testGetOffsetInQueueByTime() throws Exception {
- MessageStore messageStore = createMessageStore(null, true, CQType.BatchCQ);
+ @Before
+ public void init() throws Exception {
+ messageStore = createMessageStore(null, true);
messageStore.load();
messageStore.start();
+ }
+
+ @After
+ public void destroy() {
+ messageStore.shutdown();
+ messageStore.destroy();
+
+ File file = new File(messageStore.getMessageStoreConfig().getStorePathRootDir());
+ UtilAll.deleteFile(file);
+ }
+
+ @Test
+ public void testSendMessagesToCqTopic() {
+ String topic = UUID.randomUUID().toString();
+ createTopic(topic, CQType.SimpleCQ, messageStore);
+
+ int batchNum = 10;
+
+ // case 1 has PROPERTY_INNER_NUM but has no INNER_BATCH_FLAG
+ MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, batchNum);
+ messageExtBrokerInner.setSysFlag(0);
+ PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
+
+ // case 2 has PROPERTY_INNER_NUM and has INNER_BATCH_FLAG, but is not a batchCq
+ messageExtBrokerInner = buildMessage(topic, 1);
+ putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
+
+ // case 3 has neither PROPERTY_INNER_NUM nor INNER_BATCH_FLAG.
+ messageExtBrokerInner = buildMessage(topic, -1);
+ putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+ }
+
+ @Test
+ public void testSendMessagesToBcqTopic() {
+ String topic = UUID.randomUUID().toString();
+ createTopic(topic, CQType.BatchCQ, messageStore);
+
+ // case 1 has PROPERTY_INNER_NUM but has no INNER_BATCH_FLAG
+ MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, 1);
+ PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
+
+ // case 2 has neither PROPERTY_INNER_NUM nor INNER_BATCH_FLAG.
+ messageExtBrokerInner = buildMessage(topic, -1);
+ putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+
+ // case 3 has INNER_BATCH_FLAG but has no PROPERTY_INNER_NUM.
+ messageExtBrokerInner = buildMessage(topic, 1);
+ MessageAccessor.clearProperty(messageExtBrokerInner, MessageConst.PROPERTY_INNER_NUM);
+ messageExtBrokerInner.setSysFlag(MessageSysFlag.INNER_BATCH_FLAG);
+ putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+ }
+
+ @Test
+ public void testConsumeBatchMessage() {
+ String topic = UUID.randomUUID().toString();
+ createTopic(topic, CQType.BatchCQ, messageStore);
+ int batchNum = 10;
+
+ MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, batchNum);
+ PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
+ List<GetMessageResult> results = new ArrayList<>();
+ for (int i = 0; i < batchNum; i++) {
+ GetMessageResult result = messageStore.getMessage("whatever", topic, 0, i, Integer.MAX_VALUE, Integer.MAX_VALUE, null);
+ try {
+ Assert.assertEquals(GetMessageStatus.FOUND, result.getStatus());
+ results.add(result);
+ } finally {
+ result.release();
+ }
+ }
+
+ for (GetMessageResult result : results) {
+ Assert.assertEquals(0, result.getMinOffset());
+ Assert.assertEquals(batchNum, result.getMaxOffset());
+ }
+
+ }
+
+ @Test
+ public void testNextBeginOffsetConsumeBatchMessage() {
+ String topic = UUID.randomUUID().toString();
+ createTopic(topic, CQType.BatchCQ, messageStore);
+ Random random = new Random();
+ int putMessageCount = 1000;
+
+ Queue<Integer> queue = new ArrayDeque<>();
+ for (int i = 0; i < putMessageCount; i++) {
+ int batchNum = random.nextInt(1000) + 2;
+ MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, batchNum);
+ PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
+ Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
+ queue.add(batchNum);
+ }
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
+ long pullOffset = 0L;
+ int getMessageCount = 0;
+ while (true) {
+ GetMessageResult getMessageResult = messageStore.getMessage("group", topic, 0, pullOffset, 1, null);
+ if (Objects.equals(getMessageResult.getStatus(), GetMessageStatus.OFFSET_OVERFLOW_ONE)) {
+ break;
+ }
+ Assert.assertEquals(1, getMessageResult.getMessageQueueOffset().size());
+ Long baseOffset = getMessageResult.getMessageQueueOffset().get(0);
+ Integer batchNum = queue.poll();
+ Assert.assertNotNull(batchNum);
+ Assert.assertEquals(baseOffset + batchNum, getMessageResult.getNextBeginOffset());
+ pullOffset = getMessageResult.getNextBeginOffset();
+ getMessageCount++;
+ }
+ Assert.assertEquals(putMessageCount, getMessageCount);
+ }
+
+ @Test
+ public void testGetOffsetInQueueByTime() throws Exception {
String topic = "testGetOffsetInQueueByTime";
- //The initial min max offset, before and after the creation of consume queue
+ createTopic(topic, CQType.BatchCQ, messageStore);
+ Assert.assertTrue(QueueTypeUtils.isBatchCq(messageStore.getTopicConfig(topic)));
+
+ // The initial min max offset, before and after the creation of consume queue
Assert.assertEquals(0, messageStore.getMaxOffsetInQueue(topic, 0));
Assert.assertEquals(-1, messageStore.getMinOffsetInQueue(topic, 0));
@@ -54,34 +209,30 @@ public class BatchConsumeMessageTest extends QueueTestBase {
if (i == 7)
timeMid = System.currentTimeMillis();
}
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
Assert.assertEquals(80, messageStore.getOffsetInQueueByTime(topic, 0, timeMid));
+ Assert.assertEquals(0, messageStore.getMinOffsetInQueue(topic, 0));
+ Assert.assertEquals(190, messageStore.getMaxOffsetInQueue(topic, 0));
+
Thread.sleep(5 * 1000);
int maxBatchDeleteFilesNum = messageStore.getMessageStoreConfig().getMaxBatchDeleteFilesNum();
messageStore.getCommitLog().deleteExpiredFile(1L, 100, 12000, true, maxBatchDeleteFilesNum);
+ Assert.assertEquals(80, messageStore.getOffsetInQueueByTime(topic, 0, timeMid));
Thread.sleep(70 * 1000);
Assert.assertEquals(180, messageStore.getOffsetInQueueByTime(topic, 0, timeMid));
-
}
@Test
public void testDispatchNormalConsumeQueue() throws Exception {
- MessageStore messageStore = createMessageStore(null, true, CQType.SimpleCQ);
- messageStore.load();
- messageStore.start();
String topic = "TestDispatchBuildConsumeQueue";
- int batchNum = 10;
+ createTopic(topic, CQType.SimpleCQ, messageStore);
+
long timeStart = System.currentTimeMillis();
long timeMid = -1;
- for (int i = 0; i < 100; i++) {
-// MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, batchNum);
-// messageExtBrokerInner.setSysFlag(0);
-// PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
-// Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
-//
-// messageExtBrokerInner = buildMessage(topic, 1);
-// putMessageResult = messageStore.putMessage(messageExtBrokerInner);
-// Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
+ for (int i = 0; i < 100; i++) {
MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, -1);
PutMessageResult putMessageResult = messageStore.putMessage(messageExtBrokerInner);
Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
@@ -90,7 +241,9 @@ public class BatchConsumeMessageTest extends QueueTestBase {
if (i == 49)
timeMid = System.currentTimeMillis();
}
- Thread.sleep(500);
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
ConsumeQueueInterface consumeQueue = messageStore.getConsumeQueue(topic, 0);
Assert.assertEquals(CQType.SimpleCQ, consumeQueue.getCQType());
//check the consume queue
@@ -106,43 +259,41 @@ public class BatchConsumeMessageTest extends QueueTestBase {
for (int i = -100; i < 100; i += 20) {
Assert.assertEquals(consumeQueue.getOffsetInQueueByTime(timeMid + i), messageStore.getOffsetInQueueByTime(topic, 0, timeMid + i));
}
+
//check the message time
+ long latencyAllowed = 20;
long earlistMessageTime = messageStore.getEarliestMessageTime(topic, 0);
- Assert.assertTrue(earlistMessageTime > timeStart - 10);
- Assert.assertTrue(earlistMessageTime < timeStart + 10);
+ Assert.assertTrue(earlistMessageTime > timeStart - latencyAllowed);
+ Assert.assertTrue(earlistMessageTime < timeStart + latencyAllowed);
long messageStoreTime = messageStore.getMessageStoreTimeStamp(topic, 0, 50);
- Assert.assertTrue(messageStoreTime > timeMid - 10);
- Assert.assertTrue(messageStoreTime < timeMid + 10);
+ Assert.assertTrue(messageStoreTime > timeMid - latencyAllowed);
+ Assert.assertTrue(messageStoreTime < timeMid + latencyAllowed);
long commitLogOffset = messageStore.getCommitLogOffsetInQueue(topic, 0, 50);
Assert.assertTrue(commitLogOffset >= messageStore.getMinPhyOffset());
Assert.assertTrue(commitLogOffset <= messageStore.getMaxPhyOffset());
Assert.assertFalse(messageStore.checkInDiskByConsumeOffset(topic, 0, 50));
- messageStore.shutdown();
- messageStore.destroy();
}
@Test
public void testDispatchBuildBatchConsumeQueue() throws Exception {
- MessageStore messageStore = createMessageStore(null, true, CQType.BatchCQ);
- messageStore.load();
- messageStore.start();
String topic = "testDispatchBuildBatchConsumeQueue";
int batchNum = 10;
long timeStart = System.currentTimeMillis();
long timeMid = -1;
+
+ createTopic(topic, CQType.BatchCQ, messageStore);
+
for (int i = 0; i < 100; i++) {
PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(topic, batchNum));
Assert.assertEquals(PutMessageStatus.PUT_OK, putMessageResult.getPutMessageStatus());
Thread.sleep(2);
if (i == 29)
timeMid = System.currentTimeMillis();
-
- MessageExtBrokerInner messageExtBrokerInner = buildMessage(topic, 1);
- putMessageResult = messageStore.putMessage(messageExtBrokerInner);
- Assert.assertEquals(PutMessageStatus.MESSAGE_ILLEGAL, putMessageResult.getPutMessageStatus());
}
- Thread.sleep(500);
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
ConsumeQueueInterface consumeQueue = messageStore.getConsumeQueue(topic, 0);
Assert.assertEquals(CQType.BatchCQ, consumeQueue.getCQType());
@@ -176,21 +327,18 @@ public class BatchConsumeMessageTest extends QueueTestBase {
for (int i = 0; i < 10; i++) {
SelectMappedBufferResult sbr = getMessageResult.getMessageMapedList().get(i);
MessageExt messageExt = MessageDecoder.decode(sbr.getByteBuffer());
- short tmpBatchNum = Short.valueOf(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ short tmpBatchNum = Short.parseShort(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
Assert.assertEquals(i * batchNum, Long.parseLong(messageExt.getProperty(MessageConst.PROPERTY_INNER_BASE)));
Assert.assertEquals(batchNum, tmpBatchNum);
}
-
- messageStore.destroy();
- messageStore.shutdown();
}
@Test
- public void testGetBatchMessageWithinNumber() throws Exception {
- MessageStore messageStore = createMessageStore(null, true, CQType.BatchCQ);
- messageStore.load();
- messageStore.start();
+ public void testGetBatchMessageWithinNumber() {
String topic = UUID.randomUUID().toString();
+
+ createTopic(topic, CQType.BatchCQ, messageStore);
+
int batchNum = 20;
for (int i = 0; i < 200; i++) {
PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(topic, batchNum));
@@ -198,7 +346,9 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertEquals(i * batchNum, putMessageResult.getAppendMessageResult().getLogicsOffset());
Assert.assertEquals(batchNum, putMessageResult.getAppendMessageResult().getMsgNum());
}
- Thread.sleep(500);
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
ConsumeQueueInterface consumeQueue = messageStore.getConsumeQueue(topic, 0);
Assert.assertEquals(CQType.BatchCQ, consumeQueue.getCQType());
Assert.assertEquals(0, consumeQueue.getMinOffsetInQueue());
@@ -212,7 +362,7 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertEquals(batchNum, getMessageResult.getMessageCount());
SelectMappedBufferResult sbr = getMessageResult.getMessageMapedList().get(0);
MessageExt messageExt = MessageDecoder.decode(sbr.getByteBuffer());
- short tmpBatchNum = Short.valueOf(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ short tmpBatchNum = Short.parseShort(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
Assert.assertEquals(0, messageExt.getQueueOffset());
Assert.assertEquals(batchNum, tmpBatchNum);
}
@@ -236,22 +386,19 @@ public class BatchConsumeMessageTest extends QueueTestBase {
SelectMappedBufferResult sbr = getMessageResult.getMessageMapedList().get(i);
MessageExt messageExt = MessageDecoder.decode(sbr.getByteBuffer());
Assert.assertNotNull(messageExt);
- short innerBatchNum = Short.valueOf(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ short innerBatchNum = Short.parseShort(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
Assert.assertEquals(i * batchNum, Long.parseLong(messageExt.getProperty(MessageConst.PROPERTY_INNER_BASE)));
Assert.assertEquals(batchNum, innerBatchNum);
}
}
- messageStore.destroy();
- messageStore.shutdown();
}
@Test
- public void testGetBatchMessageWithinSize() throws Exception {
- MessageStore messageStore = createMessageStore(null, true, CQType.BatchCQ);
- messageStore.load();
- messageStore.start();
+ public void testGetBatchMessageWithinSize() {
String topic = UUID.randomUUID().toString();
+ createTopic(topic, CQType.BatchCQ, messageStore);
+
int batchNum = 10;
for (int i = 0; i < 100; i++) {
PutMessageResult putMessageResult = messageStore.putMessage(buildMessage(topic, batchNum));
@@ -259,7 +406,9 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertEquals(i * 10, putMessageResult.getAppendMessageResult().getLogicsOffset());
Assert.assertEquals(batchNum, putMessageResult.getAppendMessageResult().getMsgNum());
}
- Thread.sleep(500);
+
+ await().atMost(5, SECONDS).until(fullyDispatched(messageStore));
+
ConsumeQueueInterface consumeQueue = messageStore.getConsumeQueue(topic, 0);
Assert.assertEquals(CQType.BatchCQ, consumeQueue.getCQType());
Assert.assertEquals(0, consumeQueue.getMinOffsetInQueue());
@@ -272,7 +421,7 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertEquals(1, getMessageResult.getMessageMapedList().size());
SelectMappedBufferResult sbr = getMessageResult.getMessageMapedList().get(0);
MessageExt messageExt = MessageDecoder.decode(sbr.getByteBuffer());
- short tmpBatchNum = Short.valueOf(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ short tmpBatchNum = Short.parseShort(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
Assert.assertEquals(0, messageExt.getQueueOffset());
Assert.assertEquals(batchNum, tmpBatchNum);
}
@@ -293,14 +442,29 @@ public class BatchConsumeMessageTest extends QueueTestBase {
Assert.assertFalse(getMessageResult.getMessageMapedList().get(i).hasReleased());
SelectMappedBufferResult sbr = getMessageResult.getMessageMapedList().get(i);
MessageExt messageExt = MessageDecoder.decode(sbr.getByteBuffer());
- short tmpBatchNum = Short.valueOf(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
+ short tmpBatchNum = Short.parseShort(messageExt.getProperty(MessageConst.PROPERTY_INNER_NUM));
Assert.assertEquals(i * batchNum, Long.parseLong(messageExt.getProperty(MessageConst.PROPERTY_INNER_BASE)));
Assert.assertEquals(batchNum, tmpBatchNum);
}
}
- messageStore.destroy();
- messageStore.shutdown();
+ }
+
+ private void createTopic(String topic, CQType cqType, MessageStore messageStore) {
+ ConcurrentMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<>();
+ TopicConfig topicConfigToBeAdded = new TopicConfig();
+
+ Map<String, String> attributes = new HashMap<>();
+ attributes.put(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName(), cqType.toString());
+ topicConfigToBeAdded.setTopicName(topic);
+ topicConfigToBeAdded.setAttributes(attributes);
+
+ topicConfigTable.put(topic, topicConfigToBeAdded);
+ ((DefaultMessageStore)messageStore).setTopicConfigTable(topicConfigTable);
+ }
+
+ private Callable<Boolean> fullyDispatched(MessageStore messageStore) {
+ return () -> messageStore.dispatchBehindBytes() == 0;
}
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeQueueTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeQueueTest.java
index 8395d7d..1d9addc 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeQueueTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/BatchConsumeQueueTest.java
@@ -19,8 +19,8 @@ package org.apache.rocketmq.store.queue;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.store.ConsumeQueue;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.MessageArrivingListener;
+import org.apache.rocketmq.store.DefaultMessageStore;
+import org.apache.rocketmq.store.MessageStore;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import org.apache.rocketmq.store.StoreTestBase;
import org.apache.rocketmq.store.config.MessageStoreConfig;
@@ -32,7 +32,6 @@ import org.junit.Test;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
-import java.util.Map;
import java.util.Random;
import static java.lang.String.format;
@@ -45,13 +44,13 @@ public class BatchConsumeQueueTest extends StoreTestBase {
path = createBaseDir();
}
baseDirs.add(path);
- StreamMessageStore bcqMessageStore = null;
+ MessageStore messageStore = null;
try {
- bcqMessageStore = createBcqMessageStore(null);
+ messageStore = createMessageStore(null);
} catch (Exception e) {
Assert.fail();
}
- BatchConsumeQueue batchConsumeQueue = new BatchConsumeQueue("topic", 0, path,fileSize, bcqMessageStore);
+ BatchConsumeQueue batchConsumeQueue = new BatchConsumeQueue("topic", 0, path,fileSize, messageStore);
batchConsumeQueues.add(batchConsumeQueue);
return batchConsumeQueue;
}
@@ -280,7 +279,7 @@ public class BatchConsumeQueueTest extends StoreTestBase {
}
}
- protected StreamMessageStore createBcqMessageStore(String baseDir) throws Exception {
+ protected MessageStore createMessageStore(String baseDir) throws Exception {
if (baseDir == null) {
baseDir = createBaseDir();
}
@@ -299,17 +298,13 @@ public class BatchConsumeQueueTest extends StoreTestBase {
messageStoreConfig.setMaxTransferBytesOnMessageInMemory(1024 * 1024);
messageStoreConfig.setMaxTransferCountOnMessageInDisk(1024);
messageStoreConfig.setMaxTransferCountOnMessageInMemory(1024);
- messageStoreConfig.setDefaultCQType(CQType.BatchCQ.name());
messageStoreConfig.setSearchBcqByCacheEnable(true);
- StreamMessageStore defaultMessageStore = new StreamMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MessageArrivingListener() {
- @Override
- public void arriving(String topic, int queueId, long logicOffset, long tagsCode, long msgStoreTime,
- byte[] filterBitMap, Map<String, String> properties) {
-
- }
- }, new BrokerConfig());
- return defaultMessageStore;
+ return new DefaultMessageStore(
+ messageStoreConfig,
+ new BrokerStatsManager("simpleTest"),
+ (topic, queueId, logicOffset, tagsCode, msgStoreTime, filterBitMap, properties) -> {},
+ new BrokerConfig());
}
}
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
index 4345f0e..f274201 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/ConsumeQueueTest.java
@@ -16,6 +16,7 @@
*/
package org.apache.rocketmq.store.queue;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.store.ConsumeQueueExt;
import org.apache.rocketmq.store.DispatchRequest;
import org.apache.rocketmq.store.MessageStore;
@@ -30,7 +31,7 @@ public class ConsumeQueueTest extends QueueTestBase {
public void testIterator() throws Exception {
final int msgNum = 100;
final int msgSize = 1000;
- MessageStore messageStore = createMessageStore(null, true, CQType.SimpleCQ);
+ MessageStore messageStore = createMessageStore(null, true);
messageStore.load();
String topic = UUID.randomUUID().toString();
//The initial min max offset, before and after the creation of consume queue
diff --git a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
index a8e9f94..c16d7cb 100644
--- a/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
+++ b/store/src/test/java/org/apache/rocketmq/store/queue/QueueTestBase.java
@@ -23,7 +23,6 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
import org.apache.rocketmq.store.MessageArrivingListener;
import org.apache.rocketmq.store.MessageExtBrokerInner;
import org.apache.rocketmq.store.MessageStore;
@@ -33,11 +32,10 @@ import org.apache.rocketmq.store.stats.BrokerStatsManager;
import java.io.File;
import java.util.Map;
-import java.util.Objects;
public class QueueTestBase extends StoreTestBase {
- protected MessageStore createMessageStore(String baseDir, boolean extent, CQType cqType) throws Exception {
+ protected MessageStore createMessageStore(String baseDir, boolean extent) throws Exception {
if (baseDir == null) {
baseDir = createBaseDir();
}
@@ -56,28 +54,18 @@ public class QueueTestBase extends StoreTestBase {
messageStoreConfig.setMaxTransferBytesOnMessageInMemory(1024 * 1024);
messageStoreConfig.setMaxTransferCountOnMessageInDisk(1024);
messageStoreConfig.setMaxTransferCountOnMessageInMemory(1024);
- messageStoreConfig.setDefaultCQType(cqType.name());
messageStoreConfig.setFlushIntervalCommitLog(1);
messageStoreConfig.setFlushCommitLogThoroughInterval(2);
- if (Objects.equals(CQType.BatchCQ, cqType)) {
- return new StreamMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MessageArrivingListener() {
- @Override
- public void arriving(String topic, int queueId, long logicOffset, long tagsCode, long msgStoreTime,
- byte[] filterBitMap, Map<String, String> properties) {
+ DefaultMessageStore messageStore = new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MessageArrivingListener() {
+ @Override
+ public void arriving(String topic, int queueId, long logicOffset, long tagsCode, long msgStoreTime,
+ byte[] filterBitMap, Map<String, String> properties) {
- }
- }, new BrokerConfig());
- } else {
- return new DefaultMessageStore(messageStoreConfig, new BrokerStatsManager("simpleTest"), new MessageArrivingListener() {
- @Override
- public void arriving(String topic, int queueId, long logicOffset, long tagsCode, long msgStoreTime,
- byte[] filterBitMap, Map<String, String> properties) {
-
- }
- }, new BrokerConfig());
- }
+ }
+ }, new BrokerConfig());
+ return messageStore;
}
public MessageExtBrokerInner buildMessage(String topic, int batchNum) {
diff --git a/test/src/main/java/org/apache/rocketmq/test/util/MQAdminTestUtils.java b/test/src/main/java/org/apache/rocketmq/test/util/MQAdminTestUtils.java
index 8287d81..20149d4 100644
--- a/test/src/main/java/org/apache/rocketmq/test/util/MQAdminTestUtils.java
+++ b/test/src/main/java/org/apache/rocketmq/test/util/MQAdminTestUtils.java
@@ -50,20 +50,20 @@ public class MQAdminTestUtils {
private static Logger log = Logger.getLogger(MQAdminTestUtils.class);
public static boolean createTopic(String nameSrvAddr, String clusterName, String topic,
- int queueNum) {
+ int queueNum, Map<String, String> attributes) {
int defaultWaitTime = 5;
- return createTopic(nameSrvAddr, clusterName, topic, queueNum, defaultWaitTime);
+ return createTopic(nameSrvAddr, clusterName, topic, queueNum, attributes, defaultWaitTime);
}
public static boolean createTopic(String nameSrvAddr, String clusterName, String topic,
- int queueNum, int waitTimeSec) {
+ int queueNum, Map<String, String> attributes, int waitTimeSec) {
boolean createResult = false;
DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
mqAdminExt.setInstanceName(UUID.randomUUID().toString());
mqAdminExt.setNamesrvAddr(nameSrvAddr);
try {
mqAdminExt.start();
- mqAdminExt.createTopic(clusterName, topic, queueNum);
+ mqAdminExt.createTopic(clusterName, topic, queueNum, attributes);
} catch (Exception e) {
}
diff --git a/test/src/test/java/org/apache/rocketmq/test/base/BaseConf.java b/test/src/test/java/org/apache/rocketmq/test/base/BaseConf.java
index a4e07fb..32af3bd 100644
--- a/test/src/test/java/org/apache/rocketmq/test/base/BaseConf.java
+++ b/test/src/test/java/org/apache/rocketmq/test/base/BaseConf.java
@@ -37,7 +37,7 @@ import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.protocol.route.BrokerData;
import org.apache.rocketmq.namesrv.NamesrvController;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.test.client.rmq.RMQAsyncSendProducer;
import org.apache.rocketmq.test.client.rmq.RMQNormalConsumer;
import org.apache.rocketmq.test.client.rmq.RMQNormalProducer;
@@ -61,18 +61,14 @@ public class BaseConf {
//the logic queue test need at least three brokers
protected final static String broker3Name;
protected final static String clusterName;
- protected final static String steamClusterName = "steam-cluster";
protected final static int brokerNum;
protected final static int waitTime = 5;
protected final static int consumeTime = 2 * 60 * 1000;
protected final static int QUEUE_NUMBERS = 8;
protected final static NamesrvController namesrvController;
- protected static BrokerController brokerController1;
- protected static BrokerController brokerController2;
- protected static BrokerController brokerController3;
- protected static BrokerController streamBrokerController1;
- protected static BrokerController streamBrokerController2;
- protected static BrokerController streamBrokerController3;
+ protected final static BrokerController brokerController1;
+ protected final static BrokerController brokerController2;
+ protected final static BrokerController brokerController3;
protected final static List<BrokerController> brokerControllerList;
protected final static Map<String, BrokerController> brokerControllerMap;
protected final static List<Object> mqClients = new ArrayList<Object>();
@@ -80,7 +76,7 @@ public class BaseConf {
private final static Logger log = Logger.getLogger(BaseConf.class);
static {
- System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
+ System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
namesrvController = IntegrationTestBase.createAndStartNamesrv();
nsAddr = "127.0.0.1:" + namesrvController.getNettyServerConfig().getListenPort();
brokerController1 = IntegrationTestBase.createAndStartBroker(nsAddr);
@@ -91,9 +87,6 @@ public class BaseConf {
broker2Name = brokerController2.getBrokerConfig().getBrokerName();
broker3Name = brokerController3.getBrokerConfig().getBrokerName();
brokerNum = 3;
- streamBrokerController1 = IntegrationTestBase.createAndStartBroker(nsAddr, CQType.BatchCQ.toString(), steamClusterName);
- streamBrokerController2 = IntegrationTestBase.createAndStartBroker(nsAddr, CQType.BatchCQ.toString(), steamClusterName);
- streamBrokerController3 = IntegrationTestBase.createAndStartBroker(nsAddr, CQType.BatchCQ.toString(), steamClusterName);
brokerControllerList = ImmutableList.of(brokerController1, brokerController2, brokerController3);
brokerControllerMap = brokerControllerList.stream().collect(Collectors.toMap(input -> input.getBrokerConfig().getBrokerName(), Function.identity()));
}
@@ -124,7 +117,7 @@ public class BaseConf {
public static String initTopic() {
String topic = "tt-" + MQRandomUtils.getRandomTopic();
- IntegrationTestBase.initTopic(topic, nsAddr, clusterName);
+ IntegrationTestBase.initTopic(topic, nsAddr, clusterName, CQType.SimpleCQ);
return topic;
}
@@ -171,9 +164,9 @@ public class BaseConf {
}
public static RMQNormalProducer getProducer(String nsAddr, String topic, String producerGoup,
- String instanceName) {
+ String instanceName) {
RMQNormalProducer producer = new RMQNormalProducer(nsAddr, topic, producerGoup,
- instanceName);
+ instanceName);
if (debug) {
producer.setDebug();
}
@@ -191,31 +184,31 @@ public class BaseConf {
}
public static RMQNormalConsumer getConsumer(String nsAddr, String topic, String subExpression,
- AbstractListener listener) {
+ AbstractListener listener) {
return getConsumer(nsAddr, topic, subExpression, listener, false);
}
public static RMQNormalConsumer getConsumer(String nsAddr, String topic, String subExpression,
- AbstractListener listener, boolean useTLS) {
+ AbstractListener listener, boolean useTLS) {
String consumerGroup = initConsumerGroup();
return getConsumer(nsAddr, consumerGroup, topic, subExpression, listener, useTLS);
}
public static RMQNormalConsumer getConsumer(String nsAddr, String consumerGroup, String topic,
- String subExpression, AbstractListener listener) {
+ String subExpression, AbstractListener listener) {
return getConsumer(nsAddr, consumerGroup, topic, subExpression, listener, false);
}
public static RMQNormalConsumer getConsumer(String nsAddr, String consumerGroup, String topic,
- String subExpression, AbstractListener listener, boolean useTLS) {
+ String subExpression, AbstractListener listener, boolean useTLS) {
RMQNormalConsumer consumer = ConsumerFactory.getRMQNormalConsumer(nsAddr, consumerGroup,
- topic, subExpression, listener, useTLS);
+ topic, subExpression, listener, useTLS);
if (debug) {
consumer.setDebug();
}
mqClients.add(consumer);
log.info(String.format("consumer[%s] start,topic[%s],subExpression[%s]", consumerGroup,
- topic, subExpression));
+ topic, subExpression));
return consumer;
}
diff --git a/test/src/test/java/org/apache/rocketmq/test/base/IntegrationTestBase.java b/test/src/test/java/org/apache/rocketmq/test/base/IntegrationTestBase.java
index a74fd21..398bd19 100644
--- a/test/src/test/java/org/apache/rocketmq/test/base/IntegrationTestBase.java
+++ b/test/src/test/java/org/apache/rocketmq/test/base/IntegrationTestBase.java
@@ -20,13 +20,18 @@ package org.apache.rocketmq.test.base;
import com.google.common.truth.Truth;
import java.io.File;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.Objects;
import java.util.Random;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.UtilAll;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.namesrv.NamesrvConfig;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
@@ -34,10 +39,8 @@ import org.apache.rocketmq.namesrv.NamesrvController;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.apache.rocketmq.store.queue.CQType;
import org.apache.rocketmq.test.util.MQAdminTestUtils;
import org.apache.rocketmq.test.util.TestUtils;
-import org.assertj.core.util.Strings;
public class IntegrationTestBase {
public static InternalLogger logger = InternalLoggerFactory.getLogger(IntegrationTestBase.class);
@@ -142,34 +145,6 @@ public class IntegrationTestBase {
storeConfig.setMaxIndexNum(INDEX_NUM);
storeConfig.setMaxHashSlotNum(INDEX_NUM * 4);
storeConfig.setDeleteWhen("01;02;03;04;05;06;07;08;09;10;11;12;13;14;15;16;17;18;19;20;21;22;23;00");
- storeConfig.setDefaultCQType(CQType.SimpleCQ.toString());
- storeConfig.setMaxTransferCountOnMessageInMemory(1024);
- storeConfig.setMaxTransferCountOnMessageInDisk(1024);
- return createAndStartBroker(storeConfig, brokerConfig);
- }
-
- public static BrokerController createAndStartBroker(String nsAddr, String cqType, String cluster) {
- String baseDir = createBaseDir();
- BrokerConfig brokerConfig = new BrokerConfig();
- MessageStoreConfig storeConfig = new MessageStoreConfig();
- brokerConfig.setBrokerName(BROKER_NAME_PREFIX + BROKER_INDEX.incrementAndGet());
- brokerConfig.setBrokerIP1("127.0.0.1");
- brokerConfig.setNamesrvAddr(nsAddr);
- brokerConfig.setEnablePropertyFilter(true);
- brokerConfig.setLoadBalancePollNameServerInterval(500);
- brokerConfig.setBrokerClusterName(cluster);
- storeConfig.setStorePathRootDir(baseDir);
- storeConfig.setStorePathCommitLog(baseDir + SEP + "commitlog");
- storeConfig.setMappedFileSizeCommitLog(COMMIT_LOG_SIZE);
- storeConfig.setMaxIndexNum(INDEX_NUM);
- storeConfig.setMaxHashSlotNum(INDEX_NUM * 4);
- storeConfig.setDeleteWhen("01;02;03;04;05;06;07;08;09;10;11;12;13;14;15;16;17;18;19;20;21;22;23;00");
-
- if (!Strings.isNullOrEmpty(cqType)) {
- storeConfig.setDefaultCQType(cqType);
- } else {
- storeConfig.setDefaultCQType(CQType.SimpleCQ.toString());
- }
storeConfig.setMaxTransferCountOnMessageInMemory(1024);
storeConfig.setMaxTransferCountOnMessageInDisk(1024);
return createAndStartBroker(storeConfig, brokerConfig);
@@ -193,17 +168,21 @@ public class IntegrationTestBase {
return brokerController;
}
- public static boolean initTopic(String topic, String nsAddr, String clusterName, int queueNumbers) {
+ public static boolean initTopic(String topic, String nsAddr, String clusterName, int queueNumbers, CQType cqType) {
long startTime = System.currentTimeMillis();
boolean createResult;
while (true) {
- createResult = MQAdminTestUtils.createTopic(nsAddr, clusterName, topic, queueNumbers);
+ Map<String, String> attributes = new HashMap<>();
+ if (!Objects.equals(CQType.SimpleCQ, cqType)) {
+ attributes.put("+" + TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName(), cqType.toString());
+ }
+ createResult = MQAdminTestUtils.createTopic(nsAddr, clusterName, topic, queueNumbers, attributes);
if (createResult) {
break;
} else if (System.currentTimeMillis() - startTime > topicCreateTime) {
Truth.assertWithMessage(String.format("topic[%s] is created failed after:%d ms", topic,
- System.currentTimeMillis() - startTime)).fail();
+ System.currentTimeMillis() - startTime)).fail();
break;
} else {
TestUtils.waitForMoment(500);
@@ -214,8 +193,8 @@ public class IntegrationTestBase {
return createResult;
}
- public static boolean initTopic(String topic, String nsAddr, String clusterName) {
- return initTopic(topic, nsAddr, clusterName, BaseConf.QUEUE_NUMBERS);
+ public static boolean initTopic(String topic, String nsAddr, String clusterName, CQType cqType) {
+ return initTopic(topic, nsAddr, clusterName, BaseConf.QUEUE_NUMBERS, cqType);
}
public static void deleteFile(File file) {
diff --git a/test/src/test/java/org/apache/rocketmq/test/base/dledger/DLedgerProduceAndConsumeIT.java b/test/src/test/java/org/apache/rocketmq/test/base/dledger/DLedgerProduceAndConsumeIT.java
index dd71dd1..0e1bf26 100644
--- a/test/src/test/java/org/apache/rocketmq/test/base/dledger/DLedgerProduceAndConsumeIT.java
+++ b/test/src/test/java/org/apache/rocketmq/test/base/dledger/DLedgerProduceAndConsumeIT.java
@@ -30,7 +30,7 @@ import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.store.config.BrokerRole;
import org.apache.rocketmq.store.config.MessageStoreConfig;
-import org.apache.rocketmq.store.queue.CQType;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.test.base.BaseConf;
import org.apache.rocketmq.test.base.IntegrationTestBase;
import org.apache.rocketmq.test.factory.ConsumerFactory;
@@ -63,7 +63,6 @@ public class DLedgerProduceAndConsumeIT {
storeConfig.setdLegerGroup(brokerName);
storeConfig.setdLegerSelfId(selfId);
storeConfig.setdLegerPeers(peers);
- storeConfig.setDefaultCQType(CQType.SimpleCQ.toString());
return storeConfig;
}
@@ -83,7 +82,7 @@ public class DLedgerProduceAndConsumeIT {
String topic = UUID.randomUUID().toString();
String consumerGroup = UUID.randomUUID().toString();
- IntegrationTestBase.initTopic(topic, BaseConf.nsAddr, cluster, 1);
+ IntegrationTestBase.initTopic(topic, BaseConf.nsAddr, cluster, 1, CQType.SimpleCQ);
DefaultMQProducer producer = ProducerFactory.getRMQProducer(BaseConf.nsAddr);
DefaultMQPullConsumer consumer = ConsumerFactory.getRMQPullConsumer(BaseConf.nsAddr, consumerGroup);
diff --git a/test/src/test/java/org/apache/rocketmq/test/client/producer/batch/BatchSendIT.java b/test/src/test/java/org/apache/rocketmq/test/client/producer/batch/BatchSendIT.java
index bf0b9e5..3a649ed 100644
--- a/test/src/test/java/org/apache/rocketmq/test/client/producer/batch/BatchSendIT.java
+++ b/test/src/test/java/org/apache/rocketmq/test/client/producer/batch/BatchSendIT.java
@@ -23,7 +23,6 @@ import java.util.Random;
import java.util.UUID;
import org.apache.log4j.Logger;
-import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.client.consumer.DefaultMQPullConsumer;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
@@ -32,14 +31,14 @@ import org.apache.rocketmq.client.hook.SendMessageHook;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.client.producer.SendStatus;
+import org.apache.rocketmq.common.TopicAttributes;
+import org.apache.rocketmq.common.attribute.CQType;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageBatch;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.store.DefaultMessageStore;
-import org.apache.rocketmq.store.StreamMessageStore;
-import org.apache.rocketmq.store.queue.CQType;
import org.apache.rocketmq.test.base.BaseConf;
import org.apache.rocketmq.test.base.IntegrationTestBase;
import org.apache.rocketmq.test.client.consumer.tag.TagMessageWith1ConsumerIT;
@@ -101,23 +100,23 @@ public class BatchSendIT extends BaseConf {
@Test
public void testBatchSend_SysInnerBatch() throws Exception {
- waitBrokerRegistered(nsAddr, steamClusterName, brokerNum);
-
- Assert.assertTrue(streamBrokerController1.getMessageStore() instanceof StreamMessageStore);
- Assert.assertTrue(streamBrokerController2.getMessageStore() instanceof StreamMessageStore);
- Assert.assertTrue(streamBrokerController3.getMessageStore() instanceof StreamMessageStore);
+ waitBrokerRegistered(nsAddr, clusterName, brokerNum);
String batchTopic = UUID.randomUUID().toString();
- IntegrationTestBase.initTopic(batchTopic, nsAddr, steamClusterName);
- Assert.assertEquals(8, streamBrokerController1.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
- Assert.assertEquals(8, streamBrokerController2.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
- Assert.assertEquals(8, streamBrokerController3.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
- Assert.assertEquals(-1, streamBrokerController1.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
- Assert.assertEquals(-1, streamBrokerController2.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
- Assert.assertEquals(-1, streamBrokerController3.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
- Assert.assertEquals(0, streamBrokerController1.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
- Assert.assertEquals(0, streamBrokerController2.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
- Assert.assertEquals(0, streamBrokerController3.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
+ IntegrationTestBase.initTopic(batchTopic, nsAddr, clusterName, CQType.BatchCQ);
+
+ Assert.assertEquals(CQType.BatchCQ.toString(), brokerController1.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getAttributes().get(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName()));
+ Assert.assertEquals(CQType.BatchCQ.toString(), brokerController2.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getAttributes().get(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName()));
+ Assert.assertEquals(CQType.BatchCQ.toString(), brokerController3.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getAttributes().get(TopicAttributes.QUEUE_TYPE_ATTRIBUTE.getName()));
+ Assert.assertEquals(8, brokerController1.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
+ Assert.assertEquals(8, brokerController2.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
+ Assert.assertEquals(8, brokerController3.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
+ Assert.assertEquals(-1, brokerController1.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
+ Assert.assertEquals(-1, brokerController2.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
+ Assert.assertEquals(-1, brokerController3.getMessageStore().getMinOffsetInQueue(batchTopic, 0));
+ Assert.assertEquals(0, brokerController1.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
+ Assert.assertEquals(0, brokerController2.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
+ Assert.assertEquals(0, brokerController3.getMessageStore().getMaxOffsetInQueue(batchTopic, 0));
DefaultMQProducer producer = ProducerFactory.getRMQProducer(nsAddr);
MessageQueue messageQueue = producer.fetchPublishMessageQueues(batchTopic).iterator().next();
@@ -168,7 +167,7 @@ public class BatchSendIT extends BaseConf {
Assert.assertTrue(brokerController3.getMessageStore() instanceof DefaultMessageStore);
String batchTopic = UUID.randomUUID().toString();
- IntegrationTestBase.initTopic(batchTopic, nsAddr, clusterName);
+ IntegrationTestBase.initTopic(batchTopic, nsAddr, clusterName, CQType.SimpleCQ);
Assert.assertEquals(8, brokerController1.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
Assert.assertEquals(8, brokerController2.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
Assert.assertEquals(8, brokerController3.getTopicConfigManager().getTopicConfigTable().get(batchTopic).getReadQueueNums());
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExt.java b/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExt.java
index 6f551d3..378f7db 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExt.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExt.java
@@ -95,13 +95,13 @@ public class DefaultMQAdminExt extends ClientConfig implements MQAdminExt {
}
@Override
- public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, newTopic, queueNum, 0);
+ public void createTopic(String key, String newTopic, int queueNum, Map<String, String> attributes) throws MQClientException {
+ createTopic(key, newTopic, queueNum, 0, attributes);
}
@Override
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
- defaultMQAdminExtImpl.createTopic(key, newTopic, queueNum, topicSysFlag);
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
+ defaultMQAdminExtImpl.createTopic(key, newTopic, queueNum, topicSysFlag, attributes);
}
@Override
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtImpl.java b/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtImpl.java
index b7e4816..f9b4822 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtImpl.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/admin/DefaultMQAdminExtImpl.java
@@ -77,11 +77,8 @@ import org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData;
import org.apache.rocketmq.common.protocol.route.BrokerData;
import org.apache.rocketmq.common.protocol.route.QueueData;
import org.apache.rocketmq.common.protocol.route.TopicRouteData;
-import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
-import org.apache.rocketmq.common.statictopic.TopicQueueMappingOne;
-import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.RPCHook;
@@ -95,9 +92,6 @@ import org.apache.rocketmq.remoting.exception.RemotingTimeoutException;
import org.apache.rocketmq.tools.admin.api.MessageTrack;
import org.apache.rocketmq.tools.admin.api.TrackType;
-import static org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils.checkAndBuildMappingItems;
-import static org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils.getMappingDetailFromConfig;
-
public class DefaultMQAdminExtImpl implements MQAdminExt, MQAdminExtInner {
private static final Set<String> SYSTEM_GROUP_SET = new HashSet<String>();
@@ -1063,13 +1057,13 @@ public class DefaultMQAdminExtImpl implements MQAdminExt, MQAdminExtInner {
}
@Override
- public void createTopic(String key, String newTopic, int queueNum) throws MQClientException {
- createTopic(key, newTopic, queueNum, 0);
+ public void createTopic(String key, String newTopic, int queueNum, Map<String, String> attributes) throws MQClientException {
+ createTopic(key, newTopic, queueNum, 0, null);
}
@Override
- public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
- this.mqClientInstance.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag);
+ public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag, Map<String, String> attributes) throws MQClientException {
+ this.mqClientInstance.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag, attributes);
}
@Override
@@ -1176,7 +1170,7 @@ public class DefaultMQAdminExtImpl implements MQAdminExt, MQAdminExtInner {
@Override
public void setMessageRequestMode(final String brokerAddr, final String topic, final String consumerGroup, final
- MessageRequestMode mode, final int popShareQueueNum, final long timeoutMillis)
+ MessageRequestMode mode, final int popShareQueueNum, final long timeoutMillis)
throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException,
RemotingConnectException, MQClientException {
this.mqClientInstance.getMQClientAPIImpl().setMessageRequestMode(brokerAddr, topic, consumerGroup, mode, popShareQueueNum, timeoutMillis);
diff --git a/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java b/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
index 3caa477..284900b 100644
--- a/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
+++ b/tools/src/main/java/org/apache/rocketmq/tools/command/topic/UpdateTopicSubCommand.java
@@ -23,7 +23,6 @@ import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.OptionGroup;
import org.apache.commons.cli.Options;
-import org.apache.rocketmq.common.TopicAttributes;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.attribute.AttributeParser;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
[rocketmq] 08/17: Convert the consumer offset too
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit b3f9fbd3a72b2bdf54fed315b71f8361b300e107
Author: dongeforever <do...@apache.org>
AuthorDate: Wed Jan 5 17:01:11 2022 +0800
Convert the consumer offset too
---
.../broker/processor/AdminBrokerProcessor.java | 21 +++----
.../broker/processor/ConsumerManageProcessor.java | 68 +++++++++++++++++++++-
.../broker/topic/TopicQueueMappingManager.java | 2 +-
.../apache/rocketmq/common/rpc/RpcClientImpl.java | 25 ++++++++
..._Topic_Logic_Queue_\350\256\276\350\256\241.md" | 9 ++-
5 files changed, 106 insertions(+), 19 deletions(-)
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
index 6505263..568a728 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
@@ -1176,8 +1176,6 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
continue;
}
- TopicQueueMappingDetail mappingDetail = this.brokerController.getTopicQueueMappingManager().getTopicQueueMapping(topic);
-
{
SubscriptionData findSubscriptionData =
this.brokerController.getConsumerManager().findSubscriptionData(requestHeader.getConsumerGroup(), topic);
@@ -1208,26 +1206,21 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
// the consumerOffset cannot be zero for static topic because of the "double read check" strategy
// just remain the logic for dynamic topic
// maybe we should remove it in the future
- if (mappingDetail == null) {
- if (consumerOffset < 0)
- consumerOffset = 0;
- }
+ if (consumerOffset < 0)
+ consumerOffset = 0;
offsetWrapper.setBrokerOffset(brokerOffset);
offsetWrapper.setConsumerOffset(consumerOffset);
// the consumeOffset is not in this broker for static topic
// and may get the wrong result
- if (mappingDetail == null) {
- long timeOffset = consumerOffset - 1;
- if (timeOffset >= 0) {
- long lastTimestamp = this.brokerController.getMessageStore().getMessageStoreTimeStamp(topic, i, timeOffset);
- if (lastTimestamp > 0) {
- offsetWrapper.setLastTimestamp(lastTimestamp);
- }
+ long timeOffset = consumerOffset - 1;
+ if (timeOffset >= 0) {
+ long lastTimestamp = this.brokerController.getMessageStore().getMessageStoreTimeStamp(topic, i, timeOffset);
+ if (lastTimestamp > 0) {
+ offsetWrapper.setLastTimestamp(lastTimestamp);
}
}
-
consumeStats.getOffsetTable().put(mq, offsetWrapper);
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
index 04e705b..a266442 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/processor/ConsumerManageProcessor.java
@@ -29,11 +29,15 @@ import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetRequestHead
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetResponseHeader;
+import org.apache.rocketmq.common.rpc.RpcClientUtils;
import org.apache.rocketmq.common.rpc.RpcRequest;
import org.apache.rocketmq.common.rpc.RpcResponse;
+import org.apache.rocketmq.common.rpc.TopicQueueRequestHeader;
import org.apache.rocketmq.common.statictopic.LogicQueueMappingItem;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingContext;
import org.apache.rocketmq.common.statictopic.TopicQueueMappingDetail;
+import org.apache.rocketmq.common.statictopic.TopicQueueMappingUtils;
+import org.apache.rocketmq.common.sysflag.PullSysFlag;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.common.RemotingHelper;
@@ -110,6 +114,37 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
return response;
}
+ public RemotingCommand rewriteRequestForStaticTopic(final UpdateConsumerOffsetRequestHeader requestHeader, final TopicQueueMappingContext mappingContext) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ TopicQueueMappingDetail mappingDetail = mappingContext.getMappingDetail();
+ if (!mappingContext.isLeader()) {
+ return buildErrorResponse(ResponseCode.NOT_LEADER_FOR_QUEUE, String.format("%s-%d does not exit in request process of current broker %s", requestHeader.getTopic(), requestHeader.getQueueId(), mappingDetail.getBname()));
+ }
+ Long globalOffset = requestHeader.getCommitOffset();
+ LogicQueueMappingItem mappingItem = TopicQueueMappingUtils.findLogicQueueMappingItem(mappingContext.getMappingItemList(), globalOffset, true);
+ requestHeader.setQueueId(mappingItem.getQueueId());
+ requestHeader.setLo(false);
+ requestHeader.setBname(mappingItem.getBname());
+ requestHeader.setCommitOffset(mappingItem.computePhysicalQueueOffset(globalOffset));
+ //leader, let it go, do not need to rewrite the response
+ if (mappingDetail.getBname().equals(mappingItem.getBname())) {
+ return null;
+ }
+ RpcRequest rpcRequest = new RpcRequest(RequestCode.UPDATE_CONSUMER_OFFSET, requestHeader, null);
+ RpcResponse rpcResponse = this.brokerController.getBrokerOuterAPI().getRpcClient().invoke(rpcRequest, this.brokerController.getBrokerConfig().getForwardTimeout()).get();
+ if (rpcResponse.getException() != null) {
+ throw rpcResponse.getException();
+ }
+ return RpcClientUtils.createCommandForRpcResponse(rpcResponse);
+ } catch (Throwable t) {
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
+
private RemotingCommand updateConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request)
throws RemotingCommandException {
final RemotingCommand response =
@@ -119,7 +154,7 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
.decodeCommandCustomHeader(UpdateConsumerOffsetRequestHeader.class);
TopicQueueMappingContext mappingContext = this.brokerController.getTopicQueueMappingManager().buildTopicQueueMappingContext(requestHeader);
- RemotingCommand rewriteResult = this.brokerController.getTopicQueueMappingManager().rewriteRequestForStaticTopic(requestHeader, mappingContext);
+ RemotingCommand rewriteResult = rewriteRequestForStaticTopic(requestHeader, mappingContext);
if (rewriteResult != null) {
return rewriteResult;
}
@@ -144,6 +179,7 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
if (mappingItemList.size() == 1
&& mappingItemList.get(0).getLogicOffset() == 0) {
//as physical, just let it go
+ mappingContext.setCurrentItem(mappingItemList.get(0));
requestHeader.setQueueId(mappingContext.getLeaderItem().getQueueId());
return null;
}
@@ -154,6 +190,7 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
//double read, first from leader, then from second leader
for (int i = itemList.size() - 1; i >= 0; i--) {
LogicQueueMappingItem mappingItem = itemList.get(i);
+ mappingContext.setCurrentItem(mappingItem);
if (mappingItem.getBname().equals(mappingDetail.getBname())) {
offset = this.brokerController.getConsumerOffsetManager().queryOffset(requestHeader.getConsumerGroup(), requestHeader.getTopic(), mappingItem.getQueueId());
if (offset >= 0) {
@@ -194,9 +231,31 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
response.setCode(ResponseCode.QUERY_NOT_FOUND);
response.setRemark("Not found, maybe this group consumer boot first");
}
+ RemotingCommand rewriteResponseResult = rewriteResponseForStaticTopic(requestHeader, responseHeader, mappingContext, response.getCode());
+ if (rewriteResponseResult != null) {
+ return rewriteResponseResult;
+ }
return response;
} catch (Throwable t) {
- t.printStackTrace();
+ return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
+ }
+ }
+
+
+ public RemotingCommand rewriteResponseForStaticTopic(final QueryConsumerOffsetRequestHeader requestHeader, final QueryConsumerOffsetResponseHeader responseHeader,
+ final TopicQueueMappingContext mappingContext, final int code) {
+ try {
+ if (mappingContext.getMappingDetail() == null) {
+ return null;
+ }
+ if (code != ResponseCode.SUCCESS) {
+ return null;
+ }
+ LogicQueueMappingItem item = mappingContext.getCurrentItem();
+ responseHeader.setOffset(item.computeStaticQueueOffsetStrictly(responseHeader.getOffset()));
+ //no need to construct new object
+ return null;
+ } catch (Throwable t) {
return buildErrorResponse(ResponseCode.SYSTEM_ERROR, t.getMessage());
}
}
@@ -245,6 +304,11 @@ public class ConsumerManageProcessor extends AsyncNettyRequestProcessor implemen
}
}
+ RemotingCommand rewriteResponseResult = rewriteResponseForStaticTopic(requestHeader, responseHeader, mappingContext, response.getCode());
+ if (rewriteResponseResult != null) {
+ return rewriteResponseResult;
+ }
+
return response;
}
}
diff --git a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java
index 56fc792..dd7e708 100644
--- a/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java
+++ b/broker/src/main/java/org/apache/rocketmq/broker/topic/TopicQueueMappingManager.java
@@ -189,7 +189,7 @@ public class TopicQueueMappingManager extends ConfigManager {
//Do not return a null context
public TopicQueueMappingContext buildTopicQueueMappingContext(TopicRequestHeader requestHeader, boolean selectOneWhenMiss) {
- //should disable logic queue explicitly, otherwise the old client may cause dirty data to newly created static topic
+ // if lo is set to false explicitly, it maybe the forwarded request
if (requestHeader.getLo() != null
&& Boolean.FALSE.equals(requestHeader.getLo())) {
return new TopicQueueMappingContext(requestHeader.getTopic(), null, null, null, null);
diff --git a/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java b/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
index 62e6ec1..3782ab0 100644
--- a/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
+++ b/common/src/main/java/org/apache/rocketmq/common/rpc/RpcClientImpl.java
@@ -28,6 +28,8 @@ import org.apache.rocketmq.common.protocol.header.GetMinOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.PullMessageResponseHeader;
import org.apache.rocketmq.common.protocol.header.QueryConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.protocol.header.SearchOffsetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.UpdateConsumerOffsetResponseHeader;
import org.apache.rocketmq.common.statictopic.TopicConfigAndQueueMapping;
import org.apache.rocketmq.remoting.InvokeCallback;
import org.apache.rocketmq.remoting.RemotingClient;
@@ -101,6 +103,9 @@ public class RpcClientImpl implements RpcClient {
case RequestCode.QUERY_CONSUMER_OFFSET:
rpcResponsePromise = handleQueryConsumerOffset(addr, request, timeoutMs);
break;
+ case RequestCode.UPDATE_CONSUMER_OFFSET:
+ rpcResponsePromise = handleUpdateConsumerOffset(addr, request, timeoutMs);
+ break;
case RequestCode.GET_TOPIC_STATS_INFO:
rpcResponsePromise = handleCommonBodyRequest(addr, request, timeoutMs, TopicStatsTable.class);
break;
@@ -234,6 +239,26 @@ public class RpcClientImpl implements RpcClient {
return rpcResponsePromise;
}
+ public Promise<RpcResponse> handleUpdateConsumerOffset(String addr, RpcRequest rpcRequest, long timeoutMillis) throws Exception {
+ final Promise<RpcResponse> rpcResponsePromise = createResponseFuture();
+
+ RemotingCommand requestCommand = RpcClientUtils.createCommandForRpcRequest(rpcRequest);
+ RemotingCommand responseCommand = this.remotingClient.invokeSync(addr, requestCommand, timeoutMillis);
+ assert responseCommand != null;
+ switch (responseCommand.getCode()) {
+ case ResponseCode.SUCCESS: {
+ UpdateConsumerOffsetResponseHeader responseHeader =
+ (UpdateConsumerOffsetResponseHeader) responseCommand.decodeCommandCustomHeader(UpdateConsumerOffsetResponseHeader.class);
+ rpcResponsePromise.setSuccess(new RpcResponse(responseCommand.getCode(), responseHeader, responseCommand.getBody()));
+ break;
+ }
+ default: {
+ rpcResponsePromise.setSuccess(new RpcResponse(new RpcException(responseCommand.getCode(), "unknown remote error")));
+ }
+ }
+ return rpcResponsePromise;
+ }
+
public Promise<RpcResponse> handleCommonBodyRequest(final String addr, RpcRequest rpcRequest, long timeoutMillis, Class bodyClass) throws Exception {
final Promise<RpcResponse> rpcResponsePromise = createResponseFuture();
RemotingCommand requestCommand = RpcClientUtils.createCommandForRpcRequest(rpcRequest);
diff --git "a/docs/cn/statictopic/RocketMQ_Static_Topic_Logic_Queue_\350\256\276\350\256\241.md" "b/docs/cn/statictopic/RocketMQ_Static_Topic_Logic_Queue_\350\256\276\350\256\241.md"
index c06d83f..ac1cc6b 100644
--- "a/docs/cn/statictopic/RocketMQ_Static_Topic_Logic_Queue_\350\256\276\350\256\241.md"
+++ "b/docs/cn/statictopic/RocketMQ_Static_Topic_Logic_Queue_\350\256\276\350\256\241.md"
@@ -7,6 +7,8 @@
| 2021-12-03 | 增加代码走读的说明| dongforever |
| 2021-12-10 | 引入Scope概念，保留『多集群动态零耦合』的集群设计模型 | dongforever |
| 2021-12-23 | 梳理待完成事项；讨论Admin接口的适配方式 | dongforever |
+| 2021-01-05 | Offset存储改成『转换制』，以更好适配原有逻辑 | dongforever |
+
@@ -342,8 +344,9 @@ UpdateStaticTopic 命令会自动计算预期的分布情况，包括但不限
#### consumerOffsets 系列
-Offset的存储，无需转换，直接存储在 LogicQueue 所对应的最新 PhysicalQueue 中。
-读取时，采取『Double-Read-Check』机制。
+Offset的存储，进行转换，存储在对应PhysicalQueue 所在的 Broker上面。
+读取时，采取『Double-Read-Check』机制，并进行转换。
+这样可以最大程度与 PhysicalQueue 的相关逻辑进行适配，比如 ConsumerProgress 可以看到『最近拉取时间』。
#### Client
@@ -454,6 +457,8 @@ User 接口，使用范围广泛如多语言等，应该尽可能简单，把适
#### 阻止Pop模式、事务消息、定时消息使用 LogicQueue
不兼容 事务消息和定时消息。
LogicQueue 当前不支持Pop模式消费。
+#### Nameserver 相关生命周期完善
+目前没有处理Nameserver中Mapping数据的生命周期
#### ConsumeQueue 的 correctMinOffset 逻辑存在缺陷
可能导致 LogicQueue 无法清除已经过期的 MappingItem。
#### getOffsetInQueueByTime 语义有缺陷
[rocketmq] 16/17: change the level to warn when the ack message failed (#3756)
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch 5.0.0-alpha
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 5d15c3ed170f84bea092ee7d9418dd175cb00487
Author: cserwen <cs...@163.com>
AuthorDate: Sat Jan 15 17:26:38 2022 +0800
change the level to warn when the ack message failed (#3756)
---
.../rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
index abe3ca7..87440ff 100644
--- a/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
+++ b/client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
@@ -796,12 +796,12 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
@Override
public void onSuccess(AckResult ackResult) {
if (ackResult != null && !AckStatus.OK.equals(ackResult.getStatus())) {
- log.info("Ack message fail. ackResult: {}, extraInfo: {}", ackResult, extraInfo);
+ log.warn("Ack message fail. ackResult: {}, extraInfo: {}", ackResult, extraInfo);
}
}
@Override
public void onException(Throwable e) {
- log.info("Ack message fail. extraInfo: {} error message: {}", extraInfo, e.toString());
+ log.warn("Ack message fail. extraInfo: {} error message: {}", extraInfo, e.toString());
}
}, requestHeader);